> SpecStream is the live web data engine for teams and agents. This is the Markdown copy of https://specstream.ai/studies/ai-crawler-rules-2026. Every page, with what SpecStream does and costs: https://specstream.ai/llms.txt

# Which AI crawlers the web’s top sites block

On November 20, 2026 we read the robots.txt file of each of the 10,000 most linked sites of SpecStream’s index. 7,274 answered with a readable file. This is what they say to AI crawlers.

[Download the table](https://specstream.ai/studies/ai-crawler-rules-2026.tsv)[Get an API key](https://specstream.ai/signup?src=study-ai-crawlers&returnTo=%2Fworkspace%2Fsearch)

## The count

One site in five turns at least one AI crawler away by name. Almost none turn away search.

21.9%

1,591 sites block at least one AI crawler by name

1,141

sites block GPTBot from the whole site

1,047

sites block ClaudeBot from the whole site

7

sites block Googlebot

## Crawler by crawler

Share is of the 7,274 files read. A file blocks a crawler when it names it and disallows the whole site.

Crawler

Files that name it

Files that block it

Share

CCBot

Common Crawl, open archive used for training

1,503

**1,177**

16.2%

Bytespider

ByteDance, training

1,369

**1,153**

15.9%

GPTBot

OpenAI, training

1,675

**1,141**

15.7%

ClaudeBot

Anthropic, training

1,497

**1,047**

14.4%

meta-externalagent

Meta, training

1,238

**959**

13.2%

Applebot-Extended

Apple, training

1,202

**923**

12.7%

Google-Extended

Google, training

1,365

**919**

12.6%

Amazonbot

Amazon

1,178

**876**

12.0%

PerplexityBot

Perplexity, search

946

**541**

7.4%

ChatGPT-User

OpenAI, a person asking

893

**470**

6.5%

OAI-SearchBot

OpenAI, search

728

**304**

4.2%

Claude-SearchBot

Anthropic, search

537

**295**

4.1%

Bingbot

Microsoft, search

541

**18**

0.2%

Googlebot

Google, search

622

**7**

0.1%

## What the numbers say

-   Training is blocked three to four times as often as searchGPTBot is blocked by 1,141 sites and OAI-SearchBot by 304. ClaudeBot is blocked by 1,047 and Claude-SearchBot by 295. Sites refuse to feed models and still want to be cited by them.

-   The bigger the site, the more it blocksAmong the top 100 sites, 36% block an AI crawler by name. From 101 to 1,000 it is 30%, and from 1,001 to 10,000 it is 21%.

-   GPTBot is the most named, CCBot the most refused1,675 files mention GPTBot and a third of them let it in. Common Crawl’s CCBot is refused by 78% of the files that name it.

-   A new vocabulary is spreading480 files (6.6%) carry a Content-Signal line. 365 of them say ai-train=no, and nearly all 480 say search=yes. 20 files carry a License line.

## Use the data

The table is free to [download](https://specstream.ai/studies/ai-crawler-rules-2026.tsv) and to quote with a link to this page. To see what a site allows before your agent reads it, or to build the same count for your own list of sites, ask SpecStream in plain words. Related data: [sites and what they publish](https://specstream.ai/data/sites), [companies](https://specstream.ai/data/companies) and [subprocessors](https://specstream.ai/data/subprocessors).

## Questions

Something we didn't answer? [Talk to a human](https://specstream.ai/talk-to-a-human.md)

How many top websites block GPTBot?

1,141 of the 7,274 robots.txt files we read on November 20, 2026 block GPTBot from the whole site. That is 15.7%.

How many top websites block ClaudeBot?

1,047 of 7,274, or 14.4%. Claude-SearchBot, which fetches pages for answers, is blocked by 295.

Do sites that block AI crawlers also block Google?

Almost never. Only 7 of the 7,274 files block Googlebot and 18 block Bingbot, while 1,591 block at least one AI crawler by name.

When does a site count as blocking a crawler?

When its robots.txt names the crawler and disallows the whole site for it. A site that only blocks some folders is not counted.

Which sites were read?

The 10,000 most linked sites of SpecStream’s index of the web. 7,274 answered with a readable robots.txt file.

## Run this on your own list

Describe the sites and the facts you want. SpecStream reads the live web and returns rows with a source address for each value.

[Get an API key](https://specstream.ai/signup?src=study-ai-crawlers&returnTo=%2Fworkspace%2Fsearch)[Talk to a human](https://specstream.ai/talk-to-a-human.md)
