Which AI crawlers the web’s top sites block
On November 20, 2026 we read the robots.txt file of each of the 10,000 most linked sites of SpecStream’s index. 7,274 answered with a readable file. This is what they say to AI crawlers.
The count
One site in five turns at least one AI crawler away by name. Almost none turn away search.
- 21.9%
- 1,591 sites block at least one AI crawler by name
- 1,141
- sites block GPTBot from the whole site
- 1,047
- sites block ClaudeBot from the whole site
- 7
- sites block Googlebot
Crawler by crawler
Share is of the 7,274 files read. A file blocks a crawler when it names it and disallows the whole site.
What the numbers say
- Training is blocked three to four times as often as searchGPTBot is blocked by 1,141 sites and OAI-SearchBot by 304. ClaudeBot is blocked by 1,047 and Claude-SearchBot by 295. Sites refuse to feed models and still want to be cited by them.
- The bigger the site, the more it blocksAmong the top 100 sites, 36% block an AI crawler by name. From 101 to 1,000 it is 30%, and from 1,001 to 10,000 it is 21%.
- GPTBot is the most named, CCBot the most refused1,675 files mention GPTBot and a third of them let it in. Common Crawl’s CCBot is refused by 78% of the files that name it.
- A new vocabulary is spreading480 files (6.6%) carry a Content-Signal line. 365 of them say ai-train=no, and nearly all 480 say search=yes. 20 files carry a License line.
Use the data
The table is free to download and to quote with a link to this page. To see what a site allows before your agent reads it, or to build the same count for your own list of sites, ask SpecStream in plain words. Related data: sites and what they publish, companies and subprocessors.
Questions
Something we didn't answer? Talk to a human
How many top websites block GPTBot?
How many top websites block ClaudeBot?
Do sites that block AI crawlers also block Google?
When does a site count as blocking a crawler?
Which sites were read?
Run this on your own list
Describe the sites and the facts you want. SpecStream reads the live web and returns rows with a source address for each value.