AI crawler and robots.txt checker
Check whether Googlebot, OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot, and eight other crawlers can read any domain, plus llms.txt and RSL status.
AI crawler checker questions
Something we didn't answer? Talk to a human
Why separate AI search bots from AI training bots?
Bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot index your site for live search citations, while GPTBot, ClaudeBot, and CCBot collect data for model training. Many sites allow search bots while blocking training bots.
What is llms.txt?
A short Markdown index at /llms.txt that gives AI assistants and coding agents a clean map of your documentation, pricing, and product pages without parsing HTML chrome.
What is RSL?
Really Simple Licensing (rsl.xml or a License line in robots.txt) lets publishers state machine-readable licensing terms for search, AI input, and model training.
Monitor crawler rules and site changes automatically
Use SpecStream to extract, verify, and monitor live web pages on a schedule via API or MCP.