Skip to the page

AI crawler and robots.txt checker

Check whether Googlebot, OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot, and eight other crawlers can read any domain, plus llms.txt and RSL status.

Checks 14 search and AI bots, robots.txt, sitemaps, llms.txt, and RSL. No sign-up required.

AI crawler checker questions

Something we didn't answer? Talk to a human

Why separate AI search bots from AI training bots?
Bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot index your site for live search citations, while GPTBot, ClaudeBot, and CCBot collect data for model training. Many sites allow search bots while blocking training bots.
What is llms.txt?
A short Markdown index at /llms.txt that gives AI assistants and coding agents a clean map of your documentation, pricing, and product pages without parsing HTML chrome.
What is RSL?
Really Simple Licensing (rsl.xml or a License line in robots.txt) lets publishers state machine-readable licensing terms for search, AI input, and model training.

Monitor crawler rules and site changes automatically

Use SpecStream to extract, verify, and monitor live web pages on a schedule via API or MCP.