AI Crawlers
AI crawlers are the bots that fetch web content for AI systems. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI training crawler) and Applebot-Extended all index or read your pages for use in answers, training and retrieval. They identify themselves in the user agent, respect robots.txt, and are the retrieval layer behind RAG-based AI answers.
Why It Matters
What the AI crawlers can read is what AI search can cite. If robots.txt blocks GPTBot and PerplexityBot, then every answer engine that would have named you answers without you - and names your competitors instead. For businesses targeting AI-visible queries, allowing the AI crawlers is as important as allowing Googlebot.
The crawlers differ from Googlebot in one key way: they're less forgiving of JavaScript-heavy sites and they value clean, parseable content. A technically strong site that ranks on Google but renders content client-side may be effectively invisible to the AI crawlers, and invisible to AI search.
In Practice
Check your robots.txt allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended (with the caveat that Google-Extended is for training; Google's AI Overviews use regular Googlebot). Audit what the crawlers actually fetch - they should see your content the way a reader does. Keep answers in the HTML, not injected by JavaScript. Monitor the AI search engines to confirm your content is being retrieved and cited.
There's a legitimate reason to block a crawler: if a bot is scraping content you don't want used. But blocking AI crawlers by default is like blocking Googlebot because you're nervous - it trades visibility for a fear that usually doesn't materialise.
Common Mistakes
Blocking all AI crawlers in robots.txt without knowing what they do, or assuming Googlebot coverage means AI coverage. The bots are different readers with different access - manage them deliberately.
Sources & Further Reading
Where to verify this yourself - Google's own documentation, industry reporting, and how we apply it at Underdog. Don't take our word for it; check the source.
Related Terms
Glossary
Robots.txt
A file telling search engine crawlers which parts of your site they can access.
Glossary
Crawling
How search engine bots discover and download your pages - the first step to ranking.
Glossary
AI Search (Answer Engines)
Perplexity, ChatGPT Search, Gemini - the AI engines people now ask questions to instead of Google. Different rules, same goal: being the cited answer.
Glossary
RAG (Retrieval-Augmented Generation)
The technique that lets AI answers draw from your content instead of guessing. RAG is why having crawlable, citable pages still matters in the AI era.
Glossary
JavaScript SEO
Optimising JS-heavy sites so search engines can actually see and index the content.
Know the Words.
Now See Them in Action.
Free teardown. No jargon. Just what's broken and how to fix it.
Get The Teardown