How GPTBot, ClaudeBot and PerplexityBot Actually Crawl Your Site
The technical differences between AI crawlers, why robots.txt typos silently fail, and why one major engine doesn't reliably respect it at all.
GPTBot, ClaudeBot and PerplexityBot are not one thing. Some are slow training crawlers that respect robots.txt strictly, others are real-time search crawlers that need to be allowed for your page to show up in a live answer, and none of them render JavaScript. Perplexity has also been documented using undeclared crawlers that ignore disallow rules entirely, so robots.txt alone is not a guarantee.
Two different jobs, two different crawlers
"AI crawler" gets used as if it's one category. It isn't. There are two families with different missions and different HTTP behavior:
Training crawlers (GPTBot, ClaudeBot, anthropic-ai) feed the datasets that get baked into a model at training time. They crawl in bulk, without urgency, and generally respect robots.txt.
Real-time search crawlers (OAI-SearchBot for ChatGPT Search, PerplexityBot, Claude-SearchBot) fetch pages continuously to answer a live query. If one of these can't reach your page, you don't just miss training data, you miss being cited in an answer someone is looking at right now.
That distinction matters for what you block. Disallowing GPTBot in robots.txt keeps your content out of a future training run. It does nothing to stop OAI-SearchBot from citing the same page in ChatGPT Search today, because it's a separate declared agent.
| Crawler | Company | Job | Respects robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Real-time search | Yes |
| ClaudeBot | Anthropic | Training | Yes |
| PerplexityBot | Perplexity | Real-time search | Documented exceptions (see below) |
| Google-Extended | AI training / Gemini | Yes | |
| Googlebot | Search + AI Overviews | Yes |
robots.txt only works if the string matches exactly
Every one of these crawlers identifies itself with a specific user-agent string, and the match has to be exact. User-agent: GPT-Bot (with a hyphen) does nothing against GPTBot (no hyphen). The rule silently fails, you see no error, and you just quietly keep serving a crawler you thought you'd blocked. This is the single most common reason a site owner is confident they've blocked an AI crawler and hasn't.
Run a free LLM Access Check against your own domain and it will show you exactly what each of these agents actually receives right now, not what your robots.txt file intends.
None of them render JavaScript
This is the part that surprises people who are used to thinking about Googlebot. Googlebot renders your page in a headless browser and sees the final DOM. GPTBot, ClaudeBot and PerplexityBot fetch the raw HTTP response and read it as-is. If your content, headings, or product details are injected client-side by JavaScript, these crawlers see an empty shell where Googlebot sees a full page. A page can rank on page one of Google search and be completely invisible to every one of these three at the same time.
If you serve content via client-side rendering, the fix is server-side rendering or prerendering the parts that matter, not a robots.txt change.
Perplexity's compliance problem is real, not a rumor
On August 4, 2025, Cloudflare published a report showing Perplexity using undeclared crawlers that rotate user-agents, IPs and ASNs specifically to get around disallow rules. Cloudflare's own conclusion was direct: robots.txt is not a reliable defense against Perplexity if Perplexity doesn't want to respect it. If keeping a specific crawler off your content matters for legal or business reasons, robots.txt alone isn't sufficient evidence that you've done it, you need server-side blocking (WAF rules, IP-based blocking) as a backstop.
Where llms.txt actually stands right now
You'll see llms.txt recommended everywhere as the modern equivalent of robots.txt for AI. The reality in 2026 is more modest: it's a community convention, not an IETF or W3C standard, adoption sits around 10% of domains studied, and no major AI provider (OpenAI, Anthropic, Google, Meta) has publicly confirmed their production crawlers actually read or act on it. It's worth having if you're already investing in AEO, since it costs almost nothing to add, but it is not currently a crawler requirement the way robots.txt is. Our checklist flags it as a nice-to-have, not a blocker, for exactly this reason.
What to actually check
- Confirm your robots.txt uses the exact user-agent string for each crawler you're trying to allow or block.
- Fetch your page's raw HTML (not the rendered DOM) and confirm the content you care about is actually there, not injected by JS.
- If a specific crawler must be blocked for compliance reasons, back robots.txt with server-side enforcement.
- Run the full AEO & GEO audit to see this scored against your actual page, not a general checklist.
FAQ
Does blocking GPTBot in robots.txt stop ChatGPT from citing my page?
No. GPTBot is a training crawler; ChatGPT's live citations come from OAI-SearchBot, a separate declared agent you'd need to block on its own.
Is llms.txt worth adding right now?
It's low-cost and harmless, but no major AI provider has confirmed their crawlers use it yet, so treat it as a nice-to-have, not a fix for a crawlability problem.
Why does Perplexity still show up after I disallowed PerplexityBot?
Cloudflare documented Perplexity using undeclared crawlers that rotate identifiers specifically to bypass disallow rules. robots.txt alone may not be enough for this one engine.