30 terms across crawlers, surfaces, metrics and technique — each with what it is, and why it actually matters for your site.
User-agent: GPTBot OpenAI's crawler for gathering model training data. It fetches raw HTML and does not execute JavaScript.
Why it mattersBlocking it keeps your content out of future model training, but has no effect on whether ChatGPT cites you in live answers — that is a different bot entirely.
User-agent: OAI-SearchBot OpenAI's crawler for ChatGPT Search — the live, cited web answers inside ChatGPT. Separate from GPTBot.
Why it mattersThis is the OpenAI bot that determines whether ChatGPT can cite you today. Sites routinely block it by accident while meaning to opt out of training only.
User-agent: ClaudeBot Anthropic's web crawler for Claude. Respects robots.txt and reads raw HTML only.
Why it mattersLike every answer-engine crawler, it never runs your JavaScript — so a client-rendered page is invisible to it regardless of robots.txt.
User-agent: PerplexityBot Perplexity's crawler, used to fetch pages for its live cited answers.
Why it mattersPerplexity surfaces its sources prominently, so a citation here is unusually visible — and unusually easy to lose to a WAF rule you did not know existed.
User-agent: Google-Extended A robots.txt token that opts your content out of Gemini and AI Overviews grounding, separately from ordinary Googlebot search indexing.
Why it mattersIt is the only way to stay in Google Search while opting out of Google's AI surfaces. Disallowing it does not hurt your blue-link rankings.
User-agent: Googlebot Google's classic search crawler — and the one major crawler that runs a full browser engine to render JavaScript before indexing.
Why it mattersGooglebot is the reason JS-rendering problems stay hidden. Your page indexes perfectly on Google while every AI crawler sees an empty shell.
A single bot used for both traditional search indexing and AI training or answer generation, making it impossible for a site to allow one and refuse the other.
Why it mattersCloudflare set a September 2026 deadline for AI companies to split these, blocking non-compliant bots on monetised pages. Expect crawler identities to fragment further.
User-agent: Applebot-Extended Apple's opt-out token for using crawled content in Apple Intelligence training, distinct from the Applebot that powers Siri and Spotlight.
Why it mattersSame two-bot pattern as OpenAI and Google — the opt-out and the citation crawler are different tokens.
Google's AI-generated summary above traditional results, synthesising and citing a handful of sources.
Why it mattersCorrelated with a 58% CTR drop for top-ranking pages (Ahrefs, Feb 2026) — but cited brands earn ~120% more clicks per impression than uncited ones on the same query.
Google's conversational, multi-turn AI search experience — a deeper AI-first surface than AI Overviews.
Why it mattersFollow-up questions mean a single page can be re-queried from several angles, rewarding pages that answer a cluster of related questions rather than exactly one.
A link block Google added to the end of AI Overviews in May 2026, pointing to specific articles, case studies and reports.
Why it mattersA separately winnable citation slot that favours specific, substantive, single-topic pages over broad category or hub pages.
Live, cited web answers inside ChatGPT, powered by OAI-SearchBot plus a re-ranked hybrid index.
Why it mattersOnly ~10% of URLs it cites rank in Google's top 10 — strong evidence that AI visibility and SEO rank are genuinely different problems.
Product comparison and buyer's-guide blocks that ChatGPT and other engines generate on queries signalling buying intent.
Why it mattersIt reads product attributes, specs and review text straight from your HTML. Client-rendered price and review widgets simply do not exist to it.
How often an AI engine names your domain as a source across a defined set of prompts.
Why it mattersThe core AEO metric, and the one that maps most directly to traffic. Unlike rank, it must be measured by actually querying the engines.
Your citation frequency relative to named competitors across the same prompt set — "cited in 24% of answers vs their 40%".
Why it mattersCitation rate alone is not actionable. Share of voice tells you whether a low number is a category problem or specifically your problem.
How many of the realistic questions a user might ask about a topic are directly and completely answered on a given page.
Why it mattersUsually the cheapest lever available. Most pages answer one question well and four adjacent ones not at all.
A search where the user gets their answer on the results page and never visits a site.
Why it mattersThe structural reason AEO exists. If the answer is consumed in the interface, being the cited source is the only remaining form of visibility.
Content that only appears after JavaScript executes in a browser.
Why it mattersThe single most common cause of AI invisibility. Googlebot renders it; GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot do not. The failure is silent and total.
Generating full HTML on the server so content exists in the first response, before any client JavaScript runs.
Why it mattersThe fix for the JS blind spot. You do not have to drop your framework — the substance just has to be in the initial HTML, with JS layered on for interactivity.
/robots.txt A root-level file declaring which crawlers may fetch which paths.
Why it mattersNecessary but no longer sufficient. A CDN or WAF rule can block GPTBot at the edge while your robots.txt cheerfully allows it — your server never sees the request.
/llms.txt A proposed root-level Markdown file listing a site's key pages for AI crawlers.
Why it mattersLargely ineffective in practice. An SE Ranking study of 300,000 domains found its presence carried no predictive signal for citation frequency; the major crawlers skip it and read HTML directly.
schema.org Machine-readable markup describing a page's type, attributes and relationships — Product, Article, FAQPage, Review.
Why it mattersIt removes guesswork. An engine reading explicit Product markup does not have to infer your price from prose, and Q&A markup turns "readable" into "quotable".
aggregateRating declares a score and a review count; Review entries carry actual individual review text.
Why it mattersDeclaring "1,200 reviews" while rendering three means the engine has three reviews of evidence. The aggregate number persuades humans; the review text persuades models.
Cloudflare mechanisms for charging AI companies for content access — per fetch originally, evolving to payment when content surfaces in an answer.
Why it mattersSignals that crawler access is becoming a negotiated commercial arrangement rather than a default, especially for publishers.
How easily a model can lift a clean, self-contained, quotable answer out of your content.
Why it mattersA direct sentence near the top of a section outperforms the same fact hedged and buried three paragraphs down. This is a formatting problem more than a writing-quality one.
Naming concrete entities — products, people, places, versions — explicitly rather than leaning on pronouns and back-references.
Why it mattersModels retrieve passages, not whole pages. A chunk saying "it supports this natively" is unusable once separated from the paragraph that defined "it".
The passage-sized unit of text a retrieval system actually indexes and feeds to a model — typically a section, not a page.
Why it mattersExplains most AEO advice at once. Each section has to stand alone, because a chunk is often all the model ever sees of you.
A missing element — no FAQ block, no author byline, no specs table, no comparison — that a competitor supplies and wins the citation with.
Why it mattersCitations are competitive, not absolute. You lose them to a page that answered one more question than you did.
The newsroom structure: conclusion first, then supporting detail, then background.
Why it mattersIt happens to be the ideal shape for extraction. The answer sits at the top of the chunk where a model is most likely to lift it cleanly.
Experience, Expertise, Authoritativeness, Trust — Google's quality framework, expressed on-page through bylines, credentials, citations and dates.
Why it mattersThe signals overlap heavily with what makes a source citable: named authors, dated content and linked primary sources all raise the odds of being quoted.
No terms match that. Try a broader word.
The useful question is which of these your own pages are currently failing. A check takes about 20 seconds and tells you exactly what each AI crawler can and can't read.
Run an LLM Access Check