Your robots.txt is perfect. Your CDN is blocking ChatGPT.

Every AEO checklist you have read is a list of things that earn a citation. Almost none of them list the things that prevent one, which is backwards, because the blockers win. A page an AI engine cannot fetch cannot be cited however well it is written. There are five structural blockers: a blocked search-tier crawler, a CDN or firewall quietly overriding your robots.txt, JavaScript-only content, an answer buried under a long intro, and no third-party corroboration anywhere on the web. Clear those and the clever tactics start to matter. Skip them and nothing else does.

I know the second one from the inside. In May 2026 our robots.txt was immaculate, every AI crawler explicitly allowed, and our citation rate was sitting near zero because Cloudflare's managed AI-bot toggle was returning a 403 at the edge before any crawler reached the file. Nothing in the file was wrong. The file was not the control. That is the problem with reading your robots.txt and calling it done, and on 15 September it gets a wrinkle worth understanding before somebody sells you a panic about it.

Five things, ranked by how thoroughly they finish you.

One: the engine's search crawler is disallowed in robots.txt. Two: your CDN or firewall blocks it at the edge regardless of what robots.txt says. Three: the content only exists after JavaScript runs, and most AI crawlers do not run JavaScript. Four: the answer sits four sections down, below a warm-up narrative nobody quotes. Five: nothing on the rest of the web corroborates you, so there is no third-party page for an engine to cite instead.

This ordering is not a hunch. A meta-analysis of 54 studies scored crawler and URL accessibility as the single strongest citation factor at 9.5 out of 10 (Zyppy, 2026). It is near-binary: fail it and nothing further down the list can compensate. Structured data, for scale, scores 5.6 on the same measure, and Google states plainly that it is not required for generative AI search. Most AEO budgets are spent in inverse order to that evidence, which is how you end up with a schema audit on a site the crawlers cannot reach.

Because robots.txt is a request and your CDN is a wall. Different layers, and only one of them answers the door.

Cloudflare's managed AI-bot toggle, Bot Fight Mode, and any WAF rule that catches non-browser user agents all act before your origin sees the request. The crawler gets a 403 and your file gets no say. Citation just stops, and since volatility here is high anyway, it reads as weather rather than a wall.

Ours ran that way for an unknown period in May 2026, caught only because we read the served file rather than the one in our repo. So that is the check now. Fetch your live robots.txt over the public internet: if the first line is a comment block about conditions of access instead of your own directives, a managed rule is rewriting it. Then request a real page with a crawler user agent from outside your network and confirm a 200, not a challenge. Cloudflare now sorts AI bots into Search, Agent and Training and lets you handle each separately, so a refusal can also arrive as an HTTP 402.

A blocked crawler produces no error, no alert, and no gap in your analytics. It produces silence, and silence in this channel is indistinguishable from a slow month.

Less than the headlines suggest, and one thing more dangerous than either.

From 15 September, domains newly onboarding to Cloudflare get Training and Agent crawlers blocked by default, but only on pages that display ads, and the Search tier remains allowed. It does not change existing domains and it is not a blanket AI block. Search is the tier that gates citation, so the default itself does not stop you being cited. If you see a post claiming Cloudflare is switching off AI citation on that date, close it.

The real hazard is quieter and already live. Cloudflare applies the strictest matching rule to a crawler serving several purposes, and Googlebot crawls for both search and training. A site that switches on Training-tier blocking can therefore block Googlebot, and lose AI Overviews and classic organic search together. That is the most damaging easily-made mistake in this area, and it is made by people being careful, not careless. Blocking training crawlers is defensible if your content is the product. Blocking search-tier crawlers removes the only path most brands have to citation at all.

The search-tier one. Almost every robots.txt argument on the internet is about the wrong bot.

Each major engine runs a split fleet. OAI-SearchBot indexes for ChatGPT search while GPTBot collects training data, and they are independent: block GPTBot and you can still be cited, block OAI-SearchBot and you cannot. Anthropic runs the same split with Claude-SearchBot and ClaudeBot. Perplexity uses PerplexityBot. Google's AI Overviews and AI Mode draw from the ordinary Googlebot index.

Which makes Google-Extended the most expensive misunderstanding in the category. It is a training opt-out token only. It does not affect AI Overview eligibility in either direction, so disallowing it costs you nothing there and allowing it wins you nothing. Two more asymmetries are worth knowing: OpenAI documents that ChatGPT-User, the user-triggered fetcher, does not respect robots.txt, so a rule there is not a control. Anthropic's equivalent, Claude-User, does respect it, and a hit from it is the clearest signal that Claude is actively recommending your page to a person.

Being quotable in your first 300 words, and being fetchable by OAI-SearchBot. Not ranking in Bing.

The Bing thing needs retiring properly, because it is still in circulation. In early 2025 it was fair to say ChatGPT leaned on Bing and that most of its citations matched Bing's top results. That is obsolete as architecture. Mid-2026 instrumentation shows retrieval fanning out across several commercial fetch networks, with the mix varying by user cohort and changing week to week, plus OpenAI's own index and cache. Bing is one gated experiment among several, not the foundation. Optimising for Bing rank as a ChatGPT proxy is now a strategy pointed at a pipe that may not be in the path.

What holds is duller and more durable. Allow OAI-SearchBot. Answer the actual question in the first 150 to 300 words, in language a model can lift without editing. Roughly 44.2 percent of citation extractions come from the first 30 percent of body text (AirOps, 2026), so answer placement is a bigger lever than answer length. Comparative and alternatives coverage performs strongly here, both yours and other people's.

Community presence and recency. Schema effort here is close to wasted.

Perplexity runs its own crawler and index with a heavy recency bias, and extracts at passage level, so every paragraph has to stand on its own. It can cite content published within the hour and skews hard to current-year pages. Its declared crawl footprint is small: Googlebot reaches around 167 times more unique URLs than PerplexityBot (Cloudflare, January 2026), which is a useful corrective if you assumed it was crawling the whole web.

Reddit remains its single largest source domain. Note the correction there, because the widely quoted 46.7 percent figure was a pre-litigation peak measured on a top-citations denominator, and the share now sits at roughly a fifth to a quarter. Still the biggest single domain on any engine, still volatile enough that you should quote it softly. We covered the mechanics of that in why Reddit outranks you. What prevents a citation here: blocking PerplexityBot, stale pages with no dateModified, corporate pages with zero community corroboration, and astroturfed presence, where Reddit's own detection is the binding constraint rather than Perplexity's.

Being findable in Brave. This is the most neglected AI-visibility action available, because almost nobody does it.

Claude's web search is powered by Brave, which Anthropic lists as a subprocessor, and independent measurement in June 2026 put Claude-versus-Brave citation overlap at around 79.2 percent across roughly 400 queries. Brave runs one of the few fully independent major indexes, around 30 billion pages, with its own robots-respecting crawler. So there is a second index in this market with a fraction of the competition, and the entire SEO industry is queuing at the other door.

There is no Brave webmaster console, so the sequence is short. Confirm Bravebot is not caught by a blanket disallow in robots.txt or at your CDN, the trap almost everyone falls into because nobody blocks Bravebot on purpose. Submit at search.brave.com/submit-url. Part of Brave's index comes from its Web Discovery Project and is outside your influence; the rest is ordinary ranking. Beyond that, Claude rewards a neutral authoritative register with clear author attribution, and punishes superlatives harder than any other engine. Write the way you would want to be quoted.

SEO fundamentals, done on a page the engine can actually quote. But the top-10 assumption has quietly expired.

Google's own position is that there are no additional requirements to appear in AI Overviews or AI Mode: no AI files, no special markup. What it asks for is non-commodity content with a unique point of view, indexed, snippet-eligible and server-rendered. Retrieval is grounded in the core Search ranking systems plus query fan-out, where the model fires several related queries at once. So answer the questions adjacent to the main one, not just the literal one.

The shift worth pricing in is the top-10 assumption. The coupling between AI Overview citation and organic rank is weakening fast: top-10 share of AIO citations has fallen from roughly 76 percent to about 38 percent, and around 60 percent now come from URLs outside the top 20. Rank top-10 and you will be cited is no longer a safe restatement of Google's it-is-still-SEO thesis. The gate is moving from do you rank toward can the engine fetch and quote you. Gemini shares the index and the guidance; treat them as one surface.

Five moves, in this order. The first two are unglamorous and they are the ones that decide it.

One: confirm no AI search crawler is blocked, in robots.txt and at your CDN, checked against the live served file rather than the one in your repo. Two: server-render whatever you want quoted, because most AI crawlers do not execute JavaScript and will read a blank page instead, as we measured in AI crawlers can't read your website. Three: answer the buyer's real question in the first paragraph, in plain language. Four: build third-party corroboration, because around 85 percent of AI citations point at domains other than the brand's own. Five: keep it fresh, quarterly at minimum. Pages off that cadence are around three times more likely to lose citations.

Because the engines are probabilistic, and you are reading one sample off a noisy distribution.

Volatility runs 40 to 60 percent month to month, only about 30 percent of brands stay visible from one answer to the next, and only 20 percent remain present across five consecutive runs (AirOps, 2026). Run the same prompt twice and you can get two different answers citing two different sources, minutes apart. So a single check that shows you cited proves little, and a single check that shows you absent proves less.

One query on one engine on one day is a sample, not a measurement, which is why our own AI Radar runs each prompt several times and reports a band rather than a verdict. The full sequence, with effect sizes on every move, is the AI Citability Playbook.