69% of AI crawlers can't read your website. And one was caught wearing a disguise.

Can AI crawlers read JavaScript? Mostly, no. In an audit of 32 AI crawlers across 1.3 billion requests, GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User and PerplexityBot all read static HTML only (searchVIU, November 2025). Only Google's crawlers, Applebot and OpenAI's browsing agent render JavaScript. If your content appears only after the page hydrates, most AI search engines never see it.

Here is what that looks like in practice. You spent six months building a SaaS landing page. Beautiful hero section. Social proof carousel. Pricing table that animates on scroll. You ship it. Then ChatGPT's crawler visits your homepage, sees a white rectangle with a nav bar, and leaves. Your React app is a haunted house to it: it walks in, sees nothing, and walks out. The nav bar survives. Congratulations.

This post covers what AI crawlers actually see on your site (not much), the bot that was caught impersonating a real browser, why blocking them all usually hurts a small site more than it helps, and the six checks that fix most of it.

You have more AI bots visiting your site than you have customers. Here is the roll call.

GPTBot, OpenAI's training crawler, grew 305% year over year (Cloudflare, May 2025). PerplexityBot grew 157,490%. That is not a typo. ChatGPT-User, the bot that fires when someone asks ChatGPT to look something up, grew 2,825%.

The part most people miss: OpenAI and Anthropic each run three separate bots with three separate jobs. One collects training data (GPTBot, ClaudeBot). One builds a search index (OAI-SearchBot, Claude-SearchBot). One fetches pages live when a user asks (ChatGPT-User, Claude-User). The search bots are the ones that decide whether you can be cited. Blocking the training bot does nothing to the other two.

Then there is the crawl-to-referral ratio: pages crawled for every click sent back to you. Google's runs from 3:1 to 30:1. OpenAI's was 3,700:1. Anthropic's was 25,000:1 to 100,000:1 (Cloudflare, 2025). They crawl hard and send very little back. That is a reason to make every crawl count, not a reason to hide.

GPTBot fetches your HTML. Your HTML says "load React." GPTBot does not load React. GPTBot sees nothing.

It is like handing someone a book where the words only appear under a reading lamp. They do not have a reading lamp. They are very enthusiastic about reading. They just cannot do it.

searchVIU's audit tested 32 AI crawlers. Googlebot, Applebot and OpenAI's browsing agent render JavaScript. GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Meta-ExternalAgent, Amazonbot and Bytespider read static HTML only. GPTBot fetches JavaScript files in 11.5% of its requests and executes none of them. ClaudeBot fetches them in 23.8% and executes none.

searchVIU also ran a price extraction test: could five AI systems find pricing that only appeared through JavaScript? Gemini managed 50%. ChatGPT 37.5%. Google AI Mode 25%. Perplexity 12.5%. Claude 0%. Not low. Zero.

So if you check your site the way Google sees it, you are checking with one of the few bots that can read it. A page can look perfect in Google Search Console and be blank to ChatGPT, Claude and Perplexity.

Here is everything GPTBot and ClaudeBot walk straight past when it is rendered on the client.

If you are counting, that is your pricing, your social proof, your conversion path, your product demo and your best customer evidence. Gone. The nav bar survived.

One ironic exception. Some news paywalls render through JavaScript, so crawlers that skip JavaScript also skip the paywall and see the full article sitting in the raw HTML before the gate appears. The bots that cannot read anything accidentally read the things they were not supposed to.

If your plan was to put your facts in JSON-LD so the AI bots could find them without rendering anything, the tests are not kind to it.

searchVIU tested five AI systems (ChatGPT, Claude, Gemini, Perplexity, Google AI Mode) against eight schema scenarios. Could they extract a fact that existed only in JSON-LD, with no visible text saying the same thing? None of the five did. Hidden Microdata and concealed RDFa were ignored too.

Google's own guidance for AI features points the same way: structured data is not required to appear in AI Overviews or AI Mode, and there is no special markup for them. A 2026 meta-analysis of 54 studies rated structured data as a consistent but small positive for AI citation, well behind crawler access, which ranked first (Zyppy, 2026).

So schema is a small extra on top of visible text, not a workaround for missing text. If a fact matters, it needs to be in the HTML a person can read. Like a human reader, except less patient.

This is where it gets weird.

In August 2025 Cloudflare published an investigation into Perplexity. It reported that when its declared crawler was blocked, Perplexity fetched pages with undeclared crawlers that impersonated Chrome on macOS, rotated IP addresses across several networks, and generated 3 to 6 million requests a day on top of the declared bot's 20 to 25 million.

The test that made the post famous: Cloudflare created brand-new, unpublished domains with disallow-all robots.txt files and blocking rules. Content from those domains still turned up in Perplexity's answers. Perplexity disputed Cloudflare's account and said the traffic came from user-requested fetches and a third-party service, not a stealth crawler.

For comparison, OpenAI's and Anthropic's declared bots respect robots.txt. The practical lesson is not that robots.txt is useless. It is that robots.txt is a request, not a lock, and the only bots you can reliably keep out are the polite ones.

If your own domain is the only place you exist, blocking the search crawlers does not protect you. It just removes you.

The obvious response: block them all. Add GPTBot to robots.txt. Take back control. For most small sites, that is the wrong move.

Blocking is common. GPTBot is the most blocked AI bot on the web, disallowed by 5.89% of roughly 140 million sites (Ahrefs, 2025), and 79% of large news publishers block AI training bots. But publishers are a special case: they are cited all over the web through other people's pages. Around 85% of AI citations point at sites other than the brand's own (AirOps, 2026). If you are a SaaS company or an agency client, your own site is usually the only way an AI engine can cite you.

The useful distinction is purpose. Cloudflare now sorts AI bots into three tiers (search, agent and training) and lets you allow, block or charge each one separately. Blocking training bots is a legitimate choice. Blocking search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) takes you out of AI answers.

Two traps catch people. First, your CDN can block AI bots even when robots.txt allows them: Cloudflare's managed AI-bot setting returns errors at the edge before your robots.txt is ever read. Second, from 15 September 2026, domains newly joining Cloudflare get training and agent bots blocked on pages that show ads, while search bots stay allowed. Cloudflare applies the strictest rule to bots with several purposes, and Googlebot crawls for both search and training, so a heavy-handed training block can cost you Google AI Overviews and ordinary rankings too.

Being readable is the entry ticket. Here is what decides whether you get picked once you are in the room. The practice has a name: generative engine optimization, or GEO.

Access comes first. In a 2026 meta-analysis of 54 studies, crawler and URL accessibility was the highest-rated citation factor, 9.5 out of 10 (Zyppy, 2026). No amount of good copy helps a page the bot cannot fetch or read.

Then brand. Across 75,000 brands, mentions of a brand on other sites correlated with AI visibility at r=0.664, about three times the correlation of backlinks at r=0.218 (Ahrefs, 2026). AI systems triangulate: if you only exist on your own domain, you are a single-source claim.

Then the content itself. In the Princeton GEO study, adding statistics lifted visibility by about 41%, adding quotations by about 28%, and citing sources by up to 115% for lower-ranked pages (Aggarwal et al., KDD 2024). Freshness matters too: 83% of citations for commercial queries came from pages updated in the past 12 months (AirOps, 2026). And answer early: 44.2% of citation extractions come from the first 30% of a page's text (AirOps, 548,000 pages).

Format still matters. Comparison and "best of" content is one of the most cited formats in B2B, though estimates of its share vary widely between studies (Goodie, 2025). Write self-contained paragraphs of roughly 40 to 60 words, one question answered per paragraph, so an engine can lift a passage without the rest of the page.

All of these are signals that make a citation more likely. None of them tells you whether you were actually cited. The only way to know is to ask the engines your buyers' real questions and read back which pages they cite. WhyIQ AI Radar does exactly that every week across ChatGPT, Perplexity, Claude, Gemini and Google AI, so you can see whether a fix moved anything.

None of this is complicated. It is just not what traditional SEO advice covers.

Your schema can wait. Making your actual content visible to the bots that visit thousands of times per referral cannot.