Teardown: What AI Crawlers Actually See When Your Site Runs on JavaScript

Google spent a decade teaching the industry that JavaScript rendering was a solved problem: its crawler executes scripts, waits for hydration, and indexes what a browser would show. AI crawlers broke that assumption. Most retrieval systems fetch raw HTML and read what is there, and if your content only exists after JavaScript runs, what is there can be approximately nothing.

The failure, concretely

Fetch your key pages the way a bot does: no script execution, just the HTTP response. For a server-rendered site, the response contains your content. For a client-rendered single-page app, it can be a shell: an empty root div, a bundle reference, and a loading state. To an AI crawler, that page is blank. It cannot be lifted, cited, or synthesized, no matter how strong the content looks in a browser.

The insidious cases are partial. The page renders its frame server-side but loads the substance client-side: pricing tables fetched after mount, FAQ accordions hydrated from an API, review widgets injected by a third-party script. The bot sees a page that exists but says nothing, which quietly fails every liftability goal while passing every visual QA.

Why AI crawlers do not just render

Rendering at crawl scale is expensive: a headless browser per page, seconds of compute versus milliseconds for HTML parsing. Google absorbed that cost over years of infrastructure. Most AI retrieval pipelines, fetching pages at answer time or crawling under cost pressure, skip it. Some systems render sometimes; none of the major chat engines document guarantees. The safe operating assumption is the harsh one: if it is not in the HTML response, it does not exist.

The diagnostic, in fifteen minutes

• Curl your money pages (or use a no-JS fetch tool) and read the actual response. Is the pricing there? The service descriptions? The FAQ answers?
• Compare against the rendered page and list every element that exists only client-side.
• Check your key third-party surfaces too: some review widgets and headless CMS setups are invisible end to end.

The fixes, in order of invasiveness

Server-side render or statically generate the pages that carry your liftable facts; modern frameworks make this a configuration decision, and marketing pages rarely need client rendering at all. Where full SSR is impractical, inline the critical facts: prices, descriptions, and Q&A as real HTML even if a widget enhances them later. And keep structured data in the initial response, since JSON-LD injected by scripts inherits the same invisibility.

One more habit: after any redesign or framework migration, re-run the no-JS fetch on the top pages. We have watched sites lose their AI citations in a single deploy that moved content behind hydration, and the broader self-audit starts with exactly this check because it is the single most common silent failure in the discipline.