Your browser can display content missing from your server’s initial HTML. Learn how to inspect the response, identify gaps, and verify your fixes.

Your browser can display content that was missing from the HTML your server first sent. JavaScript may add the pricing table, populate the FAQ, or load a booking widget after the page arrives. A crawler that does not execute those scripts may never see that content. Vercel and MERJ’s December 2024 traffic study found no JavaScript rendering by the OpenAI, Anthropic, Meta, ByteDance, and Perplexity agents it examined, a dated and scoped finding, but a useful one. The practical question is what your server delivers before browser-side JavaScript runs.

Key Takeaways

Why Does the Browser View Differ From the Crawler View?

Because a browser keeps working after the HTML arrives. A browser can display the initial HTML and then update the page as scripts execute and data arrives. That updated document is what you see and what DevTools shows under the Elements tab.

A crawler that does not execute JavaScript cannot reproduce those browser-side updates. Information may still be available to it in embedded data or separately accessible resources, which is why the check below looks at where content sits, not only whether it is visible. Two crawlers document that they do render: Googlebot, whose rendering service can process JavaScript content as long as the scripts are not blocked, and Applebot, whose documentation says it may render your site within a browser and which the Vercel and MERJ study also identified as a rendering crawler. Google points site owners to its URL Inspection tool to see the HTML Googlebot received, which makes “what the crawler got” a documented question, not a theory. That tool is Google-specific and includes rendering diagnostics, a rendered screenshot, and loaded resources; the manual check below is narrower.

For the AI crawlers, the evidence is narrower and we should say so. OpenAI’s crawler documentation describes what GPTBot, OAI-SearchBot, and ChatGPT-User are for and publishes their IP ranges, but says nothing about rendering. Anthropic and Perplexity are the same. The Vercel and MERJ study, published December 17, 2024, reported no JavaScript rendering by the named OpenAI, Anthropic, Meta, ByteDance, and Perplexity agents it examined. That is a finding about those agents at that time, not a rule about every AI retrieval system today.

One more distinction matters. OpenAI separates search crawling, training collection, and user-triggered fetching, and Anthropic draws similar lines, so behaviour observed for GPTBot or ClaudeBot is not a claim about every way ChatGPT or Claude reaches a site. Google also describes browser agents that work from screenshots, the DOM, and the accessibility tree. This article is about crawlers that fetch pages on their own schedule; agents driving a browser are a different mechanism.

What Is Usually Missing From the Initial HTML?

Anything a script creates after the page arrives. The areas to inspect:

On structured data, the tag is not the problem. A literal script type=”application/ld+json” block delivered in the response is data, and needs no execution to be read. JavaScript that builds JSON-LD after loading does. Google documents support for JavaScript-generated structured data, including through Google Tag Manager, so the point is that markup delivered by the server removes a rendering dependency, not that tag-manager markup is unreadable.

Kajabi note: Kajabi documents that its Header Page Scripts field is included in the page head, excluding Offer checkout and upsell pages, so a literal JSON-LD block placed there should not require JavaScript execution. Verify the actual response rather than assuming it. Areas to check on Kajabi are custom code blocks and embedded third-party widgets. Squarespace, Webflow, and WordPress sites vary by theme and plugin, which is why the inspection below exists instead of a rule.

How Do You Inspect What a Crawler Receives?

Fetch the page, confirm you got the page, then look for your content in what arrived. First, capture the browser view: open the page, note a distinctive short phrase from the section you care about, and take a screenshot. Second, fetch the initial HTML from outside your network:

curl --silent --show-error --location --compressed \
  --dump-header headers.txt \
  --output raw.html \
  --write-out 'Status: %{http_code}\nFinal URL: %{url_effective}\nContent type: %{content_type}\n' \
  'https://example.com/page'
grep -Fc -- 'a distinctive short phrase' raw.html

Read the status line before anything else. Confirm the status code, final URL, and content type, then inspect the response body to verify that it contains the intended page rather than an error, login screen, or challenge, since all three can arrive with a 200. A 403, a redirect ending elsewhere, or a challenge page is useful access evidence, but it is the wrong response for diagnosing missing page content.

The grep is a screening check, not a verdict. Zero means that exact string was not found on any line: a sentence split by a formatting tag, an HTML entity, or different wording in the served version all return zero without JavaScript being involved. A match proves the string exists somewhere in the file, which could be inside an inline script rather than in page text, and -c counts matching lines, not occurrences. So when you get a hit, show the matching lines with grep -Fn -- 'phrase' raw.html and inspect the surrounding markup. For structured data, find the application/ld+json block itself and validate it, since matching one property value proves nothing about the block.

This is a view-source test, not a DevTools test. The Elements panel shows the rendered DOM after scripts have run, which is precisely the view a non-rendering crawler does not have.

A second request adding a crawler user-agent can reveal user-agent-dependent behaviour. It does not authenticate you as that crawler. Firewalls and CDNs verify by IP range and other request characteristics, so the genuine crawler may be treated differently from your test, which is the argument of our firewall article.

What Does the Fix Look Like?

Deliver the missing content in the HTML the server sends. What that means depends on where the inspection placed the gap. Content fetched from an API after load moves into the server-rendered template, or is pre-rendered, so it arrives with the page. A script-built accordion becomes a static block containing the same questions and answers, which the script can still open and close. Structured data built by JavaScript becomes a literal JSON-LD block in the response; if the block was already delivered in the response, leave it alone.

The visible design does not have to change. The difference is where the text exists: in the response before any script runs, instead of after.

How Do You Verify the Result?

Rerun the fetch, confirm the status, final URL, and content type, and inspect the body again. Screen for the same phrase with grep -Fc, then use grep -Fn to confirm the match sits in page markup, not inside a script. Find the JSON-LD block and validate it.

Then check whether a verified crawler fetched the page after the change, not just your test request. Look in server or CDN logs, using published IP ranges where available. A verified fetch with a 200 shows a fetch occurred; it does not by itself show which version was delivered or what was extracted. If response-version evidence is available, use it to establish which version was served: a strong ETag in the logged response that matches your saved post-fix response is useful, while content length alone does not identify a version and a weak ETag does not establish byte-for-byte identity.

None of this measures whether the change affects how often a site is cited. Access is necessary for that crawler to retrieve that content directly; citation is a separate observation with its own causes, and any claim about it needs its own controlled comparison over time.

Frequently Asked Questions

Q: Do AI crawlers like GPTBot and ClaudeBot execute JavaScript?
A: Vercel and MERJ’s December 2024 study reported no JavaScript rendering by the agents it examined, including GPTBot and ClaudeBot. This describes those agents during the study period, not a guarantee about every current AI retrieval system.

Q: How do I check what an AI crawler sees on my website?
A: Fetch the page with curl, following redirects and recording the status code, and confirm you received the intended page. Then search the saved HTML for a distinctive phrase as a screening check and inspect where any match sits. A zero result means that exact string was not found, which can have causes other than JavaScript, so treat it as a prompt to look closer rather than a conclusion.

Q: Does Google render JavaScript for AI Overviews and AI Mode?
A: Yes. Google states it can process content within JavaScript as long as it is not blocked, and AI features in Google Search draw on the same index. Google’s URL Inspection tool shows the HTML Googlebot received plus rendering diagnostics.

Q: Does my Kajabi site have this problem?
A: Kajabi documents that its Header Page Scripts field is included in the page head, with checkout exceptions. A literal JSON-LD block placed there should not require JavaScript execution, but verify the actual response. Check custom blocks and third-party widgets separately.

Q: Will fixing the initial HTML get my site cited by ChatGPT or Perplexity?
A: It removes one reason a crawler could not read the content. It does not by itself cause citation, and we do not claim a citation effect without measuring it.

Q: Does Google require special schema or files for AI features?
A: No. Google’s guidance says no special schema.org markup, AI text files, or new machine-readable files are needed for AI Overviews or AI Mode. Structured data remains useful for eligible rich-result types, though FAQ rich results were retired starting May 7, 2026.

Everyone tests their site by loading it. A successful load is only the starting point. Check what was delivered, what required rendering, and what you can actually verify.

AI Visibility Studio helps websites structure content so AI systems can find it, understand it, cite it, and actually use it when generating answers. aivisibilitystudio.com

References