A growing number of websites have done the responsible thing: opened robots.txt to the AI crawlers they want, checked the syntax, and moved on. Some of those sites are still invisible to the systems they invited in, because permission and access are different gates, and a firewall, CDN, or bot filter can turn away a crawler robots.txt explicitly welcomed.

OpenAI now says this plainly, listing robots.txt alongside WAF, CDN, bot mitigation, authentication, and rate limiting in its crawler-access guidance, and telling publishers separately not to block OAI-SearchBot if they want their content discovered and surfaced in ChatGPT search. The rule this article is built on fits in seven words: robots.txt expresses intent, server logs reveal reality.

Key Takeaways

What Are the Three Gates a Crawler Has to Pass?

Permission, access, and identity, in that order, and most advice only covers the first. Robots.txt is the guest list: it states whether a named crawler is permitted to fetch your pages. Well-behaved crawlers read it and comply. It is a declaration of your intent, and nothing more.

Your firewall, CDN, or bot-management layer is the bouncer, and robots.txt permission does not guarantee that it will let the crawler through. It decides per request, based on behaviour, reputation, and rules, whether to return your page, a 403 Forbidden, a 429 Too Many Requests, a CAPTCHA, or a JavaScript challenge. A crawler that cannot execute JavaScript or solve a CAPTCHA may receive the challenge instead of the page content it came to fetch.

Bot verification is the ID check, and it cuts the other way. Anyone can send a request claiming to be Googlebot or OAI-SearchBot, so a user-agent name proves nothing on its own. Google recommends verifying crawler requests against DNS or its published IP ranges, and Bing warns explicitly that user-agent strings are easily spoofed and provides its own verification methods. A site can be public, indexable in theory, and explicitly open to a crawler in robots.txt, yet inaccessible in practice at gate two, or open to impostors at gate three.

What Does OpenAI Actually Say About This?

That robots.txt is necessary and nowhere near sufficient. OpenAI’s crawler-access guidance for advertisers, written for ChatGPT ad landing pages, tells site owners to make pages crawlable by fixing “robots.txt, WAF, CDN, bot mitigation, authentication, and rate-limiting blocks,” with robots.txt first and five infrastructure layers after it. The same guidance tells operators to inspect web-protection and firewall configuration and points them to OpenAI’s published SearchBot IP list. Its general publisher documentation makes the simpler point for everyone else: do not block OAI-SearchBot if you want your content discovered and surfaced in ChatGPT search.

The diagnostic advice is specific. If you suspect blocking, review HTTP response codes, especially 429 Too Many Requests, along with firewall and CDN logs, bot-mitigation events, throttling rules, and traffic analytics around the time the crawler attempted access. That is a company telling you where its crawler actually fails, and it is not only in your robots.txt file.

One detail from the same guidance deserves quoting for what it reveals: “Do not rely on a manual bypass.” A one-off exception that lets a crawler through today gets wiped in the next security update. OpenAI wants the underlying configuration fixed, not patched around.

Which Crawlers Are You Actually Deciding About?

At least three different kinds, for OpenAI alone, and conflating them causes accidental blocks. GPTBot is used for potential model training. OAI-SearchBot supports discovery and surfacing in ChatGPT search. User-action traffic such as ChatGPT-User fetches pages in response to a person’s request. Blocking GPTBot therefore does not, by itself, opt a site out of ChatGPT search citations.

This is why the intent question comes before the audit. Plenty of site owners have decided, reasonably, that they do not want their content used for model training, and block GPTBot accordingly. That is a legitimate choice, and this article is not here to argue with it. The failure this article addresses is the site that meant to allow retrieval and blocked it anyway, or the site that blocked training and believed it had blocked citation, or the reverse.

The recurring configuration mistakes are easy to make: a blanket User-agent: * Disallow: / added in the belief it only blocks AI, when it blocks compliant search crawlers too; blocking GPTBot and forgetting OAI-SearchBot; or an edge rule returning 429s to a crawler that robots.txt welcomes.

Why Is the CDN Often the Culprit?

Because CDNs default to suspicion, and AI crawlers can look suspicious. In July 2025, Cloudflare changed the default for new domains so AI training crawlers were blocked unless domain owners chose otherwise, a defensible security posture and a policy choice.

From September 15, 2026, Cloudflare’s new-domain defaults become more granular: Search remains allowed, while Training and Agent bots are blocked on pages displaying ads, with site owners able to configure each category separately. Cloudflare also notes that mixed-purpose crawlers combining Search and Training will be caught by Training-blocking configurations. The point is that crawler purpose now matters. Allowing a crawler is an action you take, not a state you should assume.

On Cloudflare, first confirm that the OpenAI crawler is being correctly identified, then review the site’s AI bot, WAF, and bot-management policies to ensure verified OpenAI traffic is actually allowed. Verified identity and access policy are separate controls, which is the three-gate argument in miniature. Equivalent controls exist on other providers, and the resulting 403 and 429 rates for crawler traffic are numbers worth monitoring over time.

The part worth internalising is that this can be hard to see in the tools most site owners check. Search Console may expose a crawl failure, but it generally will not identify the exact CDN or firewall rule that challenged the crawler. Your analytics will not show a visit that never rendered. The edge logs know, which is why the rule is that server logs reveal reality.

Why Is Identity Becoming the Interesting Gate?

Because both sides of the transaction are moving from names to proof. On the site side, verification stops impostors: bots that spoof Googlebot’s user-agent to bypass rules built for Googlebot. Both Google and Bing publish methods for confirming a request really came from them, through reverse DNS or published IP ranges.

On the automated-agent side, the newer move is cryptographic, and it is a different mechanism from the crawler traffic above. ChatGPT’s cloud browser signs its outbound requests under Web Bot Auth using the HTTP Message Signatures standard, RFC 9421, with public keys published at a well-known URL so a CDN or firewall can validate them. Cloudflare treats signed agents as verified bots, and other bot-management vendors can recognise the same agent through their own verified-bot systems.

That supplements the brittle old method of maintaining allowlists of IP addresses that rotate with a key you can check. It also points at where this is going: a web where legitimate agents prove who they are and sites decide access based on verified identity rather than guesswork. Technical visibility is becoming an identity and infrastructure problem, not merely a content problem.

What Does the Four-Part Audit Look Like?

Four questions, in order, each with a concrete check.

First, does robots.txt grant permission? Open it and read it as the crawler would, looking for the specific names — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bingbot — and confirm each one you want is not disallowed. This is the gate everyone checks, and the only one most people check.

Second, does the edge actually return the page? As an initial test, request a key page using the crawler’s user-agent from outside your network and inspect the status code and response body. A 403, a 429, or a redirect to a challenge means the bouncer overruled the guest list. This tests how your edge treats that user-agent, but it does not prove the genuine crawler has access, since you have just spoofed one yourself. Confirm that separately in CDN, firewall, or server logs using verified crawler traffic or published crawler IP ranges where available.

Third, does the response contain the real page? A 200 can still deliver a JavaScript challenge, a CAPTCHA shell, or an empty template. Read the body of what came back and search it for a sentence of your actual content, the same test we use for extractability in our crawler test article.

Fourth, can the request be verified as a genuine crawler without trusting its user-agent? For inbound traffic, verify the major crawlers against their published DNS or IP methods before granting them exceptions, so your allow rules do not become a spoofing invitation. For signed agents, let your CDN validate the signature. Permission, access, content, identity: pass all four and the crawler is actually in.

Frequently Asked Questions

Q: Can a firewall block AI crawlers even if robots.txt allows them?
A: Yes. Robots.txt states permission, but firewalls, CDNs, and bot-management tools decide access per request and can return 403, 429, CAPTCHAs, or JavaScript challenges regardless of robots.txt. OpenAI’s own guidance lists WAF, CDN, bot mitigation, authentication, and rate limiting alongside robots.txt as things that can block its crawlers.

Q: How do I check whether AI crawlers are actually reaching my website?
A: Review your server, CDN, or firewall logs for the crawler user-agents you care about and count 403 and 429 responses and challenge pages. OpenAI recommends checking response codes, especially 429s, plus bot-mitigation events and throttling rules around the time of crawl attempts. Search-platform tools may show a crawl problem, but edge logs identify where infrastructure rejected the request.

Q: Why does Cloudflare block some AI crawlers?
A: Cloudflare changed its default for new domains in July 2025 to block AI training crawlers. From September 15, 2026, its new-domain defaults keep Search allowed while blocking Training and Agent bots on pages displaying ads, and its bot-management features can block automated traffic according to the site owner’s configured policy. Confirm the crawler is correctly identified, then check that your AI bot, WAF, and bot-management policies match your intent.

Q: What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
A: GPTBot is used for potential model training, OAI-SearchBot supports discovery and surfacing in ChatGPT search, and ChatGPT-User represents user-action fetching. Blocking GPTBot does not, by itself, opt a site out of ChatGPT search citations, so decide about training and retrieval separately.

Q: How do I verify that a crawler is genuine and not spoofing its user-agent?
A: Google and Bing both publish verification methods using reverse DNS lookups or published IP ranges, and both warn that user-agent strings can be faked. Newer agents, including ChatGPT’s cloud browser, sign requests under the HTTP Message Signatures standard with public keys at a well-known URL, and compatible CDNs can validate those signatures.

Q: Should I allow all AI crawlers?
A: Not necessarily. Blocking training crawlers is a legitimate choice many publishers make deliberately. The goal of an access audit is to make sure your configuration matches your decision, so crawlers you meant to allow are not blocked by accident and crawlers you meant to block are not slipping through under a spoofed name.

Most sites that believe they are open to AI have checked one gate out of three and stopped. The guest list is fine. The bouncer has not been asked. Robots.txt expresses intent. Server logs reveal reality, and without access, citation never gets the chance to happen.

AI Visibility Studio helps websites structure content so AI systems can find it, understand it, cite it, and actually use it when generating answers. aivisibilitystudio.com

References