Why is OAI-SearchBot blocked even when robots.txt allows it?
A CDN, WAF, bot manager, rate limit, or origin rule can deny requests after robots permission. Diagnose the complete request path and verify with current OpenAI ranges.
Written for technical site owners who want public pages discoverable in ChatGPT search but see denied or missing crawler requests.
Key facts
- OpenAI identifies OAI-SearchBot as the crawler used to surface sites in ChatGPT search.
- robots.txt expresses crawl preference but does not override a WAF or authentication gate.
- OpenAI publishes crawler information and IP ranges that should be verified from official sources.
A useful rule: make each important claim understandable and verifiable without requiring the reader to reconstruct your meaning from the rest of the page.
What does robots.txt permission prove?
It proves only that a conforming crawler reading the file is not disallowed for the requested path. The request can still fail at DNS, a CDN, a web application firewall, a bot-management challenge, a country block, rate limiting, authentication, or the origin server. Start by fetching `robots.txt` publicly and confirming the OAI-SearchBot group has the intended rule. Then inspect edge and origin logs for the crawler request, status code, security action, hostname, path, and user agent. A missing log entry means the failure may occur before your application.
- Test robots.txt on the canonical hostname.
- Trace request handling from edge to origin.
- Record exact status codes and security rule identifiers.
Which security controls commonly cause the block?
Bot products often challenge unknown automation, block data-centre networks, require JavaScript, or apply aggressive rate thresholds. A crawler cannot complete an interactive CAPTCHA or browser challenge. Managed rules may also reject uncommon user agents before robots policy is considered. Review denied requests in the provider dashboard and identify the precise rule rather than creating a broad allow-all exception. Permit only the intended crawler identity and public paths. Keep administrative routes, account areas, and sensitive APIs protected through authentication, not crawler-name assumptions.
- Look for managed bot, challenge, reputation, and rate-limit actions.
- Scope allow rules to the documented crawler and public content.
- Never weaken authentication for crawler access.
Can a user-agent string be trusted by itself?
No. Any client can copy a user-agent string, so a security exception based only on that header can be spoofed. Use OpenAI's current official crawler documentation and published IP information, and follow your edge provider's method for validated bot identity when available. Do not paste a static third-party IP list into permanent rules without an update process. Log the verification result and review changes regularly. If your platform cannot validate identity safely, use a narrowly scoped rate-limited rule and monitor it rather than granting broad privileged access.
- Verify crawler identity with official current data.
- Automate or schedule updates to changing network ranges.
- Keep exceptions narrow, logged, and rate limited.
How should access be tested after the change?
Confirm `robots.txt` returns 200, the canonical article returns 200 without cookies, and important assets or rendered text are not gated. Inspect logs for successful OAI-SearchBot requests over several days rather than relying on a spoofed curl request as final proof. Check that canonical tags, noindex directives, and response content are correct for the crawler path. A successful fetch does not guarantee inclusion or citation in ChatGPT search, but it removes the network barrier. Keep a rollback path for any security exception and watch error and traffic volume.
- Verify public HTML and required assets anonymously.
- Observe genuine crawler requests in logs.
- Monitor security impact after allow-rule deployment.
What should be documented for future audits?
Record the business purpose, crawler name, official documentation URL, robots rule, WAF rule ID, scope, deployment date, owner, and review date. Include a sample successful request and the method used to verify identity. This turns an emergency exception into a maintained security control. Review the record when OpenAI changes crawler guidance, when the CDN changes bot products, or when the site moves hosts. Remove obsolete exceptions promptly. Discoverability and security can coexist when crawler access is explicit, minimal, observable, and based on authoritative identity data.
- Assign an owner and review date.
- Store evidence of identity and successful access.
- Delete stale network rules after migrations or policy changes.
Put it to work
Find the highest-impact fix on your site.
Review robots directives, public responses, preview controls, and evidence from edge logs.
Check AI crawler accessSources
- 1.OpenAI Help Center: Publishers and developers FAQChecked 2026-07-26
- 2.OpenAI Help Center: ChatGPT searchChecked 2026-07-26