What is the difference between GPTBot and OAI-SearchBot?
OAI-SearchBot supports ChatGPT search discovery, while GPTBot relates to potential model improvement. Publishers can set separate robots policies for each crawler.
Written for publishers choosing crawler policies for ChatGPT search visibility and model-training preferences.
Key facts
- OpenAI describes OAI-SearchBot as supporting links and answers in ChatGPT search.
- OpenAI describes GPTBot as a crawler used for content that may help improve generative AI models.
- robots.txt can define distinct user-agent groups for the two crawlers.
A useful rule: make each important claim understandable and verifiable without requiring the reader to reconstruct your meaning from the rest of the page.
What job does OAI-SearchBot perform?
OAI-SearchBot is the crawler OpenAI identifies for surfacing websites in ChatGPT search results and answers. If a publisher wants public pages considered for that discovery experience, the crawler needs robots permission and network access. Permission does not guarantee crawling, inclusion, ranking, or citation. The page also needs useful public content, stable canonical identity, and fetchable responses. Treat the crawler as a search-discovery agent and document the policy alongside other search bots rather than grouping every OpenAI request under one broad allow or deny rule.
- Use the exact documented user-agent token.
- Allow only public paths intended for discovery.
- Verify access beyond robots.txt at the CDN and origin.
What job does GPTBot perform?
OpenAI describes GPTBot separately in the context of content that may be used to improve generative AI models. That purpose is distinct from current search discovery. A publisher can therefore make a separate policy choice based on rights, licensing, governance, and business goals. Do not tell stakeholders that blocking GPTBot necessarily removes pages from ChatGPT search when OpenAI documents OAI-SearchBot for that function. Likewise, allowing GPTBot should not be presented as a citation strategy. Review official documentation because crawler roles and controls can change.
- Route training-use decisions through the appropriate policy owner.
- Do not conflate model improvement with live search retrieval.
- Review official crawler descriptions on a schedule.
Sources: 1
How can robots.txt express different choices?
Create separate user-agent groups and set paths according to the intended policy. For example, a site may allow OAI-SearchBot on public articles while disallowing account, search-result, and internal API paths, and independently disallow GPTBot. Keep the default group clear so later rules do not create accidental conflicts. Test the public file after deployment and preserve comments in the source configuration to explain business intent. Remember that robots rules are public preferences, not access control; sensitive data must remain behind authentication regardless of crawler policy.
- Use separate groups rather than one generic OpenAI block.
- Protect private content with authentication.
- Version and review the robots policy like code.
Sources: 1
What about ChatGPT-User?
OpenAI also documents user-triggered access separately. A request initiated by a user can have different expectations from automated crawling. Read the current publisher FAQ for the role and available controls rather than inferring it from the name. Decide how user-requested fetching interacts with paywalls, terms, and security. Network teams should avoid a single exception that grants all similarly named agents identical access. Keep an inventory of every allowed automated identity, its purpose, validation method, and permitted paths.
- Document user-triggered access as its own request class.
- Apply normal authentication and authorization boundaries.
- Avoid wildcard user-agent exceptions.
How should policy success be measured?
For OAI-SearchBot, monitor verified crawl activity, public fetch success, ChatGPT referral traffic, and observed citations for a fixed query set. For GPTBot, success is compliance with the organization's stated model-improvement preference, not traffic. Keep those reports separate. Annotate policy changes so crawl or referral shifts have context. If logs show no requests, confirm WAF handling before assuming lack of demand. A clear crawler matrix prevents teams from promising outcomes that the robots file cannot guarantee.
- Use different success metrics for discovery and governance.
- Observe verified requests rather than spoofed headers.
- Record deployment dates and responsible owners.
Put it to work
Find the highest-impact fix on your site.
Separate discovery, model-improvement, and user-triggered agents with documented paths and owners.
Map crawler policiesSources
- 1.OpenAI Help Center: Publishers and developers FAQChecked 2026-07-26
- 2.OpenAI Help Center: ChatGPT searchChecked 2026-07-26