What is OAI-SearchBot?

OAI-SearchBot is OpenAI’s web crawler for surfacing websites in ChatGPT’s search features. Allowing it helps your pages appear in ChatGPT search answers; opting out removes your pages from those answers while they may still show as plain navigational links. Control of OAI-SearchBot (search) is independent from GPTBot (training), and from ChatGPT-User (user-initiated fetching) (https://developers.openai.com/api/docs/bots).

What OAI-SearchBot does

OpenAI states that OAI-SearchBot is used to “surface websites in search results in ChatGPT’s search features.” If a site is opted out of OAI-SearchBot, it “will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from their published IP ranges to help ensure inclusion. It can take around 24 hours after a robots.txt change for systems to adjust (https://developers.openai.com/api/docs/bots).

How it differs from GPTBot and ChatGPT-User

• OAI-SearchBot: For search only. Controls whether your content can be pulled into ChatGPT search answers (https://developers.openai.com/api/docs/bots).
• GPTBot: For training generative AI foundation models. Disallowing GPTBot indicates your content should not be used to train OpenAI’s models (https://developers.openai.com/api/docs/bots).
• ChatGPT-User: Used when a ChatGPT user or Custom GPT visits a page via a user-initiated action. It’s not an automatic crawler, robots.txt may not apply to these user-initiated fetches, and it’s not used to decide what appears in Search. Use OAI-SearchBot rules for Search inclusion/opt-outs (https://developers.openai.com/api/docs/bots).

OpenAI notes these settings are independent. You can allow OAI-SearchBot (to appear in ChatGPT search) while disallowing GPTBot (to avoid training). If both are allowed, OpenAI may use results from one crawl to avoid duplicate crawling (https://developers.openai.com/api/docs/bots).

Control search and training independently in robots.txt

The following examples show common, source-aligned patterns using the specific OpenAI user-agent names. Adjust paths as needed for your site.

Appear in ChatGPT Search, opt out of model training

# Allow OAI-SearchBot (search)
User-agent: OAI-SearchBot
Allow: /

# Disallow GPTBot (training)
User-agent: GPTBot
Disallow: /

After changes, allow ~24 hours for ChatGPT search adjustments (https://developers.openai.com/api/docs/bots).

Opt out of ChatGPT Search but leave training decisions separate

# Disallow OAI-SearchBot (search)
User-agent: OAI-SearchBot
Disallow: /

# (Optional) Keep other bots unchanged; decide about GPTBot separately

Verify a request is really from OpenAI

To reduce false positives and avoid blocking genuine access, check the markers OpenAI documents:

• User-agent string: OAI-SearchBot includes a clear identifier and version. Example (version may change):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

When fetching robots.txt, you may see a robots.txt marker to distinguish file fetches from other requests:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

Other documented agents for comparison:

# GPTBot (training)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

# ChatGPT-User (user-initiated, not an automatic crawler)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

• IP ranges: Compare the request’s source IP to OpenAI’s published JSON lists to confirm origin and allowlist if needed:
  • OAI-SearchBot IPs: https://openai.com/searchbot.json
  • GPTBot IPs: https://openai.com/gptbot.json
  • ChatGPT-User IPs: https://openai.com/chatgpt-user.json

For a broader, step-by-step process to validate crawlers without exposing private pages, use our guide: How to verify search and AI crawler access without exposing private pages.

When to expect changes

For ChatGPT search results, OpenAI indicates it can take about 24 hours after a robots.txt update for systems to adjust (https://developers.openai.com/api/docs/bots). Training-related effects from GPTBot allow/deny depend on when content is crawled and processed.

Actionable checklist

• Decide your policy: appear in ChatGPT search, allow training, both, or neither.
• Update robots.txt with OAI-SearchBot and GPTBot rules that match your policy.
• Allow OpenAI’s published IPs if you use IP-based filtering (searchbot.json, gptbot.json, chatgpt-user.json).
• Monitor server logs for the documented user-agent strings and the robots.txt marker on robots.txt fetches.
• Recheck after ~24 hours for search inclusion changes.
• Track brand presence and traffic from AI surfaces; see our audit and evidence guides:
  • How to measure AI search visibility with a ten-question audit
  • AI citations, brand mentions and visits: how to read the evidence
  • An AI search readiness checklist you can work through with evidence

FAQ

Can I allow ChatGPT search but block training?

Yes. Allow OAI-SearchBot and disallow GPTBot in robots.txt. OpenAI confirms these controls are independent (https://developers.openai.com/api/docs/bots).

Does robots.txt apply to ChatGPT-User?

ChatGPT-User is used for user-initiated actions and is not an automatic crawler. Because these actions are initiated by a user, robots.txt rules may not apply. It is not used to determine whether content can appear in Search (https://developers.openai.com/api/docs/bots).

How long do OAI-SearchBot changes take?

OpenAI notes it can take about 24 hours for search results to adjust after a robots.txt update (https://developers.openai.com/api/docs/bots).

Related guides

• What is AI Performance in Bing Webmaster Tools? A practical guide to citations, limits and workflow
• What is an llms.txt file?

Related guide from CheckMySchema

For the difference between page indexing and link-following directives, see Noindex vs nofollow: a clear guide for developers.