How to block GPTBot without blocking OAI-SearchBot

What GPTBot and OAI-SearchBot do

OpenAI says GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot is used to surface websites in ChatGPT Search. OpenAI’s documentation describes the robots.txt settings for the two crawlers as independent.

OpenAI also says that sites opting out of OAI-SearchBot will not be shown in ChatGPT Search answers, although they may still appear as navigational links. Allowing the crawler does not mean that a site will necessarily appear in answers.

ChatGPT-User is a separate agent. It may visit a page in response to a user action in ChatGPT or a Custom GPT, but it is not used to determine whether content can appear in ChatGPT Search. OpenAI notes that robots.txt rules may not apply to these user-initiated actions. For search access, its documentation identifies OAI-SearchBot as the relevant crawler.

Before editing robots.txt

Open the file for the right host

Check the robots.txt file at the root of the host serving the pages you want to control. For example, the main site’s file does not automatically govern a staging subdomain. If your pages are hosted on several hosts, check each file separately. Save a copy of the current version so you can restore the previous rules if needed.

Review the existing groups

Read every User-agent group. Look for rules for GPTBot and OAI-SearchBot, as well as general rules for *. If a group for either crawler already exists, inspect it before making changes. Do not add a duplicate group without checking the full file and how its existing rules address the paths you want to control.

Crawlers can differ in their support for and interpretation of syntax. Google’s documentation explains how robots.txt works for Google, not OpenAI. OpenAI confirms that the settings for GPTBot and OAI-SearchBot are independent, but its documentation does not describe every possible combination of directives. If the file contains several groups or rules that may conflict, do not assume that one new line cancels the others. Check the current documentation for the relevant crawler or ask a technical specialist to review the complete file.

Example: separate rules for the two crawlers

If you want to disallow GPTBot from the whole site and allow OAI-SearchBot to access all paths, the rules could look like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

In this example, Disallow: / sets a restriction on all paths for GPTBot, while Allow: / indicates access to all paths for OAI-SearchBot. This is an example of commonly used robots.txt directives, not confirmation that every crawler will interpret every combination in the same way. OpenAI confirms that the two settings are independent, but does not give an exhaustive account of how it processes every syntax combination. Check the complete file and OpenAI’s current instructions before publishing.

If you only want to restrict a section from GPTBot, specify that path instead of /. Compare it with the site’s URLs so that you do not restrict pages you intend to leave accessible. Use separate groups for GPTBot and OAI-SearchBot when you want different rules for each.

Check the published file

1. Open the public robots.txt URL. Visit the file on the host you intend to configure. Check that it shows your changes rather than an error or an older version.
2. Check the GPTBot group. Confirm that its directive matches the paths you chose to restrict. In the example, Disallow: / sets a restriction for all paths.
3. Check the OAI-SearchBot group. Make sure you have not disallowed pages you intend to leave accessible. In the example, Allow: / indicates access to all paths.
4. Review general and repeated groups. If you are unsure how several directives interact, do not assume that a new line resolves a possible conflict. Check the complete file against the relevant crawler documentation or ask a technical specialist to review the rules.
5. Keep a rollback copy. Record what you changed. If the public file does not match your intended settings, restore the copy, revise the rules and check the file on the correct host again.

OpenAI says its search systems may take about 24 hours to adjust after a robots.txt update. This is an approximate processing window, not a guarantee that the changes will be applied within that time or that your pages will appear in ChatGPT Search.

What robots.txt cannot do

Robots.txt is a public instruction for crawlers, not a data-security measure. Anyone can read the file, and its directives do not replace access controls for pages. Use a password or another access-control measure for confidential material.

Google warns that a URL disallowed from crawling may still appear in Google Search, for example if other pages link to it. This describes Google’s behaviour; do not assume that it applies in the same way to ChatGPT or other search systems. See Google’s introduction to robots.txt for details.

A rule in the published file confirms only which instructions you have made available. By itself, it does not show whether a crawler has visited a particular page.

Before you publish

• You are editing the file for the right host and have saved the original version.
• You have checked the GPTBot, OAI-SearchBot and * groups.
• GPTBot and OAI-SearchBot have separate rules that match your goal.
• The restrictions cover only the paths you chose and are not being used to protect confidential data.
• You have opened the public file after publishing and checked its contents.
• You understand that allowing OAI-SearchBot does not guarantee visibility in ChatGPT Search.

Next step

Decide which pages GPTBot should not crawl and which pages OAI-SearchBot may access. Set separate rules and check the published file on the correct host. These are instructions for crawlers, not a way to secure confidential data or a promise of visibility in ChatGPT Search.

If you are also reviewing how your business appears in AI-powered search, see SEO.LISA’s AI search optimisation service. Robots.txt is one technical consideration related to crawler access.