Is Your Website Blocking AI Search? How to Check robots.txt for ChatGPT, Perplexity, Claude and Google

Short answer: It might be, and many owners never notice. AI assistants use named crawlers, and your robots.txt file or your firewall can block them. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and Perplexity recommends allowing PerplexityBot if you want to appear in its results. The fix is usually simple: allow the search crawlers, decide separately whether you want to allow the training crawlers, and confirm your firewall isn't blocking what your robots.txt allows.

What robots.txt does (and doesn't do)

Robots.txt is a small text file at the root of your website (yourdomain.com/robots.txt). It tells crawlers which parts of your site they may visit. Google Search Central describes it as a way to manage crawler traffic, and says it is not a mechanism for keeping a page out of Google; for that you use noindex or password protection (Google Search Central, "Robots.txt Introduction and Guide").

Two points matter for AI visibility:

• Robots.txt is a request, not a lock. Google notes that respectable crawlers obey it, but it can't enforce behavior.
• One line can block a lot. A rule like User-agent: * followed by Disallow: / asks every crawler that honors it to stay off your whole site.

Search crawlers vs. training crawlers

The big AI companies now separate the crawler that powers their search answers from the crawler that collects content for training models. That means you don't have to choose between "all AI" and "no AI."

Here is what each company says in its own documentation:

• OpenAI: OAI-SearchBot is used to surface websites in ChatGPT's search features, and sites opted out of it "will not be shown in ChatGPT search answers." GPTBot is separate; disallowing it signals your content shouldn't be used to train OpenAI's foundation models. OpenAI says each setting is independent and that it can take about 24 hours for search to reflect a robots.txt change (OpenAI, "Overview of OpenAI Crawlers").
• Perplexity: PerplexityBot surfaces and links websites in Perplexity's search results and "is not used to crawl content for AI foundation models." Perplexity recommends allowing it to appear in results. Perplexity-User fetches pages when a person asks a question and, Perplexity says, generally ignores robots.txt because a user requested it (Perplexity Docs, "Perplexity Crawlers").
• Anthropic: ClaudeBot collects content that could contribute to model training. Claude-SearchBot improves search results, and Anthropic says disabling it "may reduce your site's visibility and accuracy in user search results." Disabling Claude-User can likewise reduce visibility for user-directed web search (Claude Help Center).
• Google: Googlebot covers Google Search and its features. Google-Extended is a separate token that controls whether Google may use your content for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect your inclusion in Google Search and isn't a ranking signal (Google, "Google's common crawlers").

How to check your site in 10 minutes

1. Open your robots.txt

Type yourdomain.com/robots.txt into a browser. If you get a "not found" page, you have no robots.txt, and crawlers that honor it generally treat your site as open.

2. Look for blanket blocks

Watch for User-agent: * paired with Disallow: /. That asks every compliant crawler, including search engines, to stay away.

3. Look for AI crawlers by name

Search the file for OAI-SearchBot, GPTBot, PerplexityBot, Claude-SearchBot, ClaudeBot and Google-Extended. Any of them followed by Disallow: / is blocked from your whole site.

4. Check your platform settings

Google notes that website builders such as Wix may not let you edit robots.txt directly and instead offer a search-visibility setting. Look for any "hide from search engines" or "block AI bots" toggle in your website builder or hosting dashboard.

5. Check your firewall or CDN

A firewall can block a crawler even when robots.txt allows it. In July 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay content creators (Cloudflare blog, July 1, 2025). Perplexity's own setup guide warns that Cloudflare's Bot Fight Mode can still challenge its bots even after you add an allow rule. If your site uses Cloudflare or another web application firewall, review its bot settings.

A simple robots.txt for a service business

If your goal is to be found and recommended in AI search, a common setup is to allow the search crawlers and make a deliberate choice about the training crawlers. For example:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

If you prefer that your content not be used for model training, you can add separate Disallow: / groups for GPTBot, ClaudeBot and Google-Extended. Each company says those training controls are separate from its search crawler. Keep any rules that protect private areas, such as client portals or admin pages, and test the file after every change.

What allowing crawlers won't do

Allowing crawlers makes your pages eligible to be read. It doesn't guarantee you'll be recommended. AI assistants still need clear, answer-first pages, consistent business details and trusted mentions elsewhere on the web. Our guides to what makes AI recommend a business and how to show up in Perplexity cover those next steps. Anyone who promises a guaranteed spot in AI answers is overselling; see is generative engine optimization safe.

How to tell if the fix worked

1. Wait a day, then recheck. OpenAI and Perplexity both say changes can take up to about 24 hours to take effect.
2. Check your server or CDN logs for visits from the crawler names above.
3. Ask real buying questions on ChatGPT, Perplexity, Claude and Google, and note whether your site is cited.
4. Track by assistant over time. If an assistant wasn't checked, record it as missing data, never as zero. See how to track your AI visibility.

How Visionary AI Partners approaches it

Crawler access is one of the first technical checks we run, because no amount of content will matter if AI search can't read your pages. Our AI visibility services review your site's crawl access, pages and business details, then fix what's most likely to change how AI assistants answer your customers' questions. The simplest first step is our free instant AI Visibility Score at https://visionary-ai-visibility-score.ed-moran.chatgpt.site/, which reviews your website and points out improvement opportunities. Ongoing monitoring and optimization are separate paid services; see how it works.

Frequently asked questions

Will blocking GPTBot keep me out of ChatGPT search?

Not according to OpenAI. GPTBot is the training crawler; OAI-SearchBot is the one used for ChatGPT search, and OpenAI says each setting is independent.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal.

Can robots.txt keep a page out of Google entirely?

No. Google says a blocked URL can still be indexed if other sites link to it. Use noindex or password protection instead.

Do AI assistants always obey robots.txt?

Not every fetch. OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User visits, and Perplexity says Perplexity-User generally ignores robots.txt. Their automatic search crawlers are the ones robots.txt controls.

How long after I change robots.txt will AI search notice?

OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours. Being cited still depends on your content and reputation, not crawl access alone.

I don't have a robots.txt file. Is that a problem?

Not by itself. Without one, crawlers that follow the standard generally assume they can crawl your site. Still check your firewall and website platform settings.

Sources & References

1. OpenAI, "Overview of OpenAI Crawlers": https://platform.openai.com/docs/bots
2. Perplexity Docs, "Perplexity Crawlers": https://docs.perplexity.ai/guides/bots
3. Claude Help Center, "Does Anthropic crawl data from the web, and how can site owners block the crawler?": https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
4. Google Crawling Infrastructure, "Google's common crawlers" (last updated Oct 30, 2026): https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
5. Google Search Central, "Robots.txt Introduction and Guide" (last updated Dec 10, 2025): https://developers.google.com/search/docs/crawling-indexing/robots/intro
6. Cloudflare Blog, "Content Independence Day: no AI crawl without compensation!" (July 1, 2025): https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/