Free tool
AI crawler checker
Paste your robots.txt to see, for each of 17 AI crawlers, whether it may read your homepage and one more path, and which rule decides. They are grouped by what they are for: collecting training data, search and AI answers, or fetching a page when someone asks an assistant about it.
Every crawler name and purpose comes from its maker's own documentation, checked October 19, 2026. The checker runs in your browser: nothing is fetched, sent or saved.
Check your robots.txt for AI crawlers
4 of 17 crawlers are blocked from the homepage: 4 of 7 training and dataset crawlers, 0 of 5 search crawlers and 0 of 5 user-request fetchers.
Checking the homepage and /account/orders.
AI training and datasets
4 of 7 blocked
Search and AI answers
0 of 5 blocked
Fetching on a user's request
0 of 5 blocked
AI training and datasets
Collect pages, or decide whether pages already crawled may be used, to train AI models or build datasets.
GPTBot
OpenAI
HomepageBlockedDisallow: /line 6(its own group, line 2)Your pathBlockedDisallow: /line 6(its own group, line 2)ClaudeBot
Anthropic
HomepageBlockedDisallow: /line 6(its own group, line 2)Your pathBlockedDisallow: /line 6(its own group, line 2)Google-Extended
Google
HomepageBlockedDisallow: /line 6(its own group, line 2)Your pathBlockedDisallow: /line 6(its own group, line 2)A robots.txt setting, not a crawler: it decides how the pages Google's crawlers fetch may be used.
Applebot-Extended
Apple
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)A robots.txt setting, not a crawler: it decides how the pages Apple's crawlers fetch may be used.
Meta-ExternalAgent
Meta
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)Amazonbot
Amazon
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)CCBot
Common Crawl
HomepageBlockedDisallow: /line 6(its own group, line 2)Your pathBlockedDisallow: /line 6(its own group, line 2)
Search and AI answers
Index pages so a search engine or AI assistant can show and link them in its results and answers.
Googlebot
Google
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)OAI-SearchBot
OpenAI
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)Claude-SearchBot
Anthropic
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)PerplexityBot
Perplexity
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)Meta-WebIndexer
Meta
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)
Fetching on a user's request
Open a page when someone asks an assistant something. Several of their makers say robots.txt may not apply to these visits.
ChatGPT-User
OpenAI
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)May not follow robots.txt: OpenAI says robots.txt rules may not apply to it, because a user started the visit.
Claude-User
Anthropic
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)Perplexity-User
Perplexity
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)May not follow robots.txt: Perplexity says this fetcher generally ignores robots.txt rules, because a user asked for the page.
Meta-ExternalFetcher
Meta
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)May not follow robots.txt: Meta says it may bypass robots.txt, because a user asked for the fetch.
Amzn-User
Amazon
HomepageAllowedNo rule matches(the * group, line 8)Your pathBlockedDisallow: /account/line 9(the * group, line 8)May not follow robots.txt: Amazon says it may not follow all robots.txt rules, because a user can start these visits.
The example file blocks four training crawlers and keeps everyone out of /account/. Replace it with your own; the result updates as you type.
What each AI crawler does
Most of these companies run separate crawlers for separate jobs, each with its own robots.txt name, and blocking one leaves the others as they were. Two of the names, Google-Extended and Applebot-Extended, aren't crawlers at all: they only decide how pages already crawled may be used.
AI training and datasets
Collect pages, or decide whether pages already crawled may be used, to train AI models or build datasets.
GPTBot
OpenAI
Crawls pages that may be used to train OpenAI's generative AI models.
If you block it: OpenAI reads a Disallow as a request not to use your content for training. ChatGPT search uses OAI-SearchBot instead, and OpenAI says each setting is independent.
Follows robots.txtOpenAI's documentation
ClaudeBot
Anthropic
Collects web content that could be used to train Anthropic's AI models.
If you block it: Anthropic says that signals your future content should be excluded from its training data. Claude's search and user requests use their own tokens.
Follows robots.txtAnthropic's documentation
Google-Extended
Google
Doesn't crawl: Google's usual crawlers fetch your pages, and this token decides whether that content may train future Gemini models and ground answers in Gemini Apps and Grounding with Google Search on Vertex AI.
If you block it: Your content isn't used for that training and grounding. Google says it doesn't affect inclusion in Google Search and isn't a ranking signal, and AI Overviews and AI Mode are part of Search, which Googlebot's rules control.
A robots.txt setting, not a crawlerGoogle's documentation
Applebot-Extended
Apple
Doesn't crawl: it decides whether pages Applebot already crawls may train Apple's generative AI models for Apple Intelligence, Services and Developer Tools.
If you block it: Apple doesn't use your pages for that training. Apple says they can still appear in search results in Spotlight, Siri and Safari.
A robots.txt setting, not a crawlerApple's documentation
Meta-ExternalAgent
Meta
Crawls the web for uses such as training Meta's foundation AI models or improving its products by indexing content directly.
If you block it: This crawler stops reading the blocked pages. Meta-WebIndexer is the one Meta says helps it cite and link your content in Meta AI's answers.
Follows robots.txtMeta's documentation
Amazonbot
Amazon
Crawls to improve Amazon's products and services. Amazon says the data may be used to train Amazon AI models.
If you block it: Amazonbot stops reading the blocked pages, for both uses.
Follows robots.txtAmazon's documentation
CCBot
Common Crawl
Builds Common Crawl's free, open repository of web crawl data, which anyone can download for research and analysis.
If you block it: Common Crawl stops collecting the blocked pages in its crawls. Its FAQ also mentions an opt-out registry.
Follows robots.txtCommon Crawl's documentation
Search and AI answers
Index pages so a search engine or AI assistant can show and link them in its results and answers.
Googlebot
Google
Crawls pages for Google Search. Google says AI is built into Search, so AI Overviews and AI Mode draw on what Googlebot crawls.
If you block it: Google stops crawling the blocked pages. Their address can still show in results, without a description, and a page needs to be indexed with a snippet to be a supporting link in AI Overviews or AI Mode.
Follows robots.txtGoogle's documentation
OAI-SearchBot
OpenAI
Finds pages to show and link in ChatGPT's search features.
If you block it: OpenAI says sites opted out of OAI-SearchBot aren't shown in ChatGPT search answers, though they can still appear as navigational links.
Follows robots.txtOpenAI's documentation
Claude-SearchBot
Anthropic
Indexes pages to make the search results in Claude's answers more relevant and accurate.
If you block it: Anthropic says your content then isn't indexed for search, which can reduce your visibility in its search results.
Follows robots.txtAnthropic's documentation
PerplexityBot
Perplexity
Finds and links websites in Perplexity's search results. Perplexity says it isn't used to collect content for training AI models.
If you block it: Perplexity recommends allowing it to make sure your site appears in its search results.
Follows robots.txtPerplexity's documentation
Meta-WebIndexer
Meta
Indexes pages to improve the search results in Meta AI.
If you block it: Meta says allowing it helps Meta AI cite and link your content in its answers, so blocking it gives that up.
Follows robots.txtMeta's documentation
Fetching on a user's request
Open a page when someone asks an assistant something. Several of their makers say robots.txt may not apply to these visits.
ChatGPT-User
OpenAI
Opens pages when someone asks ChatGPT or a custom GPT something. It doesn't crawl on its own or decide what appears in ChatGPT search.
If you block it: OpenAI says robots.txt rules may not apply to it, because a user started the visit.
May not follow robots.txtOpenAI's documentation
Claude-User
Anthropic
Opens pages when someone asks Claude a question.
If you block it: Anthropic says blocking it stops Claude fetching your content for a user's question, which can reduce your visibility when people search through Claude.
Follows robots.txtAnthropic's documentation
Perplexity-User
Perplexity
Opens a page to answer someone's question in Perplexity and links it in the answer. It isn't used for crawling or training.
If you block it: Perplexity says this fetcher generally ignores robots.txt rules, because a user asked for the page.
May not follow robots.txtPerplexity's documentation
Meta-ExternalFetcher
Meta
Fetches individual links at a user's request, for features such as AI that navigates websites to complete tasks for users.
If you block it: Meta says it may bypass robots.txt, because a user asked for the fetch.
May not follow robots.txtMeta's documentation
Amzn-User
Amazon
Opens pages for actions people start, such as Alexa answering a question that needs current information. It isn't used for training.
If you block it: Amazon says it may not follow all robots.txt rules, because a user can start these visits.
May not follow robots.txtAmazon's documentation
Google-Extended and AI Overviews
Google-Extended is the name most often misread. It has no crawler of its own: Google's usual crawlers fetch your pages, and the token decides whether that content may train future Gemini models and ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal.
AI Overviews and AI Mode are different. Google says AI is built into Search, so the robots.txt control for them is Googlebot, the same crawler as your ordinary results. To be a supporting link in them, a page must be indexed and eligible to show in Search with a snippet. Blocking Googlebot to stay out of AI Overviews would also strip your pages' descriptions from ordinary results. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex instead.
Want to be cited in those answers rather than left out? See how BlogAmplify tracks Google AI Overviews and how to rank in ChatGPT and get your business cited.
How to block or allow AI crawlers
Each rule goes in your robots.txt at the top level of your site. Here are two common setups; paste either into the checker above to see the result.
Block AI training, stay in search and AI answers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /One group with several User-agent lines: every crawler named follows its rules. Your * group still applies to everything else.
Keep one crawler out of one folder
User-agent: GPTBot
Disallow: /members/
Disallow: /cart
User-agent: *
Disallow: /cartA crawler with its own group ignores the * group, so the rules you still want it to follow (here /cart) are repeated in its group.
- Put each crawler's name on its own User-agent line. Lines in a row share the rules under them.
- A crawler with its own group ignores your * group, so repeat in its group the rules you still want it to follow.
- Add the rules on every host: Anthropic asks site owners to do it for each subdomain they want to opt out.
- Don't rely on blocking IP addresses: Anthropic says that may not work as an opt-out, because it gets in the way of its bots reading your robots.txt.
- Allow time: Google generally caches robots.txt for up to 24 hours, and OpenAI and Perplexity say changes can take about as long.
Common mistakes
- Blocking search crawlers when you meant training. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer are how those assistants find and link pages. Blocking GPTBot or ClaudeBot doesn't touch them, and blocking them can cost you citations.
- Expecting Google-Extended to remove you from AI Overviews. It doesn't; see above.
- Expecting robots.txt to stop every visit. OpenAI, Perplexity, Meta and Amazon say their user-request fetchers may not follow it, because a person asked for the page.
- Treating robots.txt as a lock. RFC 9309 says its rules are not a form of access authorization, and Google says it is up to each crawler to obey them. Pages that must stay private need a login.
- A forgotten block for everyone. Disallow: / under User-agent: * blocks every AI crawler without a group of its own, and Googlebot too unless it has one.
What this checker doesn't do
- It doesn't fetch anything. It reads the text you paste, so it can't see a firewall or CDN rule that blocks crawlers before they reach robots.txt, or what your server sends.
- It lists only crawlers whose makers document them. Bytespider is left out because we found no page from its operator describing it, and Amazon's Amzn-SearchBot because, when robots.txt doesn't name it, it follows the rules you give other search crawlers, which a file alone can't settle.
- It reads files as Google does. Group choice and matching follow RFC 9309 and Google's documentation, the same as our robots.txt tester, where you can test any crawler name and see every matching rule.
Frequently asked questions
How do I block AI crawlers from training on my site?
Add a group with Disallow: / for the training crawlers, each on its own User-agent line: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot and CCBot. Search crawlers such as OAI-SearchBot and PerplexityBot are separate tokens, so they keep reading your site unless you block them too.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended decides whether your content trains Gemini models and grounds answers in Gemini Apps and Vertex AI, and that it doesn't affect your inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, and the robots.txt control for them is Googlebot, the crawler behind your ordinary results.
Will blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT's search features, and says each setting is independent. You can block GPTBot and still be shown and linked in ChatGPT search answers, as long as OAI-SearchBot may read your pages.
Can robots.txt stop an assistant opening my page when a user asks?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta and Amazon say much the same of their user-request fetchers. Anthropic says its bots, Claude-User included, follow robots.txt. The file is a request: RFC 9309 says it is not access authorization.
Should I block AI crawlers?
It's your call, and the crawlers do different jobs. Training crawlers collect pages for AI models. Search and answer crawlers are how assistants such as ChatGPT search, Claude and Perplexity find and link pages, so blocking them can cost you visibility in their answers. You can block one kind and allow the other.
Does this checker read my live robots.txt?
No. You paste the file, and the checker works it out in your browser without fetching anything or sending it to our server. Changes to your live file take a while to count: Google generally caches robots.txt for up to 24 hours, and OpenAI and Perplexity say changes can take about as long.
Sources
- Overview of OpenAI Crawlers, OpenAI. Checked October 19, 2026.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic. Last updated April 7, 2026. Checked October 19, 2026.
- Perplexity Crawlers, Perplexity. Checked October 19, 2026.
- List of Google's common crawlers, Google Search Central. Last updated October 15, 2026. Checked October 19, 2026.
- AI features and your website, Google Search Central. Last updated December 10, 2025. Checked October 19, 2026.
- About Applebot, Apple. Published September 4, 2026. Checked October 19, 2026.
- Meta Web Crawlers, Meta. Checked October 19, 2026.
- About AmazonBot, Amazon. Checked October 19, 2026.
- CCBot, Common Crawl. Checked October 19, 2026.
- Frequently asked questions, Common Crawl. Checked October 19, 2026.
- Introduction to robots.txt, Google Search Central. Last updated December 10, 2025. Checked October 19, 2026.
- RFC 9309: Robots Exclusion Protocol, IETF. Published September 2022. Checked October 19, 2026.
Google's robots.txt rules themselves: How Google interprets the robots.txt specification.
See what AI assistants say about your business
robots.txt decides what AI crawlers may read. AI visibility tracking shows what the assistants then say: on Growth + AI Visibility, BlogAmplify asks ChatGPT, Gemini, Perplexity, Google AI Overviews and Google AI Mode up to 50 questions your customers ask, every day, and records whether each answer names your business, links your site or names competitors instead. Questions where you are left out become goals for your next content plan. How AI visibility tracking works.
Growth + AI Visibility: Card required. Free for 3 days, then $199/mo. Cancel before then and pay nothing.