On this page
The AI crawler checker: eighteen crawlers, one verdict
The AI crawler checker reads your robots.txt the way eighteen crawlers read it and tells you, crawler by crawler, whether it may read your site, with the rule that decides. Then it answers the question that matters for being found: may the assistants that look for answers come in? Only then can ChatGPT, Claude or Perplexity show your page as a source. If the address check is not running yet where you are, paste the contents of your robots.txt: the tool reads it in your browser with the same rules.
Pick what you want next to it: allow everything, be cited without your text training a model, or keep all AI out. The tool lays your robots.txt against that choice, names every crawler that does not fit and gives you a robots.txt that does exactly what you chose.
Training, search and user fetches: three kinds of AI crawler
An AI crawler does one of three jobs, and the difference decides everything. A training crawler collects text for the next model. A search crawler fetches pages to answer questions and show sources with a link. A user crawler opens a page because someone asked for it in a conversation. OpenAI says it plainly: each setting is independent of the others. Blocking GPTBot does not take you out of ChatGPT. How those answers pick their sources is the field of generative engine optimization, and how it differs from classic search is in SEO vs GEO.
| Crawler | Operator | What it does | What blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Collects text that may train models | Nothing in ChatGPT search; your text stays out of the next model |
| OAI-SearchBot | OpenAI | Shows and links sites in ChatGPT search | Your place as a source in ChatGPT, at most a navigational link remains |
| ChatGPT-User | OpenAI | Opens a page when someone asks for it | Little: OpenAI says robots.txt may not apply here |
| ClaudeBot | Anthropic | Collects text that may train Claude | Nothing in Claude's answers |
| Claude-SearchBot | Anthropic | Indexes for Claude's search results | Your visibility in Claude |
| Claude-User | Anthropic | Opens a page for a user's question | Claude cannot fetch your page when asked |
| PerplexityBot | Perplexity | Shows and links sites, does not train | Your place in Perplexity's answers |
| Perplexity-User | Perplexity | Opens a page for a question | Little: it generally ignores robots.txt |
| Google-Extended | Gemini training and grounding in the Gemini app; does not crawl itself | Your place as a source in the Gemini app, not in Google Search | |
| Googlebot | Google Search, AI Overviews included | Your whole place in Google | |
| Bingbot | Microsoft | Bing, and with it Copilot | Your place in Bing and Copilot |
| Applebot | Apple | Search in Siri, Spotlight and Safari | Your place in those search features |
| Applebot-Extended | Apple | Whether Apple trains on what Applebot read; does not crawl itself | Nothing in search |
| meta-externalagent | Meta | Collects text to train Meta's AI models | Nothing in Meta AI's sources |
| meta-webindexer | Meta | Sources and citations in Meta AI | Your place as a source in Meta AI |
| meta-externalfetcher | Meta | Opens a page for a question to Meta AI | Little: Meta says it may bypass robots.txt |
| CCBot | Common Crawl | Builds an open web archive used in many training sets | Nothing in search |
| Bytespider | ByteDance | Crawler of the company behind TikTok | Unknown: we found no ByteDance documentation for it |
Two of these names do not crawl at all. Google-Extended and Applebot-Extended are flags in your robots.txt: they say what Google and Apple may do with what Googlebot and Applebot already read. Mind the Google one: besides training it also decides whether the Gemini app uses your page as a source. Google Search and the AI Overviews are not affected by it; for those, Googlebot and tags such as nosnippet decide.
Cited without being trained on
robots.txt: Cited, not trained
Looks for answers and shows sources
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
- meta-webindexer
- Googlebot
- Bingbot
- Applebot
Allow: /
Your page can appear as a source, with a link
Fetches when a user asks
- ChatGPT-User
- Claude-User
- Perplexity-User
- meta-externalfetcher
Allow: /
An assistant opens your page when someone asks for it; OpenAI, Perplexity and Meta say robots.txt may not apply here
Trains models
- GPTBot
- ClaudeBot
- Google-Extended
- Applebot-Extended
- meta-externalagent
- CCBot
- Bytespider
Your text does not train the next model. You keep your place as a source
Google-Extended and Applebot-Extended do not crawl: they say what Google and Apple may do with what Googlebot and Applebot read. Blocking Google-Extended also removes you as a source in the Gemini app, not from Google Search.
Blocking GPTBot in robots.txt: what it costs
Very little, if you do it right. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. A site that blocks GPTBot and lets OAI-SearchBot in can still be named as a source. A site that blocks OAI-SearchBot is, in OpenAI's words, not shown in ChatGPT search answers, though it can still appear as a navigational link. Allow both and OpenAI may use one visit for both purposes. A change reaches OpenAI's systems after about 24 hours; Perplexity and Meta say up to 24 hours.
The opposite happens more often than people think. A security plugin, a Disallow: / for every unknown bot or a strict firewall rule sometimes keeps every crawler out, including the search crawlers you need. The checker shows it at once, with the deciding rule. To see whether Google itself finds your pages, run the SEO checker or the check for a website not showing on Google.
robots.txt examples per choice
| Your choice | What your robots.txt says | In ChatGPT, Claude and Perplexity | In Google and Bing |
|---|---|---|---|
| Allow everything | No rules for AI crawlers, or User-agent: * with Allow: / | You can appear as a source, and your text can train a model | Visible as usual |
| Cited, not trained | Allow for five search and four user crawlers, Disallow for seven training crawlers, Google and Bing through the asterisk | You can appear as a source; GPTBot, ClaudeBot and CCBot do not read along | Visible as usual; gone as a source in the Gemini app, because Google-Extended controls that too |
| Keep all AI out | Disallow for the sixteen crawlers other than Googlebot and Bingbot, Applebot included | No longer a source, apart from a navigational link in ChatGPT | Visible as usual, AI Overviews included |
The tool turns each choice into a complete robots.txt you can copy. Every crawler gets its own name in it, because a crawler only follows the group that names it, or else the asterisk group. Rules you already have for your own folders go in the User-agent: * group.
Changing your robots.txt, start to finish
- Decide what you want: allow everything, be cited without training, or keep all AI out.
- Open yourcompany.com/robots.txt and paste it into the checker. No file, or a 404, means everything is allowed.
- Copy the robots.txt for your choice from the tool and add your own folder rules and sitemap to the asterisk group.
- Put it at the root of the host. Every subdomain, such as shop.yourcompany.com, needs its own file.
- Check your CDN or firewall settings too: they can block a crawler before it ever reads robots.txt.
- Run the checker again a day later. OpenAI, Perplexity and Meta all mention up to 24 hours.
Cloudflare and your firewall: what robots.txt does not show
If your site sits behind Cloudflare, a setting there can stop crawlers before they read your robots.txt. On 1 July 2025 Cloudflare started blocking AI crawlers by default unless they pay, and every new domain was asked whether to allow them. Since 1 July 2026 there are separate controls for search, agents and training, on every plan including Free. On 15 September 2026 it changed again: the old switch to block AI bots and the managed robots.txt are being retired, and the Block setting now also reaches Googlebot, Bingbot and Applebot. A site set to Block can therefore drop out of ordinary search too. To refuse training and keep search open, Cloudflare's setting is Disallow AI Training. New domains without ads now allow everything; with ads, training is refused by default. What applies to your site is in your Cloudflare dashboard, under the AI crawler settings.
Limits of the checker
- Blocks outside robots.txtA firewall, Cloudflare or your host can stop a crawler by name or by IP address. The checker reads the rules, not what your server answers to each crawler.
- Crawlers that skip robots.txtChatGPT-User, Perplexity-User and meta-externalfetcher fetch for a person; their makers say robots.txt may not always apply. Fetchers a user starts at Google generally ignore it too.
- Crawlers in disguiseCommon Crawl warns that other crawlers pretend to be CCBot. To be sure, compare the IP address with the list the operator publishes.
- Other paths and subdomainsThe checker reads what applies to your homepage. A rule that only closes /blog/, or a separate subdomain, needs its own check.
- Tags on the pageA noindex or nosnippet in your HTML counts as well. The SEO checker reads those.
llms.txt: what it is and what it is not
llms.txt is a proposal by Jeremy Howard from 3 September 2024: a markdown file at your root with your site's name, a short summary and lists of links to your key pages. The checker tells you whether you have one. Google says you need no special AI files for AI Overviews or AI Mode, and no provider promises that an llms.txt gets you cited more. It does no harm, but it is not a lever. What does count: search crawlers that may come in, and pages that answer the question in their first sentences.
Which setting fits your company
| Who you are | What we would choose | Why |
|---|---|---|
| Consultancy, law or accounting firm | Cited, not trained, or allow everything | You sell expertise in the room, not the text. Every mention in an answer is a chance of a call |
| Online shop selling across Europe | Allow everything, or cited, not trained | Product questions go through assistants more and more. Your product pages must be readable for whoever answers |
| Publisher, training provider or paid newsletter | Cited, not trained, or keep all AI out | Your text is the product. A citation with a link can bring a reader; a model that knows your text does not |
| Company with a customer portal | Cited, not trained, with the portal behind a login | robots.txt is not a lock: what is private belongs behind a password, not behind a Disallow |
The rules are the same in every country: robots.txt is an internet standard (RFC 9309), and the crawlers above read it the same way for a site in Belgium, the Netherlands, Germany or the UK. What differs is the language your buyers ask in. An assistant answers a German question from German pages, so a site that serves several markets needs each language readable for these crawlers, which is how we build multilingual websites. Which assistant your own team should use is a separate question: see ChatGPT vs Claude and ChatGPT vs Gemini. How much traffic already comes from AI answers is now visible in the Search Console AI report.
GPTBot, Cloudflare and llms.txt
Should I block GPTBot?
That is a choice about training, not about visibility. Block GPTBot and OpenAI does not use your text to train its next model. You can still appear as a source in ChatGPT search, as long as OAI-SearchBot may come in. OpenAI says the two settings are independent.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects text OpenAI may use for training. OAI-SearchBot fetches pages to show them in ChatGPT search, with a link. A site that blocks OAI-SearchBot is no longer shown in ChatGPT search answers, at most as a navigational link.
Does Cloudflare block AI crawlers by default?
From 1 July 2025 Cloudflare asked every new domain whether to allow AI crawlers, with blocking as the default. Since 15 September 2026 new domains without ads allow everything and domains with ads refuse only training by default. Check what your own site is set to in the Cloudflare dashboard.
Do I need an llms.txt file?
No. Google says AI Overviews need no special AI files, and no provider promises that an llms.txt gets you cited more often. It can sit next to the rest, but access for the search crawlers and pages that answer the question straight away weigh far more.
How long until ChatGPT sees a change to robots.txt?
OpenAI mentions about 24 hours. Perplexity and Meta say up to 24 hours. The robots.txt standard lets a crawler use a cached copy for at most 24 hours, unless the file cannot be reached.
Do you store my address or my robots.txt?
No. A pasted robots.txt is read in your browser and goes nowhere. An address goes to our function, which fetches your robots.txt and answers. We only count how many checks run per day, not which sites.