On this page
One rule for every AI provider at once
Robots.txt AI training is getting a shared rule. On 30 October 2026 Google wrote on its blog for site owners that it plans to support a shared policy token in robots.txt. With that single line you tell every participating AI provider at once whether it may use your content to train models. OpenAI said the same day that it will honor the shared signal too.
Today it works differently. Every AI company has its own name for its crawler or token, and if you want to refuse training you write one rule per company. The new rule comes from the AIPREF working group at the IETF, the body behind the robots.txt standard itself. Google is aiming for 2027 and has not published a syntax yet. So there is nothing to change today, but it is a good moment to know what your robots.txt says.
Refusing training is not the same as leaving Search
Disallow Google-Extendedrobots.txt, today
AI Overviews and AI Modeno effect
Regular resultsno effect
Shared rule: refuse trainingrobots.txt, planned for 2027
AI Overviews and AI Modenot stated
Regular resultsnot stated
Turn off generative AISearch Console
Gemini trainingno effect
Grounding in Gemini appsno effect
Regular resultsno effect
noindexper page
Gemini trainingnot what it is for
Grounding in Gemini appsnot what it is for
Disallow Googlebotrobots.txt
Gemini trainingnot what it is for
Grounding in Gemini appsnot what it is for
no effectnot known yetstops or drops out
Per Google's documentation on 1 November 2026. The syntax of the shared rule is not fixed yet.
What Google announced
The post on the Search Central blog is short and carefully worded. These points are fixed:
- A shared token in robots.txt will let you set a broad preference for every AI provider that takes part. Google calls it a policy token.
- The definition of AI training comes from the IETF AIPREF working group. The rest of that group's vocabulary is still being discussed.
- If you refuse training through the shared token and say nothing separate about Google-Extended, Google will read it as if you had disallowed Google-Extended.
- Company-specific rules keep working. You can make one broad choice and still write an exception per provider.
- Sites that specifically allow Google-Extended stay eligible to appear in Gemini app answers, with clicks to the source.
- Google aims to support it together with other AI providers in 2027 and promises updates.
What is missing: the exact syntax, the list of participating companies and a date. Do not copy an example rule from a forum yet; until the standard exists, such a line does nothing.
What Google-Extended does today
Google-Extended has existed since 28 September 2023. It is not a separate crawler but a token: Google fetches your pages with its regular crawlers and checks the Google-Extended rule to decide whether it may use them to train future Gemini models and for grounding, the sourcing of answers in the Gemini apps and the Vertex AI API. Google's documentation says the token does not affect your inclusion in Google Search and is not a ranking signal.
That last point causes a lot of confusion. Refusing Google-Extended does not take you out of AI Overviews and AI Mode. Since 31 August 2026 there is a separate control in Search Console for that. And the Search Console control does not stop training. Two switches for two questions.
| Rule or control | Training Gemini models | Grounding in the Gemini apps | AI Overviews and AI Mode | Regular results |
|---|---|---|---|---|
| Disallow Google-Extended in robots.txt | Stops | Stops | No effect | No effect |
| Shared token: refuse training (2027) | Stops, unless you allow Google-Extended separately | Stops, unless you allow Google-Extended | Not stated | Not stated |
| Turn off Search generative AI in Search Console | No effect | No effect | You disappear from them | No effect |
| noindex on a page | Not its purpose | Not its purpose | The page disappears | The page disappears |
| Disallow Googlebot in robots.txt | Not its purpose | Not its purpose | Google no longer crawls your pages | You largely disappear from Search |
Use the AI crawler checker to see what your robots.txt says to Google-Extended and to other AI crawlers today. Many sites refuse training without knowing it, because a plugin or a theme once added the rules.
Why a shared rule is needed
If you want to refuse training today, you keep a list of crawlers and tokens from dozens of companies, and that list goes stale every month. Small sites do not do that, so in practice the theme or the host decides. The IETF AI Preferences working group is building a shared vocabulary for such preferences and the ways to attach them to your content: through robots.txt, through a fixed address on your domain (a well-known URI) and through HTTP headers. Robots.txt itself has been an IETF standard since 2022, RFC 9309; the new token builds on it.
For a business owner that means one rule, written once, that applies to every company following the standard. Companies that do not take part will not follow it. Robots.txt remains a request, not a lock.
What this does not change
The announcement is about training. Whether your pages show up in AI answers from ChatGPT, Perplexity or Google depends on other things: whether their search crawlers may read your pages, whether your page answers the question, and whether other sources mention you. Refusing training does not stop you from being cited; allowing it does not make you cited more often.
European rules are separate too. The AI Act puts transparency duties on AI providers, but your robots.txt stays the place where you state your own preference.
Allow or refuse training: the trade-off for a small business
- What AI knows about you
- AllowYour services, region and way of working can end up in future models
- RefuseThe model only knows you through what others write about you
- Answers with a source
- AllowYou stay eligible for grounding in the Gemini apps, with a click to your site
- RefuseRefuse Google-Extended and that grounding goes too
- Your own work
- AllowYour text can feed a model without payment
- RefuseYour text stays out of training by those who follow the rule
- Who gains
- AllowService businesses and local companies that want to be found
- RefusePublishers, creators and anyone who lives from their content
If you sell a service or a product, you usually want AI to know you and point to you. If your content is the product, refusing is defensible. Make the choice once, on purpose, instead of leaving it to a plugin.
Refusing training without leaving Google
Do I need to change anything now?
No. The rule has no syntax yet and Google is aiming for 2027. Your existing rules for Google-Extended and other crawlers keep working.
Will I lose rankings in Google if I refuse training?
No. Google says Google-Extended does not affect inclusion in Search and is not a ranking signal.
Does refusing training take me out of AI Overviews?
No. That is what the Search generative AI control in Search Console is for, since 31 August 2026.
Will every AI company follow the shared rule?
Only those that take part. Google and OpenAI announced it; other names are not known yet. Robots.txt is a request, not a technical block.
What if I use the shared rule and allow Google-Extended?
Then your specific rule for Google wins. You refuse training by the other participants and stay available to Google, including grounding in the Gemini apps.
What you can do now
- Open your robots.txt at yoursite.com/robots.txt and read it. If you see rules for Google-Extended or other AI crawlers, find out who added them and why.
- Decide per goal. Training is a different question from appearing in AI answers. If you want to get found in ChatGPT, at least let the search crawlers of those services in.
- Write your choice down, so a new plugin or theme does not quietly undo it. A short note next to your site is enough.
- Measure what AI does with you today. The AI report in Search Console shows your impressions in Google's AI features.
- Follow the standard. Once Google publishes the syntax, we will spell out the rule here for you to copy.
What we do with this
We build sites that AI can read and cite easily, with a clear page per service, structured data and a robots.txt that says what the owner wants. Every SEO project includes a look at the rules for AI crawlers. To learn how to appear in AI answers, read about getting found in ChatGPT or the comparison SEO vs GEO.