One robots.txt signal to refuse AI training: most businesses have nothing to add
Partner · Strategy, advertising and measurement
On October 30, 2026, Google, Microsoft, OpenAI and Anthropic announced that they will recognize a single signal in the robots.txt file for refusing to let a site's content be used to train their AI models. The signal affects neither rankings in Google and Bing, nor AI Overviews, nor AI Mode. For a business that wants to be found and cited by AI tools, there is nothing to add. For one that wants to refuse training, the signal needs one extra line for Google, otherwise the site also drops out of Gemini's answers.
What is changing
Until now, each AI provider had its own rule: Google-Extended at Google, GPTBot at OpenAI, ClaudeBot at Anthropic. On October 30, 2026, the four companies announced that they will recognize a shared signal, described in a draft submitted to the IETF (the body that writes internet standards) by Kevin Kelley of Anthropic.
The signal is written like a crawler, but it is not one. It goes in robots.txt with the __usage_ai-training token:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This example, taken from the draft, refuses training on the whole site except the press kit. The draft specifies that a User-agent: * group does not count as a refusal: only the token expresses a preference.
The commitments, company by company, as of October 31, 2026:
- Google will interpret the token as a refusal of
Google-Extendedwhen the latter is not written explicitly. Google is aiming for 2027; its crawler documentation says "early 2027". - Microsoft recognizes the token for training and adds a separate setting for grounding (using a page to compose an answer in Copilot or in Bing's AI summaries). The new preferences apply from February 1, 2027.
- OpenAI has announced that it will honour the signal and publish its technical details in the coming months, according to PPC Land and Search Engine Roundtable. As of November 2, 2026, OpenAI's crawler documentation does not mention it yet.
- Anthropic wrote the draft and says it has started implementing it.
Meta, Amazon and Apple have not committed.
What the signal does not affect
The token expresses a preference about how content is used, not an access rule. It blocks no crawler. It changes nothing in rankings in Google or Bing. It does not remove a site from Google's AI Overviews or AI Mode, which depend on Googlebot and on a Search Console setting. And training, as the draft defines it, does not cover reading a page at the moment of answering a question: that reading is what gets you cited by an AI tool.
The catch at Google
At Google, Google-Extended does two things at once: training Gemini models and grounding answers in Gemini apps and in Vertex AI. Since Google will read the shared token as a refusal of Google-Extended, a site that adds the token to refuse training also drops out of Gemini's answers, unless it explicitly says otherwise. Google says so itself: websites "that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Allowing Google-Extended, however, also lets Google train Gemini on your pages: Google describes no way to refuse Gemini training while staying in its answers.
A business that wants to refuse training without disappearing from Gemini must therefore write both:
User-Agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /At Microsoft, the separation is built in: refusing training does not remove a page from Copilot's answers. A separate setting, ai-grounding, exists for that, page by page. Microsoft warns that refusing it "will significantly reduce your content's visibility in AI-generated experiences."
What we recommend
| Your situation | What we recommend |
|---|---|
| You want ChatGPT, Gemini, Copilot and Claude to find and cite your business | Nothing to add. The token only serves to refuse. Check instead that your robots.txt does not already exclude Google-Extended or the reading crawlers, as we explain in AI crawlers: which should you allow or block on your website? |
| You produce content you do not want used for training (media, publishing, online courses) | Add the token, then add an explicitly allowed Google-Extended if you care about Gemini's answers, knowing that this line also lets Google train Gemini. Expect the effect to start in 2027: all four companies speak of implementation to come. |
Your site uses noarchive or nocache to keep Bing out of its AI answers | These tags will stop being read as a refusal on March 1, 2027. From December 15, 2026, Bing Webmaster Tools lets you record a temporary preference. The new ai-grounding tag must be in place before March 1. |
The shared signal simplifies refusing training; it creates no obligation. An SME that sells products or services has every interest in being read by AI tools: that is what gets it cited. The only decision to make now concerns sites that already refuse training with Google-Extended, GPTBot or ClaudeBot: their rules keep working. The shared token may replace them in 2027, provided you decide what you want at Google: refusing Gemini training there also removes your pages from its answers.
Knowing whether AI tools read and cite your site is the first step of our GEO audit.
Frequently asked questions about refusing AI training
Should I add this signal to be cited more by AI tools?
No. The __usage_ai-training token only serves to refuse training. It improves neither rankings nor citations. To be cited, AI reading crawlers need to be able to access your pages.
If I add the token, will my site disappear from ChatGPT or Gemini?
From ChatGPT, no: the token targets training, not reading a page at the moment of answering. From Gemini, yes, unless you explicitly allow Google-Extended in the same file, because Google reads the token as a refusal of Google-Extended, which also covers grounding. Allowing Google-Extended keeps you in Gemini, but also lets Google train its models.
Do my current rules for GPTBot or Google-Extended keep working?
Yes. Google writes that each company's own controls keep working alongside the shared signal. There is no rush: the shared signal will only be applied in 2027.
Does the signal remove my site from Google's AI Overviews?
No. Google's AI Overviews and AI Mode depend on Googlebot and on a Search Console setting, not on the token. The token does not change rankings in Google or Bing either.
- IETF, Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), Kevin Kelley (Anthropic), October 30, 2026. Source for the token, the example, the definition of training and the handling of the
*group. - Google Search Central, Supporting a shared AI training control for websites, John Mueller, October 30, 2026. Source for the token being read as a refusal of Google-Extended, eligibility for grounding in Gemini, the 2027 timeline and existing controls remaining in place.
- Google Search Central, Overview of Google crawlers and fetchers, accessed October 31, 2026. Source for what Google-Extended covers (training and grounding), its lack of effect on Search and the "early 2027" mention.
- Microsoft Bing, Your content, your choice: clearer preferences for training and grounding, Krishna Madhavan, October 30, 2026. Source for the ai-training and ai-grounding settings, the dates of December 15, 2026, February 1, 2027 and March 1, 2027, and the lack of effect on Bing rankings.
- PPC Land, Google, Microsoft, OpenAI and Anthropic back one AI training opt-out, October 30, 2026. Source for the commitments of OpenAI and Anthropic and the absence of Meta, Amazon and Apple.
- Search Engine Roundtable, Google Adopts New AI Training Control, Barry Schwartz, October 30, 2026. Confirmation of OpenAI's announcement.
- OpenAI, Overview of OpenAI Crawlers, accessed November 2, 2026. The page still describes refusing training through the
GPTBotrule alone.
Gabriel almost always takes your first call and carries out your audit. He builds the strategy starting from your growth goal: where to put your budget, which market to test and how to connect each lead to a real sale in your CRM. He mainly leads engagements for three goals: Optimize the profitability of your digital campaigns, Develop a new market, and Generate demand and growth. With Geneviève, he also works on organic search (SEO), AI visibility (GEO) and conversion rate optimization (CRO). The sales a Google Ads or Meta Ads campaign brings in depend on the page that receives the click. He writes mainly about marketing strategy, paid advertising and measurement.
About Falia →
Strategy
SEO
Paid advertising
Conversion
AI visibility
Web design