Business owners: should you let AI crawlers access your content?
Two families of AI crawlers visit your site, and they call for opposite decisions. The one that collects content to train a model brings you nothing in the short term. The one that fetches your page while someone asks a question brings you customers. Blocking both with a single rule is the most common mistake.
- A training crawler collects content to feed a model. A retrieval crawler fetches a page while a user asks a question, and cites the source.
- Google writes that AI is integrated into Search and is a core part of how it works. You cannot refuse Search’s AI features without refusing Search.
- The Google-Extended token, the identifier you block or allow in your site’s configuration, covers the training of Gemini models and the grounding of their responses in the Gemini apps and on Vertex AI, not Google Search.
- To limit what is displayed from your pages in Search, Google points to separate controls: nosnippet, data-nosnippet, max-snippet and noindex.
- The robots.txt file, the public file where a site tells crawlers what they may visit, is a convention. The major operators document and respect it, others don’t. Real blocking happens server-side, on the machine that hosts the site.
On this page
What is a retrieval crawler
A retrieval crawler is an agent that fetches a page at the moment a user asks a question to an AI system, in order to answer from up-to-date content and cite the source. It differs from the training crawler, which collects content to feed a model with no link to a specific question. Blocking a retrieval crawler removes you from the answers your customers read. Blocking a training crawler limits the future use of your content with no immediate effect on your visibility.
Two families, two decisions
What follows is written for owners and marketing managers who have been offered a block on AI crawlers, or who wonder whether they should set one up. It covers the business decision, not the technical configuration.
We talk about AI crawlers as a single group, and that is where the confusion comes from: they do two different things.
The training crawler collects content to feed a model. Your page then serves to improve a system, without a link to your site appearing anywhere. The benefit to you is indirect and distant: your business eventually exists in the model’s general knowledge.
The retrieval crawler fetches your page while a user asks a question, and the answer generally cites the source. It is a visibility channel, with a link and a possible visitor at the end.
Blocking both in one move, which is what many configurations do, means protecting yourself from a use that costs nothing while withdrawing from a channel that pays.
The decision is made once and its effects surface months later. A full block put in place out of technical caution gradually removes your business from the answers your buyers consult, without any report flagging it. Nobody measures an absence.
The Google case, and what you can’t refuse
It is the most misunderstood point, and Google is explicit.
Its documentation states that AI is integrated into Search and is a core part of how it works. The robots.txt directives for Googlebot are what manage how a site is crawled for Search. Googlebot is the crawler that visits sites to feed Google Search.
There is therefore no way to stay in Google Search while refusing to let your content feed its AI features. The only refusal available is refusing Search itself, which is an option for almost nobody.
The Google-Extended token exists and its scope is precise: it covers the training of future Gemini models and the grounding of their responses in the Gemini apps and on Vertex AI. It is a control over other Google systems, not over Search: Google specifies that it affects neither inclusion nor ranking in its results.
A business can therefore block Google-Extended without disappearing from Search: the decision covers the training of separate systems and has no effect on the inquiries Search brings you.
What you control over display
Since a global refusal doesn’t exist, the real question is elsewhere: what can you control in what gets shown?
Google points to four controls for limiting the information displayed from your pages in Search: nosnippet, data-nosnippet, max-snippet and noindex.
They target display, not crawlers. The data-nosnippet control, in particular, lets you designate precise portions of a page that must not be reused, which is finer than a blanket block.
Mind the trade-off, though. Limiting snippets reduces what can be reused from your page, and reduces at the same time what can draw a reader in. On most business sites, restricting snippets costs more visits than it protects.
These controls keep their value on specific content: a paid knowledge base, a price you don’t want reproduced out of context, a document reserved for customers.
The robots.txt file is a convention
The nuance is technical, but its consequence is a management matter.
The robots exclusion protocol is a public standard, and Google states that its automatic crawlers respect it, relying on the corresponding specification. The other major operators also document their agents and their compliance.
But a robots.txt file technically prevents nothing. It is a request, honoured by those who choose to honour it. An operator that decides to ignore it accesses the content like any visitor.
If your goal is truly to prevent access, robots.txt is not enough: you need a server-side block or authentication. If your goal is to state a preference to the serious operators, robots.txt is the right tool.
That distinction avoids a frequent disappointment. A business that blocks through robots.txt and later finds its content showing up anyway hasn’t been betrayed: it used a preference tool believing it was using a barrier.
The real trade-off
Once the two families are distinguished, the decision comes down to three questions.
Is your content your product? A publisher, a paid database or a media outlet sells its content. For them, training represents value handed over with nothing in return, and blocking is defensible.
Does your content serve to sell something else? That is the case for nearly every business. Your content is a way to be found, not an asset you sell. Blocking it amounts to hiding from your buyers.
Do your buyers use these systems to research? If so, retrieval must stay open, whatever you choose on training.
For most of the businesses we see, the answer is the same: leave retrieval open, and settle training according to the standalone value of the content. The logic of visibility in these systems is covered in our article on AI-assisted shortlisting.
Deciding by your business model
Three profiles, three defensible positions.
The business that sells a product or a service. Manufacturer, distributor, contractor, firm. Open everything, training included. Your content has no market value of its own and every mention brings you closer to a buyer.
The publisher or media outlet. The content is the product. Blocking training is defensible; leaving retrieval open generally remains desirable since it brings readers with a link.
The business with a sensitive zone. A detailed price list, a reserved knowledge base, customer documentation. Treat the zone rather than the site: authentication on that section solves the problem better than any directive.
In all three cases, write the decision and its reason down somewhere. The block nobody owns usually comes from a robots.txt file modified three years ago by a vendor who is no longer around.
What to check on your site this week
These five checks take an hour and often reveal blocks nobody decided.
- What does your robots.txt file contain today, and who modified it last?
- Do you distinguish training crawlers from retrieval crawlers, or is everything treated the same?
- Does your content have market value of its own, or does it serve to sell something else?
- Are there zones that deserve authentication rather than a directive?
- If a block removes you from the answers your buyers consult, who will notice, and how?
The useful answer distinguishes the two families and names a decision someone owns. A hollow answer says AI is blocked as a precaution. A vendor who proposes blocking all AI crawlers without asking whether your content is your product removes you from a channel while protecting nothing sellable.
Auditing your configuration and settling this decision according to your model is part of what we cover in a paid audit.
What you keep in house: the decision to open or close, which depends on the market value of your content, and the identification of the zones that are sensitive. What can be delegated: the audit of the existing file, the distinction between agents, the configuration, the setup of a server-side block if needed, and the monitoring of mentions. A business that decides according to its business model holds a position it can explain. A business that blocks out of technical caution withdraws from answers its buyers consult, and nobody will notice for months, because an absence triggers no alarm.
What each platform makes measurable is compared in who measures what, and who measures nothing.
Deciding how your business exists in generated answers is at the heart of the Strengthen your visibility in AI answers goal.
Want to know first whether you are cited and how? See our work in answer engine visibility.
Crawler access is only one of the calls to make: the internal framework is covered in our article on the AI usage policy for SMBs.
Frequently asked questions about AI crawlers
Can you refuse Google’s AI features without leaving Search?
No. Google states that AI is integrated into Search and is a core part of how it works, and that the robots.txt directives for Googlebot manage crawling for Search. The only refusal available is refusing Search itself. You can however limit what is displayed from your pages with separate controls.
What does Google-Extended do?
This token covers the training of Gemini models and the grounding of their responses in the Gemini apps and on Vertex AI. It does not cover Google Search. You can therefore block it without disappearing from search results: it is a decision about separate systems, with no effect on the inquiries Search brings you.
Does blocking AI crawlers really protect my content?
The robots.txt file is a convention, not a barrier. The major operators document their agents and respect the protocol, others don’t. If your goal is to truly prevent access, you need a server-side block or authentication. Robots.txt states a preference to those who choose to respect it.
What do you risk by blocking everything?
Withdrawing from the answers your buyers consult, while protecting nothing of market value. For a business whose content serves to sell something else, which is nearly every case, a full block amounts to hiding from your customers. The cost only surfaces months later, because nobody measures an absence.
How do you protect a sensitive section?
With authentication on that section, not a site-wide directive. A detailed price list, a reserved knowledge base or customer documentation is protected by requiring a login. It is more effective than a blanket block and it lets the rest of your content do its job.
How often should this configuration be reviewed?
Once a year is enough in most cases, with an extra check after any redesign or change of vendor. The main risk is not how fast the field evolves, it is a file modified years ago by someone who is no longer around and whose decision nobody owns.
- Google Search Central, AI features and your website, official documentation, accessed July 2026. Source for AI being integrated into Search, the role of robots.txt directives for Googlebot and the four display-limiting controls.
- Google, Manage how Google’s crawlers and fetchers interact with your website, official documentation, accessed July 2026. Source for the scope of Google-Extended: training of Gemini models and grounding of responses in the Gemini apps and on Vertex AI.
- Google Search Central, How Google interprets the robots.txt specification, official documentation, accessed July 2026. Source for the conventional nature of the robots exclusion protocol.
Gabriel almost always takes your first call and carries out your audit. He builds the strategy starting from your growth goal: where to put your budget, which market to test and how to connect each lead to a real sale in your CRM. He mainly leads engagements for three goals: Optimize the profitability of your digital campaigns, Develop a new market, and Generate demand and growth. With Geneviève, he also works on organic search (SEO), AI visibility (GEO) and conversion rate optimization (CRO). The sales a Google Ads or Meta Ads campaign brings in depend on the page that receives the click. He writes mainly about marketing strategy, paid advertising and measurement.
About Falia →
SEO
Paid advertising
Conversion
AI visibility
Web design