Definition
What is an independent search index?
An independent index is a web index built and maintained by the company that operates it, with its own crawler, without borrowing the results of another search engine. Only a handful exist worldwide: Google, Bing and Brave Search are the known examples. A page missing from an independent index is invisible to every product that relies on it, whatever its ranking elsewhere.
How an independent index is built
An engine that resells another's results has no need for a crawler of its own. An independent index fills up through crawling: an in-house crawler visits pages, stores them and refreshes them. Brave adds a second source, the Web Discovery Project: an opt-in system, off by default, that contributes anonymized data from the Brave browser.
Independence is sometimes won in stages. On April 27, 2023, Brave removed the last share of its results that still depended on a call to Bing, about 7% according to its announcement. Since then, every web result on Brave Search comes from its own index.
Why it matters for AI visibility
An assistant that searches the web relies on one or more indexes. Anthropic has listed Brave Search on its public list of subprocessors, with the label “Web Search”, since March 2025. Claude's web search therefore relies, at least in part, on an index that is neither Google's nor Bing's. Ranking first on Google does not automatically put you in what Claude sees of your business.
At OpenAI, sworn testimony and measurements from the summer of 2026 describe an index of the company's own, which it has never publicly claimed. No public tool lets you consult it.
The robots.txt trap
Brave's crawler does not announce a distinct identity. According to Brave's help page, it aligns itself with what your site grants Googlebot: a page Googlebot cannot crawl will not be crawled by Brave either. A robots.txt file that disallows everything for unidentified agents can therefore take you out of the index that feeds Claude, and nothing tells you so.
The same page specifies that robots.txt does not prevent a page from being indexed. To remove a page from Brave's index, the noindex tag is what you need.
How to check your presence
There is no Brave console for site owners, no indexing report and no rank tracking. Checking is done by hand: type your query into Brave Search to see whether your page is there, then ask Claude the same question and note what it cites. The two results do not always match, and the gap is information in itself.
Brave offers a URL submission form, which triggers a new visit from the crawler. No timeline is announced.
A scenario: a fictional technical services company ranks well on Google for its main service page. It asks Claude the question a buyer would ask and does not appear in the answer. It then types the same query into Brave Search: the page is not there.
The problem is not Claude, it is the index. The company checks that Googlebot is blocked on no template, submits the page through Brave's form, then repeats the check a few weeks later, with no guaranteed timeline.
We treat each independent index as a door to check separately. In audits, the configuration that causes trouble most often is a robots.txt file inherited from a redesign: written to keep content scrapers out, it allows only two or three known crawlers by name. The intent was defensive; the effect is to remove the site from the index that feeds Claude.
We therefore add this check to the launch checklist of every redesign, on the same footing as the sitemap. We promise no timeline for appearing, since neither Brave nor Anthropic publishes one.
Not to be confused with
- Third-party search provider
- An independent index describes who builds the index; a third-party search provider describes who sells access to it. Brave is both: it builds its index and sells it through its API.
- Retrieval crawler
- A retrieval crawler, such as Claude-User, fetches a page at the moment a person asks a question. It does not fill an index: it reads the page live for a specific answer.
Related concepts
- Third-party search provider
- Retrieval crawler
- Grounding query
- Technical SEO
- Optimizing for generated answers
- AI Overview
Further reading
- Claude and Brave Search: how to make your website accessible to AI
- What search engine does ChatGPT use?
- AI crawlers: which should you allow or block on your website?
- SEO vs. GEO: what's the difference for your online visibility?
- Measuring AI visibility: which free and paid tools should you use?
Related services
Frequently asked questions
How do I know if my site is in Brave's index?
Type your important queries into Brave Search and see whether your pages appear: Brave has no console for site owners. If a page is missing, first check that Googlebot is blocked nowhere. Then submit it through Brave's form, which triggers a new visit from the crawler.
Is robots.txt enough to remove a page from Brave's index?
No. According to Brave's help page, robots.txt prevents crawling, not necessarily inclusion in the index. To remove a page, you need the noindex tag. This confusion often produces wrong diagnoses when a blocked page keeps showing up.
