Definition
What is crawl budget?
Crawl budget is the set of URLs that Google can and wants to crawl on a site. Google describes it as the meeting of two limits: crawl capacity (what its crawlers can read without overloading your server) and crawl demand (what they want to read based on the popularity, freshness and perceived quality of the pages). It only becomes a constraint on very large sites.
Crawl capacity and crawl demand
Crawl capacity depends on your server. Google adjusts the pace of its crawlers so as not to slow your site down: a server that responds quickly gets crawled more, a server that slows down or returns errors gets crawled less. Other crawlers, including AI crawlers, hit the same server. Their load shows up in your server logs, not in your traffic reports.
Crawl demand depends on how interested Google is in your pages: their popularity, how often their content changes and the quality it perceives in the site. Google writes that the only two ways to increase crawl budget are to increase your server's capacity and, above all, to increase the value of your content for people searching.
Which sites are concerned
Google reserves the topic for three situations: sites with more than a million pages whose content changes weekly, sites with more than ten thousand pages whose content changes daily, and sites where a large share of URLs remains "Discovered, currently not indexed" in Search Console. Its documentation specifies that if your pages are crawled on the day they are published, the topic does not concern you.
A hundred-page site comes nowhere near. A catalogue of a few thousand products can still fall into it without knowing, because what counts is the number of URLs the site generates. Four filters with ten values produce ten thousand combinations for a single category. Product variants multiply the total further.
What wastes it
- A catalogue's filters. Every combination in faceted navigation creates a URL the crawler can only judge after crawling it. Google names two effects: over-crawling of worthless URLs, then slower discovery of new pages.
- Product variants that each get a URL without search demand of their own.
- Redirect chains and internal links to error pages, which waste crawl time.
- Combinations with no results that respond normally instead of returning a 404 error: they keep getting crawled.
- Thin or duplicate pages, which lower the perceived quality of the site, and therefore crawl demand.
The remedies specific to filters (blocking in the robots.txt file, URL fragments, canonical tag) are covered in our article on faceted navigation. What does not work: resubmitting a URL, repeating an indexing request or paying for a service. Google accepts no payment to crawl a site more often.
Fictional example: a store sells 3,000 products. Its brand, colour, size and price filters generate more than 200,000 listing URLs. The Crawl stats report in Search Console shows that most of Googlebot's requests target URLs with parameters. Meanwhile, the season's new arrivals stay "Discovered, currently not indexed" for weeks: a product already paid for takes longer to sell.
The store blocks the filter parameters in the robots.txt file. It keeps product pages and one complete unfiltered listing open. Crawling can then shift to the pages that sell.
We read crawl budget as a large-catalogue topic, not as an argument for spreading out a publication. On a site of a few hundred pages, the obstacle is elsewhere: a crawled page is not an indexed page. It is the quality Google assigns to the page that decides.
On a catalogue, we first measure two figures: the number of URLs the site can generate and the time between adding a product and its first online sale. The list of filters that deserve a page remains your decision, because it depends on the sales each combination can bring. What can be delegated: counting the URLs, configuring the robots.txt file, checking canonicals and monitoring indexing.
Not to be confused with
- Indexing
- Crawl budget decides which URLs Google reads and at what pace. Indexing decides which ones it keeps in its index after reading them. A crawled page is not necessarily indexed.
- Faceted navigation
- Faceted navigation is a catalogue's filtering system. Built on URL parameters, it is the most documented cause of wasted crawl budget.
Related concepts
- Faceted navigation
- Indexing
- Crawled, currently not indexed
- Distinct page threshold
- Batch publishing
- Canonical tag
- Technical SEO
- Internal linking
Further reading
- Faceted navigation: managing filters to protect your crawl budget
- Publishing 100 pages at once: what are the SEO risks for a new site?
- Technical SEO: finding crawl and indexing problems
- How long does Google take to index a site? The timings Google gave in 2026
- AI crawlers: how do they affect traffic and hosting costs?
- Product variants and SEO: when should you create a separate page?
Related services
Frequently asked questions
My site has a few hundred pages: should I worry about crawl budget?
No. Google reserves the topic for sites with more than a million pages that change weekly, sites with more than ten thousand pages that change daily, and sites where a large share of URLs remains discovered but not indexed. If your new pages are crawled on the day they are published, the topic does not concern you.
How can I tell whether Google is wasting time on useless pages?
Three sources show it. The Crawl stats report in Search Console shows which URLs Googlebot requests. The Page indexing report shows how many pages remain "Discovered, currently not indexed". Finally, your server logs show every crawler visit, including AI crawlers.
Can we pay or push for Google to crawl more often?
No. Google accepts no payment to crawl a site more often. Repeating a request for the same URL does not speed it up either. Google names two levers: your server's capacity and, above all, the value of your content for people searching. A link from the home page or a category page also helps Google find a new page.
