Crawl budget, and the sites that actually have a problem

A site with four hundred pages does not have a crawl budget problem. It may have an indexation problem, which is a different thing with different causes, and conflating the two leads to work that changes nothing. This article is about telling them apart, and about the specific structures that create genuine crawl waste.

What crawl budget actually is

Google describes crawl budget as the product of two things: crawl capacity, which is how much your server can take without degrading, and crawl demand, which is how much Google wants your content based on its perceived value and update frequency. Neither is a fixed quota you can top up.

• Capacity rises when your server responds quickly and falls when it returns 5xx errors or slows down
• Demand rises with perceived quality, freshness and popularity — and falls for URLs that keep returning near-duplicate content
• Neither is negotiable through a setting; both respond to what your site actually does

Diagnosing it properly

Before changing anything, establish whether crawling is actually the constraint. Two data sources answer this, and neither is a third-party crawler.

That last distinction is the one most often missed. "Crawled — currently not indexed" means crawl budget was spent successfully and the page was still not worth keeping. Adding crawl capacity cannot fix a page Google has already declined.

Faceted navigation: the usual culprit

Filters multiply. Four filter dimensions with five options each produce over six hundred combinations per category, most of which return substantially the same products in a different order. Left unmanaged, a catalogue of two thousand products becomes hundreds of thousands of crawlable URLs.

The fix is a deliberate policy about which combinations deserve to exist as indexable URLs, driven by whether anyone actually searches for that combination.

1. 01Identify facet combinations with genuine search demand — "waterproof walking boots" usually has it, "waterproof boots size 9 blue sorted by price" does not.
2. 02Give the valuable combinations clean, static, indexable URLs and real internal links.
3. 03For everything else, decide one mechanism and apply it consistently: canonical to the parent category, or block the parameter pattern in robots.txt.
4. 04Never rely on canonical tags alone for the long tail — Google still has to crawl the URL to see the canonical, so the budget is already spent.
5. 05Remove low-value combinations from internal linking entirely; if nothing links to it, it is far less likely to be discovered.

Parameters, session ids and sort orders

Tracking parameters, session identifiers and sort orders each create a distinct URL serving identical content. They are the cheapest crawl waste to eliminate because no legitimate search query targets them.

# robots.txt — block crawl of patterns that never deserve to rank.
# Order matters less than specificity; test in Search Console's tester.

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Disallow: /*?*utm_
Disallow: /*?*page=          # only if pagination is handled another way
Disallow: /search

# Allow the parameter that genuinely needs indexing, if any.
Allow: /*?*colour=

Sitemap: https://example.com/sitemap.xml

Verify every rule against real URLs before shipping it. A pattern that accidentally matches your category pages will deindex a section, and the failure is silent until traffic drops.

Sitemaps as a signal, not a dumping ground

A sitemap listing every URL including redirects, 404s and canonicalised duplicates is a weak signal. A sitemap containing only canonical, indexable, 200-returning URLs is a strong one, and it makes the Search Console index-coverage report interpretable.

• Include only canonical URLs that return 200 and are intended for indexing
• Exclude anything noindexed, redirected, or canonicalised elsewhere — listing them contradicts your own directives
• Keep lastmod accurate; a file where every entry claims today's date teaches Google to ignore the field
• Split large sitemaps by section so coverage reporting isolates which part of the site has a problem
• Generate it from the same source of truth that renders the pages, so it cannot drift

Pagination and infinite scroll

Deep inventory reachable only through JavaScript-driven infinite scroll is frequently unreachable in practice. Crawlers do not scroll. If page fifty of a catalogue has no crawlable link pointing at it, it may never be discovered regardless of budget.

Keep real paginated URLs with real anchor links, even if the user experience is infinite scroll. The two can coexist: render standard links server-side and enhance with scrolling behaviour for humans.

The order that actually helps

1. 01Fix 5xx errors and slow responses — these suppress crawl capacity directly and cost nothing in strategy.
2. 02Eliminate parameter-generated duplicates through robots.txt.
3. 03Impose a faceted-URL policy and enforce it in internal linking.
4. 04Clean the sitemap down to canonical, indexable URLs only.
5. 05Ensure deep inventory has crawlable paginated paths.
6. 06Only then consider whether crawl demand — that is, content quality — is the remaining constraint.

Steps one and two are usually where the measurable change comes from, and both are unglamorous server and configuration work rather than anything resembling content strategy.