# Crawl budget

> What crawl budget actually constrains, which sites need to care about it, and what wastes it.

Source: https://netweft.com/glossary/crawl-budget
Updated: 2026-03-05

**The number of pages a search engine is willing and able to fetch from a site in a given period, set by the site's capacity to serve requests and the engine's interest in the content. It matters at scale and rarely below a few thousand pages.**

Crawl budget is the most over-applied concept in technical SEO. For a site of a
few hundred pages it is not a constraint and never will be.

It becomes real somewhere in the tens of thousands of URLs, or earlier on a
site that generates URLs faster than it publishes content — faceted navigation,
calendars, session parameters, unbounded filter combinations.

**Two forces set it.** How fast your server responds without degrading, and how
much the engine wants your content. You control the first directly and the
second only slowly.

**What wastes it.** Infinite URL spaces from filters and sorts, long redirect
chains, soft 404s that return 200, and large numbers of near-duplicate URLs that
each get fetched before being discarded.

**What conserves it.** Correct [canonicals](/glossary/canonical-url), a
[`robots.txt`](/glossary/robots-txt) that excludes
parameter spaces with no unique content, honest status codes, and internal
linking that reaches your important pages in few hops.

[netweft's crawl](/features/site-audit) respects `robots.txt` for the same reason
a search engine does, and reports the structural findings that waste crawl
budget by severity.