Crawl budget
Also called crawl capacity
The number of pages a search engine is willing and able to fetch from a site in a given period, set by the site's capacity to serve requests and the engine's interest in the content. It matters at scale and rarely below a few thousand pages.
Crawl budget is the most over-applied concept in technical SEO. For a site of a few hundred pages it is not a constraint and never will be.
It becomes real somewhere in the tens of thousands of URLs, or earlier on a site that generates URLs faster than it publishes content — faceted navigation, calendars, session parameters, unbounded filter combinations.
Two forces set it. How fast your server responds without degrading, and how much the engine wants your content. You control the first directly and the second only slowly.
What wastes it. Infinite URL spaces from filters and sorts, long redirect chains, soft 404s that return 200, and large numbers of near-duplicate URLs that each get fetched before being discarded.
What conserves it. Correct canonicals, a
robots.txt that excludes
parameter spaces with no unique content, honest status codes, and internal
linking that reaches your important pages in few hops.
netweft’s crawl respects robots.txt for the same reason
a search engine does, and reports the structural findings that waste crawl
budget by severity.
Last reviewed . Also served as markdown at /glossary/crawl-budget.md.