What a 500-page crawl costs, and why Lighthouse is opt-in

The site audit is the one part of netweft that does not buy its data. The crawler is ours, so a crawled page costs compute and bandwidth rather than a vendor call. That makes it cheap, and it makes one adjacent decision expensive.

The two numbers

A crawled page is one credit. Five hundred pages, five hundred credits — about five cents of infrastructure at the basis the credit is priced on.

A performance check on a page — the Core Web Vitals pass — is 50 credits. Same page, 50 times the cost, because that one is a vendor call.

Measured on every page of a five-hundred-page crawl, the run would go from 500 credits to 25,500, 20% of the 125,000 a Pro workspace gets in a month. So netweft does not measure every page. It measures the start URL, then one page per URL shape it has not measured yet, up to 10. The most a performance pass can add to any audit is 500 credits.

Why not just make it cheaper

Because we would be lying about what it costs. The rate card is derived from the vendor rate card at a fixed basis; moving one line to make it look better would either move the margin or move the truth. Measuring fewer pages changes neither: each page measured still costs what the vendor charges.

What to do instead

The useful shape is not “audit everything, measure everything”. It is:

Crawl wide. The structural findings — broken links, missing canonicals, redirect chains, thin or duplicated titles — come from the crawl itself, at one credit a page. Run that across the whole site as often as you like, subject to your own crawl budget rather than ours.

Measure narrow. Performance is a page-template problem far more often than a page problem, which is why the sample is one page per URL shape. Ten pages that represent your templates tell you almost everything a five-hundred-page sweep would, and the sample keeps the pass at 500 credits at most. The template guide shows how the sample is chosen.

Re-measure on change. A performance check is worth spending after a deployment that touched the template, not on a schedule that touches nothing.

Severity is the other half

An audit that returns four hundred equal-weight findings is a backlog nobody starts. netweft classifies findings by severity so the list has a top, which is the difference between a report and a morning’s work.

The crawl respects robots.txt, which is worth saying plainly: netweft will not fetch what you have asked crawlers not to fetch, including on your own site. If a section is disallowed and you want it audited, the fix is in the robots file, not in a setting here.

The general rule

Where a check is cheap because we run it, run it broadly. Where a check is expensive because someone else runs it, run it where it changes a decision. The rate card is published in full precisely so that this is a calculation you can do before the run rather than a number you find afterwards. The wider argument for pricing lookups this way is in what an SEO data lookup actually costs.