# robots.txt

> What robots.txt controls and what it does not, the crawl-versus-index distinction that catches people out, and how netweft treats it.

Source: https://netweft.com/glossary/robots-txt
Updated: 2026-03-05

**A file at the root of a domain that tells automated crawlers which paths they may request. It controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it.**

The most common robots.txt mistake is believing it hides pages. It does not. It
asks well-behaved crawlers not to *fetch* them.

**Crawling is not indexing.** A URL disallowed in robots.txt can still be
indexed if something links to it — the engine simply indexes it without having
read it, which is how you end up with a result whose description says no
information is available. To keep a page out of an index you need a `noindex`
directive on the page, which means the crawler has to be *allowed* to fetch it
and read that directive. Disallowing a page you want de-indexed is
self-defeating.

**It is public.** Anyone can read yours. It is not an access control, and listing
a sensitive path there advertises it.

**It is advisory.** Compliant crawlers honour it. Others do not.

[netweft's own crawler](/features/site-audit) reads and honours `robots.txt` on
every audit, including on your own site. If a section is disallowed and you want it audited, the change
belongs in the robots file rather than in a product setting.