robots.txt
Also called robots exclusion protocol
A file at the root of a domain that tells automated crawlers which paths they may request. It controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it.
The most common robots.txt mistake is believing it hides pages. It does not. It asks well-behaved crawlers not to fetch them.
Crawling is not indexing. A URL disallowed in robots.txt can still be
indexed if something links to it — the engine simply indexes it without having
read it, which is how you end up with a result whose description says no
information is available. To keep a page out of an index you need a noindex
directive on the page, which means the crawler has to be allowed to fetch it
and read that directive. Disallowing a page you want de-indexed is
self-defeating.
It is public. Anyone can read yours. It is not an access control, and listing a sensitive path there advertises it.
It is advisory. Compliant crawlers honour it. Others do not.
netweft’s own crawler reads and honours robots.txt on
every audit, including on your own site. If a section is disallowed and you want it audited, the change
belongs in the robots file rather than in a product setting.
Last reviewed . Also served as markdown at /glossary/robots-txt.md.