Robots.txt tester
Paste a robots.txt file, name a crawler and a path, and see whether the crawler may fetch it and exactly which line decided.
Runs in your browser. Nothing you type is sent anywhere.
Matched case-insensitively against each User-agent line.
The path and query, such as /blog/post?page=2.
Sitemaps listed
- https://example.com/sitemap.xml
How to use it
- 1 Paste the robots.txt file.
- 2 Enter the crawler's user agent, such as Googlebot.
- 3 Enter a path, such as /blog/post, and read the verdict and the rule behind it.
How a crawler reads robots.txt
A crawler obeys the one group whose User-agent line names it, and only falls back to the group for * when no group names it. Rules from the * group do not add to a named group.
Within the group, the longest matching rule wins, and an Allow beats a Disallow of the same length. A * matches any characters and a $ at the end anchors the end of the path. This is how Google documents the standard, RFC 9309.
What robots.txt does not do
Blocking a path stops crawling, not indexing: a blocked page that other sites link to can still appear in results, without a description. To keep a page out of results, let it be crawled and mark it noindex.
Questions
- Is my robots.txt sent to netweft?
- No. The file is read and matched in this page. Nothing you paste is sent anywhere.
- Does netweft's own crawler follow robots.txt?
- netweft's site audit crawler identifies itself as netweft-audit and obeys the group that names it, or the * group.
- Will every crawler read it the same way?
- Matching follows RFC 9309 as Google documents it. Other crawlers may differ in details, such as how they treat unsupported lines.
Related tools
- Redirect checker Follow a URL's redirects hop by hop and read the status code and headers of every answer.
- SEO page checker Check one page against the single-page rules of netweft's site audit: title, description, headings, content and indexing.
- Hreflang generator Write and check hreflang tags for every language version of a page, as link tags or sitemap entries.
A netweft site audit reports when its crawler is blocked, and skips the paths robots.txt disallows. See how
Check it every week instead.
netweft runs these checks across a whole site and its rankings on a schedule, and keeps every result.
There is no free tier. Three plans from $50 a month, each opening with a free trial.