# Robots.txt tester

> Free robots.txt tester: paste a file, pick a crawler and a path, and see whether it is allowed and which rule decided it.

Source: https://netweft.com/tools/robots-txt-tester

Paste a robots.txt file, name a crawler and a path, and see whether the crawler may fetch it and exactly which line decided.

Runs in your browser. Nothing you type is sent anywhere.

## How to use it

1. Paste the robots.txt file.
2. Enter the crawler's user agent, such as Googlebot.
3. Enter a path, such as /blog/post, and read the verdict and the rule behind it.

## How a crawler reads robots.txt

A crawler obeys the one group whose User-agent line names it, and only falls back to the group for * when no group names it. Rules from the * group do not add to a named group.

Within the group, the longest matching rule wins, and an Allow beats a Disallow of the same length. A * matches any characters and a $ at the end anchors the end of the path. This is how Google documents the standard, RFC 9309.

## What robots.txt does not do

Blocking a path stops crawling, not indexing: a blocked page that other sites link to can still appear in results, without a description. To keep a page out of results, let it be crawled and mark it noindex.

## Questions

**Is my robots.txt sent to netweft?**

No. The file is read and matched in this page. Nothing you paste is sent anywhere.

**Does netweft's own crawler follow robots.txt?**

netweft's site audit crawler identifies itself as netweft-audit and obeys the group that names it, or the * group.

**Will every crawler read it the same way?**

Matching follows RFC 9309 as Google documents it. Other crawlers may differ in details, such as how they treat unsupported lines.

## Related tools

- [Redirect checker](https://netweft.com/tools/redirect-checker): Follow a URL's redirects hop by hop and read the status code and headers of every answer.
- [SEO page checker](https://netweft.com/tools/seo-page-checker): Check one page against the single-page rules of netweft's site audit: title, description, headings, content and indexing.
- [Hreflang generator](https://netweft.com/tools/hreflang-generator): Write and check hreflang tags for every language version of a page, as link tags or sitemap entries.

A netweft site audit reports when its crawler is blocked, and skips the paths robots.txt disallows. [See how](https://netweft.com/docs/site-audit).