Robots.txt tester

How to find your robots.txt

A site's robots.txt is always at the top level of its host. Open https://example.com/robots.txt with your own domain in place of example.com, select all of the text and paste it into the tester.

• Each host has its own file. Google applies the rules only to the host, protocol and port the file is served from, so www.example.com/robots.txt doesn't cover shop.example.com, and the https file doesn't cover http.
• No file means no rules. If the address shows a not found page, Google treats any 4xx answer except 429 as if there were no robots.txt and crawls without restrictions.
• A server error is different. When robots.txt answers with a 5xx error, Google stops crawling the site for the first 12 hours, then uses the last good copy it read for up to 30 days.
• Where to edit it. On Shopify: Online Store, Themes, Edit code, then robots.txt.liquid, which you add first with Add a new template, robots, if your theme doesn't have it. On WordPress: your SEO plugin's file editor (Yoast: Tools, File editor; Rank Math: General Settings, Edit robots.txt). Elsewhere: the robots.txt file at the root of your site.
• What Google last read. Search Console's robots.txt report lists the files Google found for the top 20 hosts of your site and when it last checked each, and URL Inspection tells you whether a page is blocked. See how to set up Search Console.

How robots.txt matching works

A crawler reads the file in three steps, and this tester does the same:

1. Pick the group. A group is one or more User-agent lines followed by their rules. A crawler follows every group that names its product token, combined into one. Case doesn't matter, and text after the name is ignored, so googlebot/1.2 is Googlebot. Only when no group names the crawler does it follow the * group; with neither, no rules apply.
2. Find the matching rules. A rule matches when the address's path, query string included, starts with the rule's path. Paths are case-sensitive. * stands for any run of characters, and $ at the end means the address must end there.
3. Take the most specific. The matching rule with the most characters wins. When an Allow and a Disallow are equally long, Google uses the least restrictive rule, so Allow wins. When nothing matches, the address is allowed.

Google reads only User-agent, Allow, Disallow and Sitemap lines and ignores the rest. Sitemap lines belong to no group, and a Sitemap or unknown line between two User-agent lines doesn't split them into separate groups.

Worked examples

These results come from the tester's own code, run on the example file:

# Example robots.txt for a small online shop
User-agent: *
Disallow: /cart
Disallow: /search
Allow: /search/help
Disallow: /*.pdf$

User-agent: Googlebot
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Google's own examples

Google's documentation shows which rule wins in six cases. Each file below is a * group with the two rules shown, and the tester gives Google's answer for every one:

Common robots.txt mistakes

• Blocking a page to keep it out of Google. robots.txt stops crawling, not indexing. A blocked page can still appear in results without a description, and Google can't see a noindex tag on a page it may not crawl. Use noindex and leave the page crawlable, or put it behind a password.
• A Disallow: / left in place. One line under User-agent: * keeps every crawler without a group of its own out of the whole site. The tester flags it.
• A crawler's own group that drops the * rules. Once Googlebot (or any crawler) has a group, it ignores the * group, so rules meant for everyone stop applying to it.
• Paths without a leading slash, or full addresses. Disallow: private and Disallow: https://example.com/private match nothing. Write Disallow: /private.
• The wrong case. Disallow: /Products doesn't block /products.
• Lines Google ignores. Crawl-delay, Noindex and Host do nothing for Google. Google's parser reads a few common misspellings, such as Dissallow, and a line with no colon after the field, but other crawlers may skip them, so the tester flags both.
• Rules before the first User-agent line. They belong to no group, so crawlers ignore them.
• The wrong host. Rules in www.example.com/robots.txt don't apply to shop.example.com.

What this tester doesn't do

• It doesn't fetch anything. It tests the text you paste, so it can't see how your server answers /robots.txt or which copy a crawler has cached. Google generally caches the file for up to 24 hours.
• It follows the standard and Google. Other crawlers can read edge cases differently, and some fall back to another crawler's group when none names them: Apple says Applebot follows Googlebot's rules when robots.txt mentions Googlebot but not Applebot.
• It doesn't check the pages themselves. A page can also stay out of search because of a noindex tag or header, a canonical pointing elsewhere or a login wall.
• It doesn't test as the search engines do. For that, use URL Inspection in Google Search Console, or the robots.txt tester in Bing Webmaster Tools, which checks a URL as Bingbot would. See how to set up Bing Webmaster Tools.

Checking what AI crawlers may read? The AI crawler checker runs the same matching for GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more, and explains what blocking each one changes. Running a Shopify store? The Shopify SEO checklist covers robots.txt.liquid along with the rest.