Test your robots.txt the way Google reads it
This free robots.txt tester shows you, line by line, which crawlers may visit which pages of your site. Paste your file, add a few URLs and see instantly whether Googlebot, Bingbot, GPTBot or any other crawler is allowed or blocked, and which rule decides it. It follows Google's published robots.txt rules, catches the mistakes that quietly hide pages from search, and runs entirely in your browser.
Google removed its old robots.txt Tester from Search Console in late 2023. The robots.txt report that replaced it shows the file Google fetched, but you can't test a URL against a rule there, and it doesn't help with a draft you haven't published yet. Google's own help page points to third-party testers for exactly that. This tool fills the gap and adds what the old tester never had: every crawler side by side, AI crawlers, and a before-and-after comparison for changes.
How to test a robots.txt file
- Add the file. Paste your robots.txt into the editor, drop the file onto it or load an example. To get your live file, type your domain and press Open live robots.txt, then copy and paste it.
- Add the URLs you care about. Full URLs or just paths, one per line: your home page, a product, a blog post, a search page, an image.
- Pick a crawler. Googlebot is selected first. Switch to Googlebot-Image, Bingbot, Applebot, an AI crawler or type any user-agent token.
- Read the verdict. Each URL shows Allowed or Blocked with the deciding line. Select a result to see every rule that matched and why one of them won, and the line lights up in the editor.
- Fix and retest. Results update as you type, so you can adjust a rule and watch the effect immediately.
Built on Google's real matching rules
A tester is only useful if it gives the same answer as the crawler. This one follows Google's robots.txt specification and the behavior of its open-source parser, including the cases most testers get wrong:
- One group per crawler. A crawler uses only the most specific group that names it. If Googlebot has its own group, every rule under
User-agent: *is ignored for Googlebot. The tool tells you which group each crawler follows. - Documented fallbacks. Googlebot-Image, Googlebot-Video and Googlebot-News follow the Googlebot group when they have none of their own, and Applebot does the same. AdsBot-Google and Mediapartners-Google ignore
*completely. - Longest match wins. Rule order doesn't matter. The rule with the longest matching path decides, and when an Allow and a Disallow are equally long, the Allow wins.
- Wildcards.
*matches any run of characters and$at the end means “end of the URL”, so/*.php$blocks/page.phpbut not/page.php?id=1. - The details. Paths are case-sensitive, rules for the same crawler are merged, non-ASCII paths are compared in their percent-encoded form,
/robots.txtitself is always allowed and Google stops reading after 500 KiB.
Every crawler at a glance
The All crawlers tab tests your whole URL list against Googlebot, Googlebot-Image, Bingbot, Applebot, AdsBot and the main AI crawlers at once, in one table. Add or remove crawlers with a click and select any cell to see the rule behind it. It's the quickest way to spot a page that is open to search engines but blocked for images, or blocked for ads by accident.
Control AI crawlers with confidence
AI companies use separate crawlers for different jobs, and the difference matters. Blocking a training crawler keeps your content out of future models; blocking an AI search crawler also removes you from AI answers and their links back to you. The AI crawlers tab shows, for any URL, which of these can visit:
- Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider and Amazonbot.
- AI search: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- User fetchers: ChatGPT-User, Claude-User and Perplexity-User, which open a page when a person asks for it.
One click adds ready-made rules to block AI training and keep AI search, block all AI crawlers or allow them all, and the tool warns you if an existing group would still override the new rules. Remember that robots.txt is a request: reputable crawlers respect it, but it can't lock anything.
See what a change will do before you publish it
A single character in robots.txt can hide a whole section of a site from Google. Under Compare versions, paste the edited file next to the current one and the tool lists every URL whose result changes, for every crawler you selected, such as “/search?q=shoes: blocked → allowed for Googlebot”. When you're happy, load the new version into the editor with one click.
Finds the mistakes that cost traffic
The Issues tab checks the file as you type and marks problem lines in the editor:
- Rules placed before any User-agent line, which no crawler uses.
- Paths without a leading slash and full URLs in Disallow, which never match.
- Typos such as “Disalow” and missing colons that Google tolerates but other crawlers don't.
- Crawl-delay (ignored by Google), Noindex (no longer supported) and other fields crawlers skip.
- Blocked CSS or JavaScript, a whole-site
Disallow: /and groups that silently overrideUser-agent: *. - Relative sitemap URLs, HTML served instead of plain text, retired crawler names and files over the 500 KiB limit.
Share results with your team
Summary & export lists every group, the number of rules and your sitemaps. Copy a share link that opens the same file, URLs and crawler for a colleague or developer, download the cleaned-up robots.txt, or export all results as a CSV for a ticket or an audit. The link carries the data itself, so nothing is stored on a server.
Common robots.txt examples
| Goal | Rules |
|---|---|
| Allow everything | User-agent: *Disallow: |
| Block one folder | User-agent: *Disallow: /private/ |
| Block a folder but allow one file in it | User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.php |
| Block URLs with a parameter | User-agent: *Disallow: /*?sort= |
| Block one file type | User-agent: *Disallow: /*.pdf$ |
| Block AI training only | User-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedDisallow: / |
Keep in mind that robots.txt controls crawling, not indexing. A blocked page can still appear in search results without a description if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
Private by design
Your file and URLs never leave your browser. Testing happens on your device, your last session is saved in this browser so you can continue later, and share links contain the data in the link itself rather than on a server. That makes it safe to test staging files and unreleased URLs.