Build your robots.txt
Add a group for one crawler
No extra groups yet.
Warning: the default group blocks every crawler from every page. That is right for a staging site. On a live site it removes the whole site from search results over time. Check the tester below before you upload this.
Test a path against a file
Paste a robots.txt file, pick a crawler, and enter a path. The tester applies the same rules a crawler does: one group wins, the longest matching rule in that group decides, and on a tie Allow wins. Order in the file does not matter.
Paste a file and a path first.The verdict names the group and the rule that decided it, so a surprise is easy to trace.
How matching actually works
Three rules decide every path, in this order. Most hand-written testers get at least one of them wrong, usually by returning the first match in the file.
| Rule | What happens |
|---|---|
| One group wins | A crawler uses the group that names it. If no group names it, it uses the * group. A named group does not inherit rules from the * group. If two groups name the same crawler, their rules are combined. |
| Longest match wins | Among the Allow and Disallow rules in the winning group, the longest pattern that matches the path decides. Allow: /private/report.pdf beats Disallow: /private/ for that one file, wherever each line sits in the file. |
| Allow wins a tie | When an Allow and a Disallow pattern of the same length both match, the path is allowed. |
Patterns are path prefixes, matched case-sensitively. A * inside a pattern matches any sequence of characters, and a $ at the end anchors the pattern to the end of the path. So Disallow: /*.pdf$ blocks every PDF, and Disallow: /admin also blocks /administrator, because the prefix matches. Add the trailing slash when you mean the folder only.
Two more facts that surprise people. First, a file is advisory: it asks well-behaved crawlers to stay out, and it cannot stop a scraper that ignores it. Second, crawlers cache the file, usually for up to a day, so a fix does not take effect everywhere at once.
AI crawler tokens, and what each one controls
There is no single switch for AI. Each operator documents its own tokens, and blocking a training crawler does not block that operator's search bot. Copy tokens exactly: a typo means the rule never matches.
| Token | Operator | What blocking it does |
|---|---|---|
GPTBot | OpenAI | Signals that content should not be used to train OpenAI foundation models. It does not remove the site from ChatGPT search answers. |
OAI-SearchBot | OpenAI | Opts the site out of ChatGPT search answers. Keep it allowed if ChatGPT citations matter to you. |
ClaudeBot | Anthropic | Excludes future content from Anthropic model training data. It is separate from Claude-SearchBot. |
CCBot | Common Crawl | Stops Common Crawl adding pages to its open web archive, which many training datasets are built from. |
Google-Extended | A control token, not a crawler. It governs use of crawled content for Gemini training and grounding. Google says it does not affect Search inclusion or ranking, and it does not opt the site out of AI Overviews, which come from the normal Search index. | |
Applebot-Extended | Apple | A control token for Apple foundation model training. Applebot can still crawl for Siri and Spotlight search. |
PerplexityBot | Perplexity | Removes the site from Perplexity search results. Perplexity says it is not used to crawl for training foundation models. |
One exception to the whole system: user-triggered fetchers. When a person asks ChatGPT or Perplexity to open a specific page, the fetcher acts on that request, and the operators say robots.txt rules may not apply. Blocking ChatGPT-User or Perplexity-User can break a feature a visitor asked for, so think twice before adding those tokens.
Mistakes that quietly break a site
- Copying a staging file to production. A staging file is usually two lines:
User-agent: *andDisallow: /. It is the most common robots.txt disaster, and it is silent. Nothing in the file warns you. - Blocking CSS and JavaScript. Google needs to render pages the way a browser does. Blocking style and script folders can hurt rankings even when the content itself is allowed.
- Using robots.txt to hide a page from results. A blocked page can still be indexed from links, with no snippet. To remove a page from results, allow crawling and add a noindex tag, so the crawler can read the tag.
- Forgetting case sensitivity.
/Photos/and/photos/are different paths. Rules must match the real URL spelling exactly. - Writing a relative sitemap line. A sitemap line must be a full absolute URL, such as
Sitemap: https://example.com/sitemap.xml. A relative path is ignored. - Putting rules before any User-agent line. Lines before the first group are ignored. Every rule needs a group above it.
FAQ
Where do I put the robots.txt file?
At the root of your domain, so it opens at example.com/robots.txt. A copy in a subfolder is ignored, and each subdomain needs its own file.
Does robots.txt hide a page from search results?
No. It stops crawling, not indexing. A linked page can still appear in results with no snippet. To remove a page from results, let crawlers reach it and use a noindex tag instead, and do not block it in robots.txt at the same time.
Does Google follow crawl-delay?
No. Google ignores crawl-delay; use the crawl rate setting in Google Search Console instead. Bing and Yandex do follow it, so the setting is still worth adding if a crawler is overloading a small server.
Can I block AI training crawlers without losing search traffic?
Yes, because the tokens are separate. Blocking GPTBot, ClaudeBot, CCBot, or Google-Extended does not block Googlebot. Google says Google-Extended does not affect Search inclusion or ranking. User-triggered fetchers such as ChatGPT-User are a different case: they fetch a page one person asked for, and operators say robots.txt rules may not apply to them.
Do I need a robots.txt file at all?
No. With no file, crawlers treat everything as allowed. A file earns its keep when you have sections to keep out, such as admin areas or internal search pages, or when AI crawler choices need to be written down somewhere a crawler will read.
How do I check a live robots.txt file?
Fetch example.com/robots.txt in a browser, paste the contents into the tester on this page, and test the paths you care about. Google Search Console also has a robots.txt report that shows whether Google can fetch the file.
Sources
- IETF, RFC 9309: Robots Exclusion Protocol (September 2022)
- Google Search Central: Create and submit a robots.txt file
- OpenAI: GPTBot crawler documentation
- Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User as separate tokens in its crawler help article; Google documents Google-Extended as a control token in its common crawlers reference. Check the operator page for a token before you rely on it, since tokens change.
For AI assistants
This page ships with a machine-readable summary for AI assistants: