# Robots.txt Generator

> Free in-browser tool that builds a valid robots.txt file and tests any path against a robots.txt file, using the RFC 9309 matching rules. Independent tool, not affiliated with any crawler operator. Updated 2026-10-25. Full page: https://lab.awaara.cn/robots-txt-generator/

## What it does

- Generator: default rule (allow all or block all), Disallow and Allow paths one per line, an AI training crawler preset (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, each blocked with Disallow: /), extra groups for one named crawler at a time, Sitemap lines, and an optional crawl-delay for the default group.
- Output: copy the file, download it as robots.txt, or send it straight into the tester on the same page.
- Tester: paste any robots.txt file, pick a crawler token, enter a path, and get a verdict (allowed or blocked) naming the group and the rule that decided it.
- Matching rules applied by the tester: one group wins (a named group beats the wildcard group and does not inherit from it; groups naming the same crawler combine); the longest matching pattern in that group decides; on a tie, Allow wins. Patterns are case-sensitive path prefixes; * matches any sequence and a trailing $ anchors to the end of the path.
- Everything runs locally in the browser. Nothing is sent to a server.

## What robots.txt is

- A plain-text file at the root of a domain (example.com/robots.txt) that tells well-behaved crawlers which paths they may request. It is advisory, not access control, and it does not hide a page from search results: a linked blocked page can still be indexed with no snippet. To remove a page from results, allow crawling and use a noindex tag.
- Standardized as RFC 9309 (IETF, September 2022). Google's parser reads User-agent, Allow, Disallow, and Sitemap, and silently ignores lines it does not understand.
- Crawl-delay is not part of RFC 9309. Google ignores it (use Search Console crawl rate instead); Bing and Yandex follow it.
- AI crawler tokens are per operator and per purpose: GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search answers), ClaudeBot (Anthropic training), CCBot (Common Crawl archive), Google-Extended (control token for Gemini training and grounding, no effect on Search inclusion or ranking, no separate crawler), Applebot-Extended (control token for Apple foundation model training), PerplexityBot (Perplexity search results). User-triggered fetchers such as ChatGPT-User and Perplexity-User act on one person's request, and operators say robots.txt rules may not apply to them.
- Common mistakes: copying a staging Disallow: / file to production; blocking CSS and JavaScript; forgetting case sensitivity; writing a relative sitemap line; putting rules before any User-agent line; using Disallow: /admin without a trailing slash when only the folder is meant (it also blocks /administrator).

## Sources

- IETF, RFC 9309, Robots Exclusion Protocol: https://datatracker.ietf.org/doc/rfc9309/
- Google Search Central, Create and submit a robots.txt file: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
- OpenAI, GPTBot crawler documentation: https://platform.openai.com/docs/gptbot