Open menu

Robots.txt Generator & Tester

Create a correct robots.txt without memorising the syntax. Start from a preset (typical site, allow everything, block everything for staging, block AI training crawlers while keeping search engines, or WordPress), edit groups of user agents with their Disallow and Allow paths and optional crawl delay, and add your sitemap URLs. A built-in tester applies the rules exactly as Google does (RFC 9309), showing whether a URL is allowed for a chosen crawler and which rule decided it.

Rules

Group 1

Output

robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Test a URL

Uses Google's rules (RFC 9309): the most specific user-agent group applies, then the longest matching path wins; on a tie, Allow wins. * matches anything and $ marks the end of the URL.

Blocked

Decided by “Disallow: /admin/” in the group for *.

Checks

  • Info robots.txt is public and only a request: well-behaved crawlers follow it, others don't. Protect private pages with a login, and use noindex (not Disallow) to keep a page out of search results.

Runs entirely in your browser. Nothing you paste or open here is uploaded or stored.

How to use the robots.txt generator

  1. Start from a preset that’s closest to your site.
  2. Edit each group: user agents (one per line, * for all), paths to disallow and exceptions to allow.
  3. Add more groups for crawlers that need different rules.
  4. List your sitemap URLs, which must be full URLs.
  5. Test URLs with the tester: type a path, choose a crawler and see Allowed or Blocked with the deciding rule.
  6. Download robots.txt and upload it to your site’s root.

How the rules work

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines followed by Disallow and Allow rules:

User-agent: *
Disallow: /admin/
Allow: /admin/public/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

A crawler uses only the group that matches its name most specifically, and falls back to * if no group names it. Rules are path prefixes: /admin/ blocks everything under that folder. * matches any characters and $ anchors the end, so /*.pdf$ blocks PDF files but not /file.pdf?download=1. An empty Disallow: allows everything.

The tester follows these rules: most specific group, longest matching rule, Allow winning ties.

Worked examples

Example 1: a typical business site

The default preset blocks /admin/, /cart/ and /search for all crawlers. Internal search results and carts add nothing to search engines and waste crawl budget. Test /admin/settings and it’s blocked by Disallow: /admin/; test /blog/ and it’s allowed because no rule matches.

Example 2: blocking AI training crawlers

Load Block AI crawlers. Test /blog/ with GPTBot and it’s blocked by Disallow: / in the AI group. Test the same path with Googlebot and it’s allowed. Note that Google-Extended controls the use of your content for Google’s AI models without affecting Google Search.

Example 3: WordPress

The WordPress preset blocks the admin area but allows admin-ajax.php, which themes and plugins use for front-end features. Test /wp-admin/admin-ajax.php: the Allow rule is longer than the Disallow rule, so it wins.

Things robots.txt can’t do

  • Hide private content. The file is public and lists exactly what you’d rather hide. Use authentication.
  • Remove pages from search results. Use noindex, and make sure the page isn’t blocked so crawlers can see the tag.
  • Control badly behaved bots. Compliance is voluntary; block abusive traffic at the server or firewall.

Common mistakes

  • Disallow: / left over from staging on a live site.
  • Blocking CSS and JavaScript, which stops Google from rendering pages properly.
  • Relative sitemap URLs. The Sitemap: line needs a full URL.
  • Paths without a leading slash, which are ignored.
  • Relying on Crawl-delay for Google, which ignores it.

Frequently asked questions

Does robots.txt keep pages out of Google?

Not reliably. It stops compliant crawlers from fetching pages, but a blocked URL can still be indexed (without content) if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag or header instead.

How do I block AI crawlers?

Use the “Block AI crawlers” preset. It adds a group for user agents such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot with Disallow: /, while regular search crawlers keep access. Compliance is voluntary, and new crawlers appear regularly, so review the list now and then.

What happens when Allow and Disallow both match?

Google uses the most specific rule, meaning the longest matching path. If an Allow and a Disallow rule are equally long, Allow wins. The tester shows which rule decided.

Where does robots.txt go?

In the root of the host, for example https://www.example.com/robots.txt. Each subdomain and protocol needs its own file, and it must be plain UTF-8 text.

Should I add my sitemap to robots.txt?

Yes. A Sitemap: line with the full URL helps crawlers find it, and you can list several sitemaps or a sitemap index.

Sources