robots.txt Generator

Build a robots.txt file with per-bot allow/disallow rules, a sitemap URL, and one-click presets for Googlebot, GPTBot, and other common crawlers.

runs locally on your browser. Your data never leaves your device.

Add a bot / user-agent group

1User-agent groups
1Rules
2Lines generated

Common Use Cases

Allow full crawling for search engines while blocking specific AI-training crawlers like GPTBot or CCBot
Disallow admin, staging, or checkout paths from being indexed on a new site before launch
Point every crawler at your sitemap URL with one "Sitemap:" line instead of writing it by hand
Quickly regenerate a clean robots.txt after adding a new section of the site that shouldn't be indexed yet

About robots.txt Generator

A robots.txt file, served from a site's root (https://example.com/robots.txt), tells well-behaved web crawlers which parts of the site they're allowed to request: search engines like Google and Bing, and increasingly, the AI crawlers that fetch pages to train models or answer questions (GPTBot, ClaudeBot, Google-Extended, CCBot, and others). The syntax is a short, plain-text format, but it's still easy to typo a directive name, forget a "User-agent:" line before a rule, or accidentally write one broad block that also disallows a bot you meant to allow. This generator builds a valid file for you from a visual rule builder instead.

Start from a bot preset: "All bots" for a blanket "User-agent: *" group, or a specific crawler like Googlebot, Bingbot, GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, CCBot, PerplexityBot, Bytespider, or Amazonbot; or add a custom user-agent for anything not listed. Each group gets its own list of Allow/Disallow rules with a path (leave a Disallow path blank to explicitly allow everything for that bot, or add specific paths like "/admin/" or "/private/" to block them), plus an optional Crawl-delay. Add a sitemap URL once at the bottom and it's appended as a "Sitemap:" line so crawlers can find your sitemap without you listing it in every group.

The output panel updates live as you edit groups and rules, always in the same order they're arranged on screen, so re-ordering how you build it never changes what gets generated. Copy it to your clipboard or download it directly as "robots.txt", no renaming needed, since (unlike a dotfile such as .gitignore or .htaccess) "robots.txt" has a normal extension a browser download keeps intact. Everything, from the presets to the rule assembly and the live preview, runs as plain JavaScript in your browser tab; nothing you configure or generate is ever uploaded to a server, logged, or stored.

Frequently Asked Questions

Where do I put the generated robots.txt file?
Upload it to the root of your domain so it's reachable at https://yourdomain.com/robots.txt: that's the only location crawlers check. Putting it in a subdirectory (like /blog/robots.txt) has no effect on the rest of the site.
What does a blank Disallow path mean?
An empty "Disallow:" line is a valid, meaningful directive: it means nothing is disallowed, i.e. that bot may crawl everything. It's the default this tool starts a new "All bots" group with, since that's the most common intent (allow full crawling) until you add specific paths to block.
Can I block AI crawlers like GPTBot or ClaudeBot without blocking search engines?
Yes, that's the point of having a separate group per user-agent. Add a preset for the specific AI crawler (GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider), give it a "Disallow: /" rule to block it entirely, and leave your Googlebot/Bingbot/"All bots" groups untouched so search indexing keeps working as normal.
Does robots.txt actually stop a bot from accessing a page?
It's a request, not an enforcement mechanism: reputable crawlers (Googlebot, Bingbot, and most named AI crawlers) respect it, but nothing stops a non-compliant bot from ignoring the file entirely. For content that must never be accessed, use real access control (authentication, a firewall rule) rather than relying on robots.txt alone.
Why does the sitemap field only accept one URL?
Most sites only need one sitemap (often a sitemap index file listing others), so a single field covers the common case cleanly. If you need multiple Sitemap: lines, generate the file with one, then add the extra lines by hand afterward; robots.txt is plain text, so extending it is straightforward.
Is anything I configure or generate sent anywhere?
No. The presets and rule assembly run entirely in your browser using plain JavaScript, so nothing is uploaded to a server, logged, or stored anywhere outside your current browser tab.

Related Tools

View all Generators

Lorem Ipsum Generator

Generate paragraphs, sentences, or words of placeholder text for mockups and designs.

Random Number Generator

Generate random numbers with custom ranges, quantity, and sorting options for testing and development.

QR Code Generator

Generate and download QR codes for any URL or text. Customize size, colors, and error correction level.