What is the robots.txt Generator?
robots.txt is a small text file at the root of a website (example.com/robots.txt) that tells search engine and other crawlers which parts of the site they may visit. It is the first thing Googlebot, Bingbot and AI crawlers like GPTBot check, so a mistake in it can hide a whole site from search, or leave private areas open to crawling.
This generator builds the file from simple choices: rules for all crawlers or specific ones, folders to allow or block, an optional crawl delay and your sitemap URLs. One click adds rules for the common AI training and AI search crawlers. Then test any URL against the rules, using the same matching Google uses (longest rule wins, with * and $ wildcards), or paste an existing robots.txt to check it.
How to use it
- Pick a starting point: allow everything, block everything, WordPress, or block AI bots.
- Edit the groups: which crawlers they apply to and the paths to allow or disallow.
- Add your sitemap URL.
- Test a few URLs with different crawlers, then copy or download robots.txt and upload it to your site’s root folder.
Example
All crawlers: block /admin/ and /cart/ GPTBot: block everything Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Features
- Visual editor. Groups of crawlers, each with allow and disallow paths and an optional crawl delay.
- Block AI crawlers. Rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and other AI bots in one click.
- URL tester. Check whether a crawler may fetch a URL, and which rule decided it.
- Check any file. Paste an existing robots.txt to test it and catch unknown or misplaced lines.
- Presets. Allow all, block all, and a WordPress-friendly starting file.
- Download. Copy the file or download robots.txt ready to upload.
Frequently asked questions
Where do I put robots.txt?
At the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file. Crawlers ignore robots.txt files in sub-folders.
Does Disallow stop a page appearing in Google?
Not always. Disallow stops crawling, but Google can still index a blocked URL (without its content) if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or protect it with a password.
How do I block AI crawlers?
Add a group for each AI crawler’s user agent with Disallow: /. The Block AI bots preset adds the main training and AI search crawlers, including GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, PerplexityBot and CCBot. Well-behaved bots follow it; it is not a security control.
What do * and $ mean in robots.txt?
* matches any characters, and $ marks the end of the URL. For example Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks URLs with a sort parameter.
Does Google follow Crawl-delay?
No, Googlebot ignores Crawl-delay (it adjusts its own speed). Bing, Yandex and some other crawlers respect it.
Last updated · Free, no sign-up, runs in your browser.