Free Robots.txt Generator - Block AI Bots, ChatGPT & Web Scrapers

Robots.txt generator

This tool builds a robots.txt file from a handful of settings: a default rule for every crawler, a list of paths to disallow, an optional crawl-delay and sitemap line, and a checklist of 19 named AI crawlers and scrapers split into two groups. Every change updates the preview immediately, and the finished file can be copied or downloaded as robots.txt.

How the file is put together

A robots.txt file is a plain list of User-agent and directive pairs, read from top to bottom. The generator always opens with User-agent: *, followed by Disallow: / when the default is set to block everything, or one Disallow: line per path entered under Restricted Paths when the default is set to allow, with Allow: / written instead if no paths are listed. An optional Crawl-delay line sets a wait time in seconds between requests; it is added regardless of which default is chosen, though not every crawler honors it. Google's crawler ignores Crawl-delay entirely, while Bing and several others still read it. Checking a box under AI Bot Protection adds a separate User-agent block using that crawler's real token, such as GPTBot for OpenAI's training crawler or CCBot for Common Crawl, each followed by its own Disallow: /. A single toggle above the list can check or uncheck every bot at once, switching to uncheck if every box is already selected. A Sitemap: line pointing at the entered URL is appended last, if one was provided.

Worked example

Take a site that keeps the default on Allow, lists /admin and /cart under Restricted Paths, sets Crawl-delay to 10, checks GPTBot and CCBot under AI Bot Protection, and enters https://example.com/sitemap.xml as the sitemap. The output opens with User-agent: * and Crawl-delay: 10, disallows /admin and /cart, adds a separate block for GPTBot and one for CCBot, each disallowing /, and closes with the Sitemap line.

Example output

# Robots.txt generated by ToolsOfEverything

User-agent: *
Crawl-delay: 10
Disallow: /admin
Disallow: /cart

# AI Bot Protection
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /


Sitemap: https://example.com/sitemap.xml