Updated · Built and checked by the PlinzoTools team, formulas tested against published sources
Generate a robots.txt with multiple rule groups, crawl-delay, sitemaps and presets for WordPress and AI crawlers.
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlA robots.txt file tells crawlers which parts of a site they may fetch, using the Robots Exclusion Protocol (standardized as RFC 9309). This tool builds a valid robots.txt from one or more rule groups, each with its own user-agents, Allow and Disallow paths and an optional Crawl-delay, plus any number of Sitemap lines. One-click presets cover the common setups: allow everything, block everything, WordPress, blocking AI training crawlers, and allowing AI search crawlers.
Pick a preset to start from, or edit Group 1 directly. In each group, list one or more user-agents (one per line; * means every crawler), the paths to disallow and allow, and a Crawl-delay if you need one. Use Add another group to give a specific crawler different rules. List your sitemap URLs at the bottom. The file and any warnings update as you type; download it as robots.txt and upload it to your site's root.
A crawler obeys only the most specific group that names it, and falls back to the * group only if no group names it. Within a group, the longest matching path wins, and Allow wins a tie. Disallow: / blocks everything; an empty Disallow: blocks nothing. * inside a path matches any characters and $ anchors the end, so Disallow: /*.pdf$ blocks every PDF.
To check how a finished file treats a specific URL, paste it into the Robots.txt Tester. To build the sitemap it points to, use the Sitemap XML Generator.
Each group starts with one User-agent: line per crawler you listed, followed by its Allow lines, then its Disallow lines, then Crawl-delay: if you set one. Groups are separated by a blank line, and the Sitemap lines come last. A group with no paths at all gets an explicit empty Disallow:, the standard way of saying "nothing is blocked", so the file can't be misread. A group with no user-agent is left out entirely.
User-agent: * with an empty Disallow.Disallow: / for every crawler. Use it on staging and development sites, and remember to remove it at launch./wp-admin/ but allows /wp-admin/admin-ajax.php, the same rules WordPress serves by default. Themes and plugins call that file from public pages, so it should stay reachable.* group for search engines and adds a second group that disallows everything for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's control for Gemini training), Applebot-Extended (Apple), CCBot (Common Crawl), meta-externalagent (Meta) and Bytespider (ByteDance). Google-Extended and Applebot-Extended are control tokens rather than separate crawlers, so blocking them doesn't affect Google Search or Apple's search features.You can combine the last two. Apply Block AI training, add a group, and list the search agents with Allow: /. Because each crawler follows only its own group, training is blocked while AI search can still cite your pages.
Crawl-delay asks a crawler to wait the given number of seconds between requests. Bing and Yandex honor it. Google ignores it and adjusts its crawl rate automatically from how your server responds, so the tool warns when you set a delay for * or Googlebot. A large delay can slow how quickly new pages are discovered, so use it only if a crawler is genuinely overloading your server.
/ or *. Rules match from the start of the URL path, so private/ matches nothing.Disallow: / for * with no Allow lines, which blocks the whole site for every crawler./Private/ doesn't cover /private/.Upload robots.txt to the root of each host and open the address in a browser to confirm it loads. Search engines cache the file, often for up to a day, so changes take a little while to apply. Google Search Console's robots.txt report shows the version Google last fetched.
Not reliably. It asks crawlers not to fetch a URL, but a blocked URL can still appear in results, without a description, if other sites link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag.
Major search engines and the large AI companies say they do, but robots.txt is a voluntary convention, not access control. Badly behaved bots can ignore it, so never rely on it to protect private content.
At the exact root of the host, such as https://example.com/robots.txt. Each subdomain needs its own file, and a copy in a subfolder is ignored.
Choose the Block everything preset. It only asks crawlers to stay away, so password-protect a staging site if it must stay private, and remove the rule at launch.
Not by itself. Those are the training crawlers. Answers that browse or cite the web use separate agents, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User, which you can allow or block independently.
Yes. List several user-agents, one per line, in the same group and they share its rules, or add separate groups when bots need different rules.
No. Google ignores the directive and manages its own crawl rate. Bing and Yandex do honor it.
Reference: MDN Web Docs