Robots.txt Generator
Generate robots.txt, including AI crawler rules
What is Robots.txt Generator?
This robots.txt generator covers the usual crawl directives and adds a dedicated section for AI crawlers, letting you allow or block retrieval bots such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot separately from pure scrapers. The file is generated locally in your browser.
How to use Robots.txt Generator
- 1Enter your sitemap URL and the paths you want to keep out of the index.
- 2Click an AI crawler tag to toggle it; highlighted means allowed.
- 3Copy the output and save it as robots.txt at the root of your site.
- 4Once it is live, verify it with a search engine's robots.txt testing tool.
Frequently asked questions
Should I allow or block AI crawlers?+
It depends on what you want from them. Retrieval bots such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot decide whether your content can be cited in AI answers and drive visibility back to you, so they are usually worth allowing. Bots that only scrape for training crawl heavily and rarely send anything back, so blocking them is reasonable when bandwidth matters.
Can robots.txt stop AI companies from crawling my site?+
No. It is a voluntary protocol. Mainstream crawlers honor it, but some agent-style User-Agents are defined by their vendors as real-time requests made on behalf of a human and are explicitly exempt from robots.txt. To genuinely limit that traffic you need rate limiting at your CDN or origin.
Does Disallow protect sensitive pages?+
No, and it makes matters worse. robots.txt is publicly readable, so listing /admin/ tells everyone exactly where your admin panel lives. Sensitive content has to be protected by authentication; robots.txt is only for managing crawl budget.
Related tools
All toolsHTTP Status Codes
HTTP status code reference with likely causes and where to look
URL Parser
Break a URL into its components and query parameters
MIME Type Lookup
MIME type and file extension reference
User Agent Parser
Parse a User-Agent string into browser and OS details
CIDR Subnet Calculator
CIDR subnet calculator with membership checks
HTTP Header Analyzer
Inspect response headers and security header coverage