Robots.txt Generator

A robots.txt file is a plain text file placed in a website's root directory that tells search engine crawlers and bots which pages they can and cannot access. For example, if your site is example.com, the file must be accessible at example.com/robots.txt. This file follows the Robots Exclusion Protocol (REP), which was formalized as an official internet standard in September 2022 under RFC 9309.

When a search engine bot like Googlebot visits a website, the first thing it does is check for a robots.txt file. If the file exists, the bot reads the instructions before crawling any pages. If no file is found, the bot assumes it can crawl everything on the site.

Having a well-configured robots.txt file serves several important purposes. First, it helps manage your crawl budget — the number of pages search engines will crawl on your site during a given visit. Google allocates a limited crawl budget to every website, and if bots waste time crawling admin panels, internal search results, or duplicate pages, your important content may take longer to get indexed. Second, it prevents non-essential or sensitive areas from appearing in search results, such as staging environments, shopping carts, user account pages, and admin dashboards.

It is important to understand that robots.txt is not a security mechanism. The file is publicly readable — anyone can view it at yoursite.com/robots.txt. Malicious bots and scrapers routinely ignore robots.txt instructions. For truly private content, use proper authentication, server-side access controls, password protection, or IP whitelisting.

Our robots.txt generator runs entirely in your browser. Nothing is uploaded to our servers, and your configuration data never leaves your device. This privacy-first approach means you can safely generate robots.txt files for any project, including internal or confidential websites.

The robots.txt file consists of one or more groups of rules, each beginning with a User-agent line that specifies which bot the rules apply to. Under RFC 9309, there are three standardized directives plus two widely-supported extensions.

User-agent specifies which crawler the rules apply to. Use an asterisk (*) to target all bots, or specify individual crawlers by name.

Disallow tells the specified bot not to crawl a path. Each Disallow directive applies to one path. An empty Disallow (Disallow:) means "allow everything."

Allow overrides a Disallow rule for a more specific path. This is useful when you want to block a directory but allow access to specific files within it.

Sitemap (extension, not in RFC 9309 but supported by all major search engines) tells crawlers where to find your XML sitemap.

Crawl-delay (extension) tells bots to wait a specified number of seconds between requests. Google does not support this directive — use Google Search Console's crawl rate settings instead. Bing and Yandex do respect it.

Wildcard patterns are supported by Google, Bing, and Yandex. The asterisk (*) matches zero or more characters, and the dollar sign ($) matches the end of the URL.

Important conflict resolution rule from RFC 9309: When multiple rules match a URL, the most specific (longest path) rule wins, regardless of whether it is an Allow or Disallow.

The rise of AI crawlers has fundamentally changed how website owners think about robots.txt. As of March 2026, over 5.6 million websites block OpenAI's GPTBot, and 5.8 million block Anthropic's ClaudeBot. Among top news sites, 79% now block at least one AI training bot. AI bot traffic quadrupled during the first half of 2025, and HUMAN Security reported a 6,900% year-over-year increase in verified AI agent traffic.

Understanding the three types of AI crawlers is essential for making informed blocking decisions:

AI Training Crawlers collect content to train large language models. Blocking these prevents your content from being used in model training but does not affect your visibility in AI search results. The major training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google Gemini training), CCBot (Common Crawl), Meta-ExternalAgent (Meta/Llama), Bytespider (ByteDance/TikTok), cohere-ai (Cohere), DeepSeekBot (DeepSeek), and Applebot-Extended (Apple Intelligence).

AI Search Crawlers index content to provide answers and citations in AI search products. Blocking these means your content will not appear in AI-powered search results like ChatGPT Search or Perplexity. The main search crawlers are OAI-SearchBot (ChatGPT Search), Claude-SearchBot (Claude search), PerplexityBot (Perplexity), and DuckAssistBot (DuckDuckGo AI Answers).

User-Triggered Crawlers fetch content in real-time when a user asks an AI to browse a specific page. These include ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, and Claude-User. These crawlers may not fully respect robots.txt because they are acting on behalf of a human user.

A critical point: blocking AI training crawlers does NOT affect your Google Search rankings. Google-Extended controls only whether Google uses your content for Gemini AI training — it is completely separate from Googlebot, which handles search indexing.

It is worth noting that as of early 2026, approximately 13% of AI bot requests ignore robots.txt entirely — up from 3.3% in late 2024. Some newer agentic browsers use standard Chrome user-agent strings, making them indistinguishable from normal browser traffic through robots.txt alone.

Mistake 1: Accidental full site block. Adding Disallow: / under User-agent: * prevents all search engines from crawling your entire site. This can completely de-index your website within weeks.

Mistake 2: Blocking CSS and JavaScript files. Google needs to render your pages exactly as users see them. Blocking CSS and JS files prevents proper rendering, which severely hurts your rankings.

Mistake 3: Case sensitivity errors. Paths in robots.txt are case-sensitive under RFC 9309. Blocking /Admin/ will not block /admin/.

Mistake 4: Missing User-agent declaration. Every Disallow or Allow rule must be preceded by a User-agent line. Rules without a User-agent are invalid.

Mistake 5: Using robots.txt for security. Robots.txt does not hide content — it actually reveals which directories you consider sensitive.

Mistake 6: Forgetting the Sitemap directive. Including your sitemap URL in robots.txt helps search engines discover your content faster.

Mistake 7: File size exceeding 500 KiB. Under RFC 9309, content beyond 500 KiB may be ignored.

After generating your robots.txt file, it must be placed in your website's root directory at https://yoursite.com/robots.txt.

For traditional hosting with FTP/SFTP access, upload the file to your public_html, www, or htdocs root folder using an FTP client like FileZilla.

For WordPress sites, plugins like Yoast SEO (under SEO → Tools → File Editor) and Rank Math (under General Settings → Edit robots.txt) provide dashboard-based editing without FTP access.

For Shopify, edit the robots.txt.liquid template. Since March 2025, Shopify supports domain-specific rules.

For Angular sites (including Angular 21 with SSR), add robots.txt to src/robots.txt and include it in the assets array of angular.json.

After uploading, verify the file by visiting yoursite.com/robots.txt in your browser. Then test it using Google Search Console's Robots.txt Tester.

The IETF AI Preferences Working Group (AIPREF), chartered in January 2025, is developing a formal mechanism for expressing AI usage preferences. The proposed draft would add a Content-Usage directive to robots.txt, allowing publishers to specify granular permissions for AI training, search indexing, and content generation.

Really Simple Licensing (RSL), launched September 2025, provides XML-based licensing terms for AI content use. RSL is supported by Reddit, Yahoo, Medium, Quora, WebMD, wikiHow, and others.

The llms.txt file, proposed in 2024, places a markdown file at /llms.txt that curates key content for large language models. About 10% of websites have adopted llms.txt, but as of March 2026, none of the three major AI providers have implemented native support for reading it.

For now, robots.txt remains the primary and most widely-respected mechanism for controlling AI crawler access. Our generator supports all current standards and will be updated as new protocols mature.