# Block or Allow AI Crawlers in robots.txt

> Generate the robots.txt rules for 21 AI crawlers, separating training from search and from fetches a real user asked for.

This generator produces robots.txt rules for 21 named AI crawlers, grouped by what the crawl is actually for: model training, AI search indexing, or fetching a single page a user asked about. The default blocks training while leaving search and user fetches allowed, because those two send readers rather than only taking content. Path-level rules, allow exceptions and crawl delay are all supported.

**URL:** https://convertto.tech/t/ai-crawler-robots-txt-generator
**Category:** SEO Tools (https://convertto.tech/c/seo-tools)
**Privacy:** Runs entirely in the browser; no upload
**Cost:** Free, no sign-up
**Last updated:** 2026-08-01

## Key facts

- **Crawlers covered:** 21, across training, AI search and user-initiated fetches
- **Common mistake:** Blocking Google-Extended does not affect Google Search ranking
- **Enforcement:** None — robots.txt is a request that operators choose to honour
- **Privacy:** Runs entirely in your browser — nothing is uploaded
- **Cost:** Free, unlimited, no sign-up

## How to use

1. Choose the policy.
2. Select one or more block these.
3. Enter or paste your block only these paths.
4. Enter or paste your but always allow these.
5. Choose the crawl delay.
6. Enter or paste your sitemap url.
7. Enter or paste your existing robots.txt.
8. The result appears immediately — copy or download it.

## FAQ

### Will blocking Google-Extended hurt my search ranking?

No. Google-Extended controls only whether your content trains Gemini or is used for grounding. Google Search crawling and ranking is Googlebot, a separate agent with separate rules. This is the single most common misunderstanding in AI robots.txt.

### Should I block AI search crawlers?

Usually not. They index you so an assistant can cite and link you, which is the same bargain as a search engine. Blocking them is the equivalent of blocking Googlebot — defensible if you have a reason, but it costs you referrals from a growing share of how people find things.

### Does robots.txt actually stop anyone?

It stops operators who choose to honour it, which includes OpenAI, Anthropic, Google and Apple. It does not stop anyone else, and several crawlers are widely reported as ignoring it. If content genuinely must not be taken, put it behind authentication — a text file is a request, not a control.

## Sources

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — IETF
- [OpenAI — crawler documentation](https://platform.openai.com/docs/bots) — OpenAI
- [Google — Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) — Google

## Related tools

- [robots.txt Generator](https://convertto.tech/t/robots-txt-generator): Build a valid robots.txt file with sitemap references and crawler rules.
- [llms.txt Generator](https://convertto.tech/t/llms-txt-generator): Build a valid llms.txt: a curated, Markdown map of your site written for a model with a limited context window.
- [X (Twitter) Card Validator](https://convertto.tech/t/twitter-card-validator): Paste your meta tags to check the X card type, title, description and image against the documented rules, including Open Graph fallbacks.
- [Keyword Density Checker](https://convertto.tech/t/keyword-density-checker): Analyse which words and phrases dominate your content, and how often they appear.
- [Meta Tag Generator](https://convertto.tech/t/meta-tag-generator): Generate title, description, Open Graph and Twitter Card tags for any page.
- [Schema Markup Generator](https://convertto.tech/t/schema-markup-generator): Generate JSON-LD structured data for organisations, articles, FAQs, products and more.
- [Google SERP Preview](https://convertto.tech/t/serp-preview): Preview how your page title and description will appear in Google search results.
- [XML Sitemap Generator](https://convertto.tech/t/sitemap-generator): Turn a list of URLs into a valid XML sitemap ready to submit to search engines.
- [UTM Link Builder](https://convertto.tech/t/utm-link-builder): Build tagged campaign URLs that track correctly in Google Analytics.