Skip to content
Convertto

Block or Allow AI Crawlers in robots.txt

Generate the robots.txt rules for 21 AI crawlers, separating training from search and from fetches a real user asked for.

This generator produces robots.txt rules for 21 named AI crawlers, grouped by what the crawl is actually for: model training, AI search indexing, or fetching a single page a user asked about. The default blocks training while leaving search and user fetches allowed, because those two send readers rather than only taking content. Path-level rules, allow exceptions and crawl delay are all supported.

Runs in your browser
Crawlers covered
21, across training, AI search and user-initiated fetches
Common mistake
Blocking Google-Extended does not affect Google Search ranking
Enforcement
None — robots.txt is a request that operators choose to honour
Privacy
Runs entirely in your browser — nothing is uploaded
Cost
Free, unlimited, no sign-up

Frequently asked questions

Will blocking Google-Extended hurt my search ranking?

No. Google-Extended controls only whether your content trains Gemini or is used for grounding. Google Search crawling and ranking is Googlebot, a separate agent with separate rules. This is the single most common misunderstanding in AI robots.txt.

Should I block AI search crawlers?

Usually not. They index you so an assistant can cite and link you, which is the same bargain as a search engine. Blocking them is the equivalent of blocking Googlebot — defensible if you have a reason, but it costs you referrals from a growing share of how people find things.

Does robots.txt actually stop anyone?

It stops operators who choose to honour it, which includes OpenAI, Anthropic, Google and Apple. It does not stop anyone else, and several crawlers are widely reported as ignoring it. If content genuinely must not be taken, put it behind authentication — a text file is a request, not a control.

How to use the ai crawler robots.txt generator

  1. 1Choose the policy.
  2. 2Select one or more block these.
  3. 3Enter or paste your block only these paths.
  4. 4Enter or paste your but always allow these.
  5. 5Choose the crawl delay.
  6. 6Enter or paste your sitemap url.
  7. 7Enter or paste your existing robots.txt.
  8. 8The result appears immediately — copy or download it.

Sources & specifications

Embed this tool

Put the working ai crawler robots.txt generator on your own site. It runs in your visitors' browsers exactly as it does here — free, no account, nothing uploaded.

Share this tool

Related tools

robots.txt GeneratorBuild a valid robots.txt file with sitemap references and crawler rules.llms.txt GeneratorBuild a valid llms.txt: a curated, Markdown map of your site written for a model with a limited context window.X (Twitter) Card ValidatorPaste your meta tags to check the X card type, title, description and image against the documented rules, including Open Graph fallbacks.Keyword Density CheckerAnalyse which words and phrases dominate your content, and how often they appear.Meta Tag GeneratorGenerate title, description, Open Graph and Twitter Card tags for any page.Schema Markup GeneratorGenerate JSON-LD structured data for organisations, articles, FAQs, products and more.

Last updated

More seo tools