Skip to content
Convertto

robots.txt Generator

Build a valid robots.txt file with sitemap references and crawler rules.

To generate a robots.txt, choose whether crawlers are allowed by default and list any paths to disallow. The file is written to the standard format with your sitemap reference included, plus optional rules for AI training crawlers.

Runs in your browser
Privacy
Runs entirely in your browser — nothing is uploaded
Cost
Free, unlimited, no sign-up

Frequently asked questions

Does robots.txt stop a page being indexed?

No. It stops crawling, not indexing. A disallowed URL can still be listed in search results — without a snippet — if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag.

Should I block AI crawlers?

It is a trade-off. Blocking prevents your content being used for training, but it also removes you from AI-generated answers that cite sources — an increasingly significant referral channel.

How to use the robots.txt generator

  1. 1Turn "Allow all crawlers by default" on or off as needed.
  2. 2Turn "Block AI training crawlers" on or off as needed.
  3. 3Enter or paste your sitemap url.
  4. 4Enter or paste your paths to disallow (one per line).
  5. 5Set the crawl delay.
  6. 6The result appears immediately — copy or download it.

Embed this tool

Put the working robots.txt generator on your own site. It runs in your visitors' browsers exactly as it does here — free, no account, nothing uploaded.

Share this tool

Related tools

XML Sitemap GeneratorTurn a list of URLs into a valid XML sitemap ready to submit to search engines.AI Crawler robots.txt GeneratorGenerate the robots.txt rules for 21 AI crawlers, separating training from search and from fetches a real user asked for.Meta Tag GeneratorGenerate title, description, Open Graph and Twitter Card tags for any page.Schema Markup GeneratorGenerate JSON-LD structured data for organisations, articles, FAQs, products and more.X (Twitter) Card ValidatorPaste your meta tags to check the X card type, title, description and image against the documented rules, including Open Graph fallbacks.Keyword Density CheckerAnalyse which words and phrases dominate your content, and how often they appear.

Last updated

More seo tools