# robots.txt Generator

> Build a valid robots.txt file with sitemap references and crawler rules.

To generate a robots.txt, choose whether crawlers are allowed by default and list any paths to disallow. The file is written to the standard format with your sitemap reference included, plus optional rules for AI training crawlers.

**URL:** https://convertto.tech/t/robots-txt-generator
**Category:** SEO Tools (https://convertto.tech/c/seo-tools)
**Privacy:** Runs entirely in the browser; no upload
**Cost:** Free, no sign-up
**Last updated:** 2026-08-01

## Key facts

- **Privacy:** Runs entirely in your browser — nothing is uploaded
- **Cost:** Free, unlimited, no sign-up

## How to use

1. Turn "Allow all crawlers by default" on or off as needed.
2. Turn "Block AI training crawlers" on or off as needed.
3. Enter or paste your sitemap url.
4. Enter or paste your paths to disallow (one per line).
5. Set the crawl delay.
6. The result appears immediately — copy or download it.

## FAQ

### Does robots.txt stop a page being indexed?

No. It stops crawling, not indexing. A disallowed URL can still be listed in search results — without a snippet — if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag.

### Should I block AI crawlers?

It is a trade-off. Blocking prevents your content being used for training, but it also removes you from AI-generated answers that cite sources — an increasingly significant referral channel.

## Related tools

- [XML Sitemap Generator](https://convertto.tech/t/sitemap-generator): Turn a list of URLs into a valid XML sitemap ready to submit to search engines.
- [AI Crawler robots.txt Generator](https://convertto.tech/t/ai-crawler-robots-txt-generator): Generate the robots.txt rules for 21 AI crawlers, separating training from search and from fetches a real user asked for.
- [Meta Tag Generator](https://convertto.tech/t/meta-tag-generator): Generate title, description, Open Graph and Twitter Card tags for any page.
- [Schema Markup Generator](https://convertto.tech/t/schema-markup-generator): Generate JSON-LD structured data for organisations, articles, FAQs, products and more.
- [X (Twitter) Card Validator](https://convertto.tech/t/twitter-card-validator): Paste your meta tags to check the X card type, title, description and image against the documented rules, including Open Graph fallbacks.
- [Keyword Density Checker](https://convertto.tech/t/keyword-density-checker): Analyse which words and phrases dominate your content, and how often they appear.
- [llms.txt Generator](https://convertto.tech/t/llms-txt-generator): Build a valid llms.txt: a curated, Markdown map of your site written for a model with a limited context window.
- [Google SERP Preview](https://convertto.tech/t/serp-preview): Preview how your page title and description will appear in Google search results.