What is robots.txt?
robots.txt is a file at the root of a site that tells compliant crawlers which paths they may request. It controls crawling, not indexing — a blocked URL can still be indexed from external links.
Also written as Robots exclusion standard
3 tools that work with robots.txt
Files never leave your browser.
SEO Tools3
AI Crawler robots.txt GeneratorGenerate the robots.txt rules for 21 AI crawlers, separating training from search and from fetches a real user asked for.llms.txt GeneratorBuild a valid llms.txt: a curated, Markdown map of your site written for a model with a limited context window.robots.txt GeneratorBuild a valid robots.txt file with sitemap references and crawler rules.
Related terms
Terms that appear alongside robots.txt on the same tools.