Robots.txt Generator: Create Basic Crawl Instructions

Enter a sitemap URL and disallowed paths, one per line. The generator creates a basic User-agent: * block with Allow: / and your Disallow rules. Upload the reviewed file to the root of the correct hostname as /robots.txt.

What to check before publishing

Robots.txt controls crawler access; it does not reliably remove a URL from search results or protect private information. A broad rule can block important pages, so inspect each path and test the published file. The generated rules apply to compliant crawlers and are deliberately simple.

A simple workflow

Check the live file after upload and submit the sitemap in your search console. Build a URL-list sitemap with the XML Sitemap Generator if you need one.

Try the free Robots Txt Generator tool. Related tool: Xml Sitemap Generator.

Related SEO guides

Browse all SEO guides

Scroll to Top