Robots.txt Generator
Generate a valid robots.txt file for your website with common presets.
Robots.txt builder
Create crawler rules and sitemap references
Choose a default crawler policy, add Allow or Disallow paths, include a sitemap URL, and copy or download the file.
Rule builder
Add crawler-specific Allow or Disallow paths.
Robots.txt is a crawler directive, not a security feature.
Generated preview
Review before uploading to your website root.
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xmlGenerated status
Generated
Place this file at https://example.com/robots.txt.
Status
Generated
Robots.txt is a crawler directive, not a security feature.
Default policy
block selected
Selected crawler policy.
User-agent count
1
Crawler groups in generated rules.
Allow rules
1
Explicit Allow directives.
Disallow rules
2
Explicit Disallow directives.
Sitemap included
Yes
Valid Sitemap directive included.
Risk level
Safe
Review high-impact crawl rules before deployment.
Copy robots.txt
Copy the generated crawler directives for deployment or documentation.
Download robots.txt
Download the generated content as a robots.txt file.
Privacy note
Robots.txt content is generated locally in your browser without external APIs.
SEO note
Robots.txt affects crawler behaviour, but it does not guarantee indexing outcomes.
Security note
Do not use robots.txt to hide private content. Robots.txt files are public.
Accuracy note
Test generated rules before deploying them to production.
Generated robots.txt output
Place this file at https://example.com/robots.txt.
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xmlPractical robots.txt examples
Click an example to load a preset into the generator.
Robots.txt quick reference
User-agent
Identifies a crawler group.
Disallow
Asks crawlers not to crawl a path.
Allow
Explicitly allows a path.
Sitemap
Points crawlers to sitemap XML.
User-agent: *
Applies to all crawlers.
Disallow: /
Blocks all crawling.
Allow: /
Allows crawling.
Public file
Robots.txt is publicly visible.
Not security
Robots.txt is not authentication or access control.
SEO and crawler guide
Understand robots.txt before publishing
Robots.txt gives crawlers guidance about which parts of a website they may crawl, but it is not a privacy, security, or indexing guarantee.
What is a Robots.txt Generator?
A Robots.txt Generator creates a plain text file that gives search engine crawlers instructions about which parts of a website they may crawl.
What is robots.txt?
Robots.txt is a plain text file placed at the root of a site, usually at /robots.txt. It uses rules such as User-agent, Disallow, Allow, and Sitemap.
When should developers use one?
Use it when launching a site, blocking low-value crawl paths, pointing crawlers to a sitemap, reducing crawl waste on search or filter pages, managing admin paths, or creating crawler-specific rules.
Robots.txt limitations
Robots.txt is a crawl directive, not access control. It does not hide private content, remove already indexed URLs by itself, or guarantee every bot will obey it.
Robots.txt vs noindex
Robots.txt controls crawling. Noindex controls indexing when crawlers can access the page. If a page is blocked in robots.txt, search engines may not see its noindex tag.
Privacy and security
Robots.txt files are public. Do not list sensitive private paths as a security measure. Use authentication or server restrictions for private content.
Common robots.txt mistakes
How to use this Robots.txt Generator
- 1Enter your site URL and optional sitemap URL.
- 2Choose a default crawler policy.
- 3Add Allow or Disallow rules for paths such as /admin/ or /search/.
- 4Review the generated robots.txt output and warnings.
- 5Copy or download the file and place it at yourdomain.com/robots.txt.
- 6Test the file before deploying to production.
Why developers use a Robots.txt Generator
Robots.txt Generator FAQs
It creates a robots.txt file with crawler directives such as User-agent, Allow, Disallow, and Sitemap.
Related tools