A site owner launches a redesign, waits for search engines to pick up the new pages, and then notices the important product pages are not being crawled. The culprit is one line in a public text file: Disallow: /products/. Nobody meant to block the money pages. The rule was copied from staging and never removed.
Robots.txt is small, plain, and easy to underestimate. It gives crawler guidance about which paths they may fetch. BlinkCalc's Robots.txt Generator can help create a clean starting file, but the safest users understand what the rules mean before uploading them.
SEO note: outcomes are not guaranteed. Search engines may interpret rules differently, and indexing depends on many factors beyond robots.txt.
What robots.txt is
Robots.txt is a plain text file that tells crawlers which parts of a site they are allowed or not allowed to crawl. It is part of the Robots Exclusion Protocol, a long-standing convention used by many search engines and bots.
The file does not contain HTML. It is made of directives such as User-agent, Disallow, Allow, and Sitemap. A basic file might look like this:
User-agent: *
Disallow: /admin/
Sitemap: https://example.test/sitemap.xmlThis says the rule group applies to all user agents, asks them not to crawl /admin/, and points them to a sitemap.
Where robots.txt lives
Robots.txt belongs at the root of the site:
https://example.test/robots.txtIf it is placed at /blog/robots.txt, it will not control the whole site. Crawlers look for the root file for that host and protocol. A subdomain has its own robots file. https://shop.example.test/robots.txt and https://www.example.test/robots.txt are separate locations.
Before editing crawl rules, use the Website Status Checker to confirm the robots file returns the expected status. A missing file, redirect, or server error can change crawler behavior.
User-agent lines
The User-agent line names the crawler the rules apply to. User-agent: * means all crawlers that follow the convention. You can also target specific crawlers:
User-agent: ExampleBot
Disallow: /private-drafts/Separate groups can define different rules for different bots. Keep groups clear. If rules become complicated, comments and testing matter because one misplaced line can change the rule group that follows.
Not every crawler is polite. Bad bots can ignore robots.txt. Do not use the file as a security boundary.
Disallow and Allow
Disallow asks crawlers not to crawl matching paths. Allow can permit a more specific path inside a broader disallowed area, depending on crawler support.
Example:
User-agent: *
Disallow: /account/
Allow: /account/help/This asks crawlers to avoid /account/ but allows /account/help/. Specificity matters. Search engines often choose the most specific matching rule, but details can vary.
An empty Disallow: line usually means nothing is blocked:
User-agent: *
Disallow:That is very different from Disallow: /, which asks crawlers not to crawl the entire site.
Sitemap line
The Sitemap directive points crawlers to a sitemap URL:
Sitemap: https://example.test/sitemap.xmlA sitemap lists URLs you want search engines to discover. It does not force indexing, but it helps crawlers find canonical pages, updated content, and large site structures.
Including a sitemap in robots.txt is a common convenience because crawlers often fetch robots.txt early. You can include more than one sitemap line if your site has multiple sitemap files.
Crawl control vs indexing control
Robots.txt controls crawling, not guaranteed indexing. If a page is blocked from crawling, a search engine may still know the URL exists from links, sitemaps, or external references. It may show a limited result without content.
If you want a page not to appear in search results, a noindex directive in the page's meta robots tag or HTTP header is usually the relevant indexing signal. But crawlers must be able to access the page to see that directive. Blocking it in robots.txt can prevent them from reading the noindex tag.
Use the Meta Tag Analyzer to check whether a page has meta robots tags, canonical tags, titles, and descriptions that match your intent.
Common examples
A permissive robots file:
User-agent: *
Disallow:
Sitemap: https://example.test/sitemap.xmlBlock internal search pages:
User-agent: *
Disallow: /searchBlock staging paths:
User-agent: *
Disallow: /staging/
Disallow: /tmp/Block the whole site, often used accidentally:
User-agent: *
Disallow: /That last example is powerful. It can be useful on staging if combined with real access controls, but it can harm crawling on a public production site.
Worked example for a small site
Imagine a small site with public pages, a blog, account pages, and internal search results. A reasonable starting file might be:
User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /search
Sitemap: https://smallsite.example/sitemap.xmlThis lets crawlers fetch public pages and blog posts while avoiding account, checkout, and search-result URLs. It does not protect account pages from people or malicious bots. Those pages still need authentication.
If the site also has a blog tag page that should be indexed, avoid broad rules like Disallow: /tag unless you are sure. Small path differences matter.
How to use the Robots.txt Generator
Open the Robots.txt Generator, choose the paths you want to block, add your sitemap URL, and copy the result. Read the output line by line before uploading it.
After publishing, visit /robots.txt in a browser and confirm the file appears exactly as expected. Then test important URLs with your search engine tools when available. A generator helps with syntax, but deployment mistakes can still happen through redirects, caching, or uploading to the wrong host.
If you are also creating meta robots tags for specific pages, the Meta Tag Generator can help draft those tags separately from the crawl rules.
Staging, previews, and private areas
Robots.txt is often used on staging sites, but it should not be the only protection. A staging site can still be visited by anyone who knows the URL if there is no authentication. The robots file is public, and a bad crawler can ignore it.
For staging, use real access control such as HTTP authentication, VPN access, platform password protection, or IP allowlists where appropriate. A Disallow: / rule can be an extra signal for polite crawlers, but it is not a lock.
Preview deployments create a similar issue. Teams may generate temporary URLs for every pull request. If those previews contain draft content, customer examples, test data, or unfinished pages, rely on platform controls rather than hoping crawlers behave.
Crawl budget and low-value URLs
Large sites think about crawl budget: how much crawler attention is spent on useful pages versus duplicates, filters, search results, and endless combinations. Small sites rarely need complicated crawl-budget strategy, but they can still avoid obvious waste.
Internal search results, calendar pages with infinite date combinations, faceted filters, sort orders, and tracking URLs can create many low-value URLs. Robots.txt may help guide crawlers away from some of those paths. Canonical tags, parameter handling, internal linking, and sitemap hygiene also matter.
Do not block a path simply because it has many URLs. First decide whether those URLs should be crawled, indexed, canonicalized, consolidated, or removed. Robots.txt is only one tool in that decision.
Testing before launch
Before launching a robots file, list your most important URL types: homepage, category pages, product or article pages, static assets, sitemap, checkout, account, internal search, and admin paths. Then check whether each should be crawlable.
For each rule, test a URL that should be blocked and a similar URL that should not be blocked. If you disallow /search, check /search?q=test. Also check unrelated paths such as /search-engine-guide if your matching pattern could be broad.
Launch checklists should include robots.txt. It is easy to remember titles and analytics while forgetting that one small file can change crawler access across the site.
Reading robots.txt like a crawler
Read from top to bottom in groups. A User-agent line starts a group. The following Allow and Disallow lines belong to that group until the next group begins. Comments usually start with #.
Path matching is literal enough that small characters matter. /admin and /admin/ may not be equivalent in every situation. Uppercase and lowercase paths can differ on many servers. A trailing slash can change which URLs match.
When rules are complex, simplify them. A robots file should be boring. If you need a diagram to understand it, future editors are more likely to make a mistake.
Robots.txt and site migrations
Site migrations are a risky moment for robots.txt. Teams may move from staging to production, switch domains, add a CDN, change URL paths, and publish new sitemaps at the same time. A stale robots file can quietly undermine the launch.
Before migration, compare old and new robots files. Check that old blocked paths are still relevant and that new important sections are crawlable. Update sitemap URLs to the new canonical host. If the site moves from non-www to www, or from one domain to another, confirm the robots file exists on the final host.
After launch, fetch /robots.txt, fetch the sitemap, and test several important page URLs. Search engine webmaster tools can provide crawler-specific testing, but a manual browser check catches obvious deployment errors quickly.
How robots interacts with internal links
Robots.txt can stop crawling of a path, but your internal links still matter. If a blocked path is linked heavily from menus, footers, sitemaps, and articles, you are sending mixed signals. Crawlers see that the site points to those URLs but then asks them not to crawl.
Clean internal linking reduces confusion. Link prominently to pages you want discovered. Avoid placing blocked utility URLs in public navigation unless users truly need them. Remove blocked URLs from XML sitemaps unless there is a deliberate reason.
This is especially important for filtered category pages. A store may link to dozens of filter combinations. Some may be useful landing pages. Others may be crawl traps. Decide which are strategic instead of blocking everything after the fact.
Examples of subtle allow and disallow issues
Consider this rule:
Disallow: /blogDepending on crawler interpretation, it may match /blog, /blog/, and paths beginning with /blog, such as /blog-post-archive. If you meant only the blog directory, /blog/ may be clearer.
Another example:
Disallow: /*?sort=Wildcard support is common in major search engines but not part of the earliest simple mental model many beginners learn. A smaller site may be better served by canonical tags and clean internal links than by complex wildcard rules.
The more advanced the pattern, the more testing it deserves. Robots syntax is short, but the consequences can cover the whole site.
A launch-day robots checklist
On launch day, check that /robots.txt returns 200, not a login page, redirect loop, or branded error page. Confirm it is served as plain text. Confirm Disallow: / is not present unless the whole site truly should be blocked. Confirm sitemap URLs use the final canonical host.
Then test crawl access for representative URLs: homepage, article page, tool page, category page, image or CSS file, and any area intentionally blocked. If the public pages are crawlable and private or low-value paths are blocked as intended, the file is doing its job.
Keep a copy of the final robots file in source control or deployment notes. When crawling changes months later, you will want to know what changed and when.
Robots.txt for assets
Search engines may need CSS, JavaScript, and image files to understand a page. Blocking /assets/ or /static/ can make rendering harder, especially for modern sites where content and layout depend on bundled files.
Before blocking asset folders, ask why. If the files are public resources needed by public pages, crawling them is usually acceptable. If the folder contains private exports, the problem is access control and file placement, not only robots rules.
For image-heavy sites, image crawling may also affect discovery in image search and rich previews. Decide deliberately instead of copying old rules from unrelated sites or outdated launch templates.
Common mistakes
One mistake is using robots.txt to hide private content. The file is public, and it can reveal paths you would rather not advertise. Use authentication and authorization for private areas.
Another mistake is copying staging rules to production. Disallow: / is a common launch accident.
A third mistake is blocking CSS, JavaScript, or images that search engines need to render pages. Modern crawlers often need page resources to understand layout and content.
A fourth mistake is assuming noindex works on blocked pages. If the crawler cannot fetch the page, it may not see the noindex directive.
A fifth mistake is writing broad path rules without checking similar URLs. Disallow: /app may affect /apple-guide depending on matching behavior.
FAQ
What is robots.txt?
Robots.txt is a public text file at the root of a site that gives crawl guidance to search engine bots and other crawlers.
Where should robots.txt be placed?
Place it at the root of the host, such as https://example.test/robots.txt. Subdomains and different hosts need their own files.
Does robots.txt stop pages from being indexed?
Not reliably. It controls crawling. Indexing is a separate process and can be influenced by noindex tags, canonical signals, links, sitemaps, and search engine behavior.
Is robots.txt a privacy feature?
No. It is public and voluntary. Sensitive content should be protected with authentication, authorization, and correct server controls.
Should robots.txt include a sitemap?
Often yes. A sitemap line helps crawlers discover the sitemap location, although it does not guarantee crawling or indexing.
Can search engines ignore robots.txt?
Major search engines usually follow it, but not every crawler does. Malicious or poorly built bots can ignore the file.