robots.txt: allow/disallow rules, common mistakes, and how to test
1. What robots.txt does and does not control
robots.txt is a plain text file in your site’s root directory that tells search engine crawlers which paths they may crawl and which they should leave alone. It is a gentleman’s agreement: well-behaved crawlers (Googlebot, Bingbot) follow it, while rogue scrapers ignore the file entirely.
The most important thing to understand: robots.txt controls crawling, not indexing. The two are often confused. If you Disallow a path, Googlebot will not crawl pages under it, but if other sites link to those URLs, Google still knows they exist and will show the URLs themselves in search results, with no title, no description, and no content snippet, just a bare link.
| Common Misconceptions | Reality | Consequence |
|---|---|---|
| Disallowed means it won’t appear in the SERP | Linked URLs still appear in the SERP | Bare URL, no title, no description |
| robots.txt can keep a page out of the index | No, use noindex to block indexing | The page still shows up in search results |
| robots.txt keeps competitors from seeing your site | Anyone can visit /robots.txt | It’s public information anyway |
| More rules in robots.txt is better | One mistake and your whole site disappears | Keep the configuration minimal |
| robots.txt protects sensitive pages | It’s a gentleman’s agreement; malicious crawlers ignore it | Protect sensitive pages with a password |
How it relates to other SEO modules: robots.txt is Google SEO the lowest layer of your technical foundation, and it works with XML Sitemap optimization Declare your Sitemap location at the end of robots.txt so Google can discover the Sitemap through it. And Technical SEO audit checklist Relationship: robots.txt is the first check in the crawl quadrant.
2. Rule syntax: User-agent, Disallow, Allow, Sitemap
The syntax of robots.txt is simple: four core directives. But simple doesn’t mean you can’t get it wrong—the direction of a single slash or the position of a single asterisk can change what a rule means.
| Directive | Example | Role | What to watch for |
|---|---|---|---|
| User-agent | User-agent: * | Specifies which crawler the rules apply to | * means all crawlers and must be the first line of the rule group |
| Disallow | Disallow: /admin/ | Blocks crawling of all pages under that path | A trailing slash means the entire directory |
| Allow | Allow: /admin/public/ | Allows crawling of a subpath covered by a higher-level Disallow | Used for fine-grained overrides; not all crawlers support it |
| Sitemap | Sitemap: https://… | Declares the location of your Sitemap | Placed at the end of the file, separate from the rule groups |
| Wildcard * | Disallow: /*?sort= | Matches any sequence of characters | Used to block URLs with parameters |
| Trailing $ | Disallow: /test.html$ | Matches URLs ending with that exact string | Used to block a single file precisely |
What should a standard B2B site’s robots.txt look like? The core setup is at the bottom: start with User-agent: *, use Disallow to block functional paths like admin, cart and checkout, block filter and sort URLs with parameters, and declare the Sitemap location at the end. Write no rules for product pages, article pages or category pages—they’re crawlable by default.
Example 1: An ecommerce B2B site wrote Disallow: /products/ in its robots.txt, intending to block the paginated versions of its product listing pages, but ended up blocking all product pages. Three weeks later, when they searched Google for their brand name, the SERP showed only the homepage and the About Us page—every product page was gone. The fix: change the rule to Disallow: /products/?page=, which blocks only URLs with pagination parameters and not /products/ itself. Verify each rule with the GSC robots testing tool before going live.
3. Two fatal mistakes: blocking the whole site and using the wrong tool
robots.txt mistakes range from minor to major, but two of them are fatal—make either one and your entire site disappears from search results.
| Error type | Symptoms | Detection Signals | How to fix | Recovery time |
|---|---|---|---|---|
| Disallow: / blocks the whole site | Google crawls no pages at all | GSC coverage “Crawled – currently not indexed” spikes site-wide | Delete Disallow: / and keep the necessary rules | Recrawled in 1 to 3 days, rankings recover in 1 to 2 weeks |
| Using robots.txt to block indexing | Pages appear in the SERP with no title | The SERP shows a bare URL with no snippet | Switch to the noindex meta tag | Disappears from the SERP in 2 to 4 weeks |
| Wrong path in the rule | What should be blocked isn’t, and what should be allowed is blocked | Abnormal crawl stats in GSC | Verify each rule with the testing tool | Takes effect as soon as you save |
| Forgetting to declare the Sitemap | Google discovers the Sitemap slowly | GSC Sitemap report shows “Not submitted” | Add a Sitemap: directive at the end of robots.txt | Found on the next crawl |
Here’s the counterintuitive part: many people think that to keep a page out of search results, they should write Disallow. That’s wrong. Disallow only stops Google from crawling the content — Google still knows the URL exists, because it learned about it from backlinks, from the sitemap, and from crawl history. The result: a bare URL shows up in the SERP with no title, no description, no snippet. Users click through and hit a 403 or a blank page. Terrible experience, and a wasted impression.
The right approach: if you don’t want a page indexed, add a noindex meta tag to that page (<meta name=”robots” content=”noindex”>), and do not block it in robots.txt — because noindex only works if Google can crawl the page. If you both Disallow it and add noindex, Google can’t crawl the page, so it can’t read the noindex tag. It won’t index the page, but it won’t deindex it either. You lose on both fronts.
Scenario two: a B2B site has a quote request download page (/pricing/quote-request/) that it doesn’t want indexed, since it’s a form page with no keywords to rank for. The ops team wrote Disallow: /pricing/quote-request/. Three months later, the URL showed up in the SERP — searching “quote request” returned a bare URL with no title. Several industry directory sites had linked to the page. The fix: delete the Disallow line from robots.txt and add a noindex meta tag in the page head instead. Two weeks later, the URL was gone from the SERP.
4. Testing and launch workflow
robots.txt takes effect the moment you save it — there’s no cache period. Google reads the new version on its next visit. That means a mistake goes live immediately. So every change has to be verified.
| Steps | Action | Tools | Pass criteria |
|---|---|---|---|
| 1. Edit locally | Edit the robots.txt content | Text editor | Syntax is correct, no stray spaces |
| 2. Verify in the browser | Go directly to https://domain/robots.txt | Browser | Returns 200, content is plain text |
| 3. Verify with the GSC testing tool | Enter the target URL and check whether it’s allowed | GSC robots testing tool | All important pages are Allow |
| 4. Monitor after launch | Watch GSC crawl stats and the coverage report | GSC | Crawl volume doesn’t drop unexpectedly |
| 5. Recheck after a redesign | Recheck robots.txt after every redesign | GSC + browser | New paths aren’t blocked by mistake |
Checklist: every time you edit robots.txt, test at least these 5 URLs: homepage (/), product listing page, product detail page, blog post page, contact page. Confirm all 5 are Allow. If any of them shows Disallow, the rule is wrong — the most common mistakes are a missing space after Disallow: /, or writing Disallow: /products when you meant to block the /products/spec/ subdirectory but wrote Disallow: /products/ instead.
and Site rebuilds and domain changes When you change domains, the old domain’s robots.txt must not Disallow the whole site, because the 301 redirects on the old domain need Googlebot to crawl them in order to pass link equity. If the old domain’s robots.txt says Disallow: /, Google won’t crawl the old pages, so it never sees the 301 directive and the equity transfer breaks. The right approach is to keep the old domain’s robots.txt open to crawling and only block functional paths like admin.
FAQ
What’s the difference between robots.txt and noindex?
What does Disallow: / mean, and why is it dangerous?
How do I test whether my robots.txt is written correctly?
Can robots.txt keep sensitive pages hidden from competitors?
How should I configure the old domain’s robots.txt after changing domains?
When should you use the Allow directive?
Final thoughts
robots.txt is the small file that causes the biggest SEO disasters. It’s only a few lines long, but one wrong slash can make your entire site vanish from search results, and recovery takes one to two weeks. Remember two core principles: robots.txt controls crawling, not indexing. To block indexing, use noindex. And after any change, you must verify with the GSC testing tool.
Three steps to put this into practice: This week, open your robots.txt in a browser and check it against the standard configuration in this article for any Disallow: / or overly broad blocking rules. Next week, use the GSC robots testing tool to verify one by one that all your important pages are Allowed, and add a Sitemap declaration at the end of the file. After that, every time you redesign or change your URL structure, recheck robots.txt right away to confirm new paths aren’t being blocked by mistake.
One line worth remembering: robots.txt is a note to Google asking for leave, not a door lock. Write too much and you’ll lock yourself out. Pages you want to hide need noindex, not Disallow.