5 robots.txt mistakes that block a whole section

Most robots.txt accidents aren't typos. They're rules that do exactly what Google's matching logic says, which is not what the author meant. These five turn up again and again.

1. A new Googlebot group that silently drops the shared rules. Someone adds User-agent: Googlebot with two lines for search and cart pages. From that moment Googlebot reads only those two lines and ignores everything in the * group, including the rules that kept filters and print pages out of the crawl. Fix: repeat the shared rules inside the Googlebot group, or don't create a separate group at all.

2. An allow rule that is longer than the disallow it was supposed to sit under. Allow: /catalog/*/ (11 characters) beats Disallow: /*?filter= (10 characters), so every filtered catalog URL stays crawlable. Google applies the longest matching rule, not the first one in the file. Fix: make the disallow at least as specific, such as Disallow: /catalog/*?filter=.

3. A staging file deployed to production. Disallow: / is the right file for a staging server and the most expensive one-line mistake on a live site. Keep separate files per environment and test the production file before every deploy.

4. Case and trailing characters. Disallow: /Admin/ doesn't block /admin/, and Disallow: /*.pdf$ doesn't block /file.pdf?v=2. Paths are case-sensitive and $ means the URL must end there.

5. Treating Disallow as a way to remove pages. A blocked URL isn't crawled, so Google never sees its noindex, and the address can stay in results if other pages link to it. To drop a page from the index, allow crawling and use noindex.

Catch them before they ship

Every one of these is visible if you test a list of real URLs against the draft file instead of reading the rules by eye. The step-by-step method, a worked example and a small Python checker are in this guide on testing robots.txt rules before you publish.

For a quick paste-and-check in the browser, IndexFix Tools has a free robots.txt tester among its indexing tools.