Unvalidated Robots.Txt Risks Google Banishment

By WebProNews Staff - Wed, 11/21/2007 - 07:53

The web crawling Googlebot may find a forgotten line in robots.txt that causes it to de-index a site from the search engine.

Unvalidated Robots.Txt Risks Google Banishment

Webmasters welcome being dropped out of Google about as much as they enjoy flossing with barbed wire. Making it easier for Google to do that would be anathema to being a webmaster. Why willingly exclude one's site from Google?

That could happen with an unvalidated robots.txt file. Robots.txt allows webmasters to provide standing instructions to visiting spiders, which contributes to having a site indexed faster and more accurately.

Google has been considering new syntax to recognize within robots.txt. The Sebastians-Pamphlets blog said Google confirmed recognizing experimental syntax like Noindex in the robots.txt file.

This poses a danger to webmasters who have not validated their robots.txt. A line reading Noindex: / could lead to one's site being completely de-indexed.

The surname-less Sebastian recommended Google's robots.txt analyzer, part of Google's Webmaster Tools, and only using the Disallow, Allow, and Sitemaps crawler directives in the Googlebot section of robots.txt.

follow me on Twitter

News Tags: Google, Robots.txt, index, Spider, Crawler, Twitter

3 Comments

By Sebastian (WPN reader) - Sun, 11/25/2007 - 06:21

Experimental robots.txt syntax

Thanks for the coverage Dave. :)

Joshua, folks not familiar with the REP syntax often leave garbage or experimental statements in robots.txt. As long as the crawlers ignore those, such forgotten stuff is not a big deal. It becomes risky when a search engine experiments itself, and the engine's interpretation doesn't match the webmaster's thoughts.

By UR Wrong (WPN reader) - Tue, 11/27/2007 - 08:17

robots.txt

When you put noindex is is to be excluded, so this presents no threat. In fact it accomplishes your goal. It's those spiders which ignore your wishes that you need worry about.

By Joshua Spaulding (WPN reader) - Sat, 11/24/2007 - 14:40