Quantcast

robots.txt Articles

Google reminds webmasters about robot invasion

Users of Google’s Webmaster Central tools have access to an effective module for creating robots.txt files for their sites.

Much ado about Delicious robots

Bots arriving from Google, Ask.com, and MSN to sample pages on the bookmarking site Delicious hit a robots.txt block. As do Yahoo’s Slurp bots too. It’s no big deal.