All Posts Tagged Tag: ‘robots.txt’

Google reminds webmasters about robot invasion

Users of Google’s Webmaster Central tools have access to an effective module for creating robots.txt files for their sites.

Read More

Much ado about Delicious robots

Bots arriving from Google,, and MSN to sample pages on the bookmarking site Delicious hit a robots.txt block. As do Yahoo’s Slurp bots too. It’s no big deal.

Read More