Google Webmaster Tools Gets Updated Robots.txt Testing Tool

Google has released an updated robots.txt testing tool in Webmaster Tools. The tool can be found in the Crawl section. The aim of the new version of the tool is to make it easier to make and maintain a “correct”…

Google Removing Subscriber Stats Feature From Webmaster Tools

Just like when Google announced the changes coming to API deprecation and some APIs even being retired, Google is now looking at the Webmaster tools it can phase out. The company decides these tools’ fate by seeing “if they’re still…

Google: Not Having Robots.txt is “A Little Bit Risky”

Robots.txt as you may know, lets Googlebot know whether you want it to crawl your site or not. Google’s Matt Cutts spoke about a few options for these files in the latest Webmaster Help video, in response to a user-submitted…

Perfect 10 Fails Where Google Succeeds

Perfect10.com, a site that features incredibly attractive female models in various positions of nude repose has been long after Google because of the site’s content appearing in Google Image Searches. Their struggle has been going on for sometime now. In…

Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt

Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google’s crawler) seems to be having some issues here. In the WebmasterWorld forum, member…

Developer Shares Story of Being Threatened by Facebook for Crawling

Pete Warden, a former software engineer at Apple, who is now working on his own start-up, posted an interesting story about how Facebook threatened to sue him for crawling the social network. I reached out to both Warden and Facebook for more details, but so far have only received response from Facebook, who calls  the incident as "violation of our terms."