Give The Googlebot A Break Already

Got one of those nifty calendars on your site that expands month after month with the click of a mouse? Be kind to the Googlebot and slap a nofollow tag on the link—he might just follow it forever!

Well, Torrey Hoffman says the Googlebot isn’t quite that stupid, but he’s close. The calendar conundrum is a classic one the bot should be already familiar with and knows not to keep clicking that button like a heroin-addicted rat. It’s also a classic example of what the search engineers at Google call an "infinite space."

If you don’t know what else qualifies as an infinite space, the Webmaster Central team has already started notifying webmasters about the problem when it occurs. The team recommends verifying your site at Webmaster Tools and checking the Message Center.

Hoffman mentions another scenario where a website will provide filtering for a set of product search results, dividing items according to color, brand, price, etc, resulting in thousands of automatically generated URLs for the bot to follow. Again, think of a nofollow tag to give the robot a break, or follow other guidelines provided by the Webmaster Help Center.