Google Slows Down Crawling So SOPA-Protesting Sites Don’t Get Hurt

    January 18, 2012
    Chris Crum

A lot of sites have gone dark today in protest of SOPA, and many others are protesting in other ways, with banners, and whatnot. Google, protesting SOPA itself, is slowing down crawling for today to help out those protesting.

“We realize many webmasters are concerned about the medium-term effects of today’s blackout,” said Google’s Pierre Far on Google+. “As a precaution, the crawl team at Google has configured Googlebot to crawl at a much lower rate for today only so that the Google results of websites participating in the blackout are less likely to be affected.”

A couple days ago, Far provided a how-to guide for webmasters looking to join the protest without hurting how Google sees their sites.

The most important of these tips, he said, is: “Webmasters should return a 503 HTTP header for all the URLs participating in the blackout (parts of a site or the whole site).”


Chris Crum
Chris Crum has been a part of the WebProNews team and the iEntry Network of B2B Publications since 2003. Follow Chris on Twitter, on StumbleUpon, on Pinterest and/or on Google: +Chris Crum.