Google Slows Down Crawling So SOPA-Protesting Sites Don’t Get Hurt
A lot of sites have gone dark today in protest of SOPA, and many others are protesting in other ways, with banners, and whatnot. Google, protesting SOPA itself, is slowing down crawling for today to help out those protesting.
“We realize many webmasters are concerned about the medium-term effects of today’s blackout,” said Google’s Pierre Far on Google+. “As a precaution, the crawl team at Google has configured Googlebot to crawl at a much lower rate for today only so that the Google results of websites participating in the blackout are less likely to be affected.”
A couple days ago, Far provided a how-to guide for webmasters looking to join the protest without hurting how Google sees their sites.
The most important of these tips, he said, is: “Webmasters should return a 503 HTTP header for all the URLs participating in the blackout (parts of a site or the whole site).”