Though it has been thought there may be somewhat of a grace period, it now seems robots only notice what’s there when they arrive.

SEORoundtable points us to a webmaster struggling with a hosting company’s reliability raised the question in a Google Groups thread about why the webmaster’s well-ranking site had disappeared from the search results.

Google’s Vanessa Fox responded verifying that the site had indeed been de-listed:

If the host is down when Googlebot tries to access your pages, then those pages may disappear from the index until Googlebot can crawl them again. . .

Googlebot can’t know if or when a page will return when it gets an error response (whether that’s a network down, 404, or other error) and since our primary aim is to have quality search results for users, we don’t want to keep pages that return these types of errors in our index.

The best way of handling these situations, therefore, is to remove the pages, but continue to try to access them, then return them to the index once we get a valid server response.

How long it takes before a site to de-listed is unclear, but Fox confirmed that Googlebot would make a few passes before declaring it dead.

This is not a case for a re-inclusion request, however, as once the site is back up and running, it will be crawled again – though no mention was made as to whether rankings would be restored.

"[I]t does make sense for Google to remove the site, because it may never come back," said a commentator who really, really wants you to know his name is Vince.

Matt Cutts chimes in and agrees with the Vincemeister:

VinceVinceVince makes a good point: sometimes temporarily down pages turn into truly-gone-forever pages, so we have to drop those pages at some point.

But it’s also true that we go back and revisit those pages pretty often and try to re-crawl them in case the site comes back up.

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews: