WebProNews

At the EclipseCon 2005 conference, a noted Google engineer offered a rare look at how Google approaches their method of operations during a keynote speech.

The speaker, vice president of engineering, Urs Hoelzle, credited batch computing as the secret sauce that drives Google. A report at InformationWeek.com offered a thorough review of Hoelzle's keynote:

To cope with outages of a variable nature, Google built the Google File System, which was closely geared to Google's search computing tasks and had a high tolerance for server failures.

Google operations are built around large files that are broken down into 64-Mbyte chunks and scattered across multiple "chunk servers." A description of each file, its number of chunks, and chunk locations are kept on a master server. Each 64-Mbyte chunk is also replicated on two other servers, so a total of three copies are kept with the path to each retained by the master server.

By scattering its files across many Red Hat Linux servers, Google gains reliability at a low cost. The master server regularly polls chunk servers with a heartbeat message, asking if they're alive. If it fails to get an answer, or if a quick check of the contents on a server indicates that its data has been corrupted, the master server sets about creating a new 64-Mbyte chunk on another server, "usually in a matter of minutes," Hoelzle explained.

Read the full story at InformationWeek.

WebProNews | Breaking eBusiness News
Your source for investigative ebusiness reporting and breaking news.