"Hadoop is where science meets big data – it’s the technical underpinning that powers our innovative consumer and advertiser products on the world’s most-advanced digital canvas," says Blake Irving, Yahoo Executive Vice President and Chief Product Officer. "Yahoo!’s cloud and Hadoop make it possible for Yahoo! to rapidly personalize our content and advertising, and deliver highly relevant experiences, while maintaining the trust of our 600 million users."

Yahoo says Hadoop plays a key role in its home page, Yahoo Search, Yahoo Mail, and other properties.

"Businesses across all sectors are looking for ways to leverage the vast quantities of data they are accumulating, and Apache Hadoop is an efficient solution for processing data at scale," says Melanie Posey, research director at IDC Research. "Now organizations of various sizes can leverage Yahoo!’s Hadoop investment and deployments to run it on their own systems and build out their own Hadoop deployments without starting from scratch on internal science experiments."

Specifically, Yahoo announced the beta release of Hadoop with Security and Oozie, the company’s workflow engine for Hadoop. This means enterprises will benefit from better controls for managing business-sensitive data, according to the company.

The Yahoo Distribution of Hadoop with Security (beta) and Oozie are available through the Yahoo Developer Network.

Amazon Elastic MapReduce uses Hadoop, a free Java software framework that runs on the company’s Elastic Compute Cloud (EC2) and Simple Storage Service (S3). Customers can use MapReduce to perform data-intensive tasks for distributed applications such as web indexing, data mining, log file analysis, financial analysis and bioinformatics research.

Customers will pay only for what they use with no up-front payments or commitments.

"Some researchers and developers already run Hadoop on Amazon EC2, and many of them have asked for even simpler tools for large-scale data analysis," said Adam Selipsky, Vice President of Product Management and Developer Relations for Amazon Web Services.

"Amazon Elastic MapReduce makes crunching in the cloud much easier as it dramatically reduces the time, effort, complexity and cost of performing data-intensive tasks."

The service automatically launches and configures the number and type of EC2 instances that a customer selects. It then kicks off a Hadoop implementation of the MapReduce programming model, which loads large amounts of user input data from S3 and subdivides it for parallel processing. Users can manage and monitor job flows through web service APIs or via AWS Management Console.
 

Hadoop comes from the Apache Software Foundation, and acts as a distributed computing platform.  Prior to Yahoo’s announcement, the software wasn’t hurting for footing; other users included Google, IBM, and Last.fm.  Still, this development should ensure Hadoop is embraced to an even greater degree in the future.

On the Yahoo Search Blog, Sean Suchter points out, "Using open source software is a win-win situation for Yahoo! and the wider community.  We achieve cost savings, faster processing, reduced maintenance, and increased scale and the community can benefit from the myriad improvements it took to make Hadoop viable for such a large-scale commercial implementation."  Quite an endorsement (and invitation), eh?

Yahoo and Hadoop are also taking steps to make sure everyone hears the news, with related posts appearing on the Yahoo Developer Network and Hadoop and Distributed Computing blogs.  An interview between Jeremy Zawodny and two team members adds to the buzz.

But as Zawodny reminds everyone, there’s more to this than PR.  "It’s not just an experiment or research project," he writes.  "There’s real money on the line."  And with their record-breaking implementation, Yahoo and Hadoop seem to be handling it well.

He writes, "To make things a bit easier, we decided to start this Hadoop and Distributed Computing blog on the Yahoo! Developer Network as a place to write about Hadoop and our distributed computing work on a more regular basis. There’s a lot going on (we already have a backlog of posts!) and we’re anxious to get the word out."

Currently featured on the new blog is a video interview Zawodny does with Eric Baldeschwieler about Hadoop and Yahoo.

For those of you firing up the old faithful laptop for a morning of surfing, blogging, maybe a little development work, get a load of what some of the lucky geeks at Carnegie Mellon University got to play with this morning:

The M45, Yahoo’s supercomputing cluster, has approximately 4,000 processors, three terabytes of memory, 1.5 petabytes of disks, and a peak performance of more than 27 trillion calculations per second (27 teraflops), placing it among the top 50 fastest supercomputers in the world.

Their ranking claim won’t be confirmed until the next Top500 Supercomputer list comes out on Tuesday at this week’s SC07 conference in Reno, so it will be interesting to see how M45 measures against the best in the world. Yahoo’s M45 figures should put it in the top 30.

We chatted with Yahoo’s Ron Brachman, VP for worldwide research operations with the company. He’s also wearing the hat as head of academic relationships. Jay Kistler, VP for engineering system tools & services, also talked with us ahead of this morning’s announcement.

Brachman said the M45 supercomputer came about from the opportunity for Yahoo and the university community to advance science and technology on an Internet scale. They have opted to focus on open source, developing solutions for large scale distributed computing.

Yahoo and Carnegie Mellon understand grid computing well. The M45 setup has been geared toward that understanding. It’s capable of partitioning large data sets thanks to the installation of Hadoop.

Hadoop accomplishes this by implementing MapReduce and Pig the latter which may be known to those who follow Yahoo’s research projects closely.

Kistler said they have been working on layering Pig over a Hadoop core. Pig’s runtime extensions for parallel computing are similar to SQL, but they are procedural rather than declarative.

In the M45 environment, the runtime maps statements down to where MapReduce can divide them into little blocks of work and run them across the supercomputing platform.

We wanted to understand better what the distributed development effort being enabled by M45 might be able to do for this level of supercomputing. Kistler rattled off a couple of achievements he would like to see happen, if developers can pull them off.

One would provide for the improvement of job scheduling across clusters; another the enhancement of monitoring and instrumentation of heterogeneous jobs, where it would be easier to find bottlenecks and faults, and correct them for better performance.

Compelling stuff for the folks who will really get into the tasty innards of supercomputing. The potential gains from the M45 go beyond the items on Kistler’s wish list.

Carnegie Mellon’s Randy Bryant, dean of the School of Computer Science, told us in a phone interview about such possibilities. Top of the list: generating statistics for language translation. It’s a demanding task due to the number of documents needed for mapping words from multiple languages.

Another potential gain would be with digital image editing. Bryant discussed this with the example of getting an ex-brother in law out of photos. Through the use of a massive digital image database, supercomputing could allow the editor to find the content of a photo minus the person to be edited out, and replace that person with the background that would be normally visible.

Semantics and language search support would benefit, and we think Yahoo will be interested in that. Bryant noted such a project would look at distinguishing linguistics, where the system would understand when a speaker means "bare" or "bear" from the context of the rest of a conversation.

Research takes time, but the M45 platform should substantially improve the total time needed for these projects to bear productive results. Some very lucky geek types started researching on this platform today.

follow me on Twitter