Google Makes Cloud Dataflow, Cloud Pub/Sub Generally Available

Chris CrumTechnology

Share this Post

Screen Shot 2015-08-12 at 3.36.09 PM

Google announced that it has removed the beta label from Cloud Dataflow, its managed cloud service and programming model for batch and streaming big data processing, and is making it generally available.

"Cloud Dataflow is specifically designed to remove the complexity of developing separate systems for batch and streaming data sources by providing a unified programming model," Google says in a blog post. "Based on more than a decade of Google innovation, including MapReduce, FlumeJava, and Millwheel, Cloud Dataflow is built to free you from the operational overhead related to large scale cluster management and optimization."

With Cloud Dataflow, Google promises a "fully managed, fault tolerant, highly available, SLA-backed service" for batch and stream processing as well as a "comprehensive model for balancing correctness, latency, and cost" for dealing with unordered data at "massive" scale. It also promises "great" performance.

"Cloud Dataflow is 2-3x faster and cheaper than Hadoop when evaluating classic MapReduce based pipelines, such as PageRank and WordCount," the company says. "And with dynamic work rebalancing, Cloud Dataflow effectively optimizes resource utilization which provides additional performance gains without requiring manual intervention."

They've also expanded partner efforts with Tamr, Salesforce, Clearstory, springML, Cloudera, data Artisans.

"Running within the Google Cloud Dataflow service Tamr will enable enterprises to unify, clean, transform and move their data from many diverse sources without investing in expensive data integration expertise or infrastructure," a spokesperson for Tamr tells WebProNews in an email.

Cloud Dataflow also includes native Cloud Platform integration for Cloud Storage, Cloud Datastore, BigQuery, and Cloud Pub/Sub.

Google also announced general availability of Cloud Pub/Sub. More on this here.

Image via Google

Chris Crum
Chris Crum has been a part of the WebProNews team and the iEntry Network of B2B Publications since 2003. Follow Chris on Twitter, on StumbleUpon, on Pinterest and/or on Google: +Chris Crum.