Cloud Computing Test Bed

Yahoo! has joined forces with HP and Intel to create a global research project that brings cloud computing to the forefront of the technology universe.

The new initiative – the Cloud Computing Test Bed – enables universities to conduct practical research in large-scale, open-source environments and re-create cloud computing scenarios that mirror the real world.

The test bed is also designed to foster collaboration among vendors, universities and government agencies by removing the financial and logistical barrier typically associated with research in data-intensive, Internet-scale computing.

Essentially, the project will tap the brightest minds in industry, academia and government to drive innovation in cloud computing. Not only can researchers test applications at Internet scale, they will also have access to the underlying computing systems. Ultimately, this will help the worldwide research community get to the next level in its understanding of large-scale computing systems -- and how they function in a cloud environment.

“We know that universities are interested in large-scale distributed computing, but very few have the wherewithal to create their own clusters and conduct research at Internet scale,” said Ron Brachman, Vice President of Worldwide Research Operations at Yahoo!. “With this test bed, we can provide the academic community with a great new opportunity to operate at Web scale.”

Prabhakar Raghavan, head of Yahoo! Research, adds that the entire planet may eventually come to rely on cloud computing, like electricity, to power daily communications. “It’s about what kinds of interesting projects you can do once you take the cloud for granted, as if it were a utility,” he says.

The project has been featured in a number of media outlets including The New York Times, Reuters, CNET and GigaOm. “The kind of investment announced by Yahoo, HP and Intel enlarges the opportunity for students and researchers to experiment with cloud computing and thus add to society’s innovation in this emerging field of technology,” trumpeted the New York Times.

Other partners in the initiative include the Infocomm Development Authority of Singapore, the University of Illinois at Urbana-Champaign, and the Karlsruhe Institute of Technology in Germany. The partnership with Illinois also includes the National Science Foundation.

The Cloud Computing Test Bed will initially comprise data centers at six different locations across the globe, including the U.S., Singapore, and Germany. Each location will host a cloud computing center that is largely based on HP hardware and Intel computer processors. Between 1,000 and 4,000 processor cores, or up to about 500 servers, will be housed at each data center.

Yahoo!, for its part, will provide resources related to open-source computing software and distribution over the Internet. The test bed will leverage Yahoo!’s technical leadership in open source projects by running Apache Hadoop -- an open source, distributed computing project of the Apache Software Foundation. It will also incorporate other open source, distributed computing software such as Pig, the parallel programming language developed by Yahoo! Research.

The effort builds on Yahoo!’s M45 supercomputing data center for cloud computing research, which is already in use by Carnegie Mellon University. M45 is a 4,000-processor supercomputer that’s one of the most powerful systems in the world. The project is enabling academic researchers to tackle some of the most complicated computing tasks known to humanity.

Yahoo!’s commitment to cloud computing is second to none. Yahoo! formed a Cloud Computing and Data Infrastructure Group as part of a major reorganization in June. And earlier this year Yahoo! partnered with Computational Research Laboratories (CRL) to jointly support cloud computing research and make one of the ten fastest supercomputers in the world available to academic institutions in India.

“Yahoo! as a whole is embracing cloud computing,” said Brachman. “We are one of only a small number of companies in the industry capable of putting together these large clusters. Cloud computing will continue to be a big priority for us going forward.”