OpenFlow Enabled Hadoop over Local and Wide Area Clusters

Author

Narayan, S. ; Bailey, Susan ; Greenway, M. ; Grossman, Robert ; Heath, A. ; Powell, R. ; Daga, Anand

Author_Institution

InfoBlox Inc., Santa Clara, CA, USA

fYear

2012

fDate

10-16 Nov. 2012

Firstpage

1625

Lastpage

1628

Abstract

Hadoop has emerged as an important platform for data intensive computing. The shuffle and sort phases of a MapReduce computation often saturate top of the rack switches, as well as switches that aggregate multiple racks. In addition, MapReduce computations often have "hot spots" in which the computation is lengthened due to inadequate bandwidth to some of the nodes. In principle, OpenFlow enables an application to adjust the network topology as required by the computation, providing additional network bandwidth to those resources requiring it. We describe Hadoop-OFE, which is an OpenFlow enabled version of Hadoop that dynamically modifies the network topology in order to improve the performance of Hadoop.

Keywords

Java; distributed processing; telecommunication network topology; MapReduce computation; OpenFlow enabled Hadoop; data intensive computing; hot spots; local area clusters; network bandwidth; network topology; wide area clusters; Hadoop; MapReduce; OpenFlow; OpenFlow over Ethernet; data intensive computing;

fLanguage

English

Publisher

ieee

Conference_Titel

High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion:

Conference_Location

Salt Lake City, UT

Print_ISBN

978-1-4673-6218-4

Type

conf

DOI

10.1109/SC.Companion.2012.340

Filename

6496126