Experiments on Networking of Hadoop

Hadoop is a popular application to analyze and process big data problems in a networked distributed system of computers. Investigations of performance for application-aware networking have been of interest with the software-defined networking paradigm through on-demand and dynamic policy enforcements. Network usage characterization of Hadoop jobs can further help understand what policy enforcements may be needed during application use cases. At scale experimentation of Hadoop jobs will help facilitate such characterizations. We report how Hadoop networking usage can be characterized in an experimentation environment using the GENI (Global Environment for Network Innovation). Furthermore, we report a distributed switch framework that may help alleviate the fault tolerance schemes in Hadoop application in the forwarding plane. Delay in recovery from failures has been reduced by almost 99% through such a distributed switch architecture deployed on the GENI experimentation environment.