Title :
Hadoop Compatible Framework for Discovering Network Topology and Detecting Hardware Failures
Author :
Ventrapragada, A. ; Samuel, Selvakumar ; Vidya, V.R. ; Prabha Satya Manepalli, V. ; Muralidharan, Sriram ; Rao, Smitha
Author_Institution :
Int. Inst. of Inf. Technol., Bangalore, India
Abstract :
We have implemented a Hadoop compatible framework that helps in detection of suspected hardware failures in DataNodes within a Hadoop cluster and in signalling the nodes accordingly. Based on the status of the various hardware components of the DataNodes, the master node signals the DataNode so that appropriate actions can be taken. In a Hadoop cluster, the knowledge of the network is important for the master node to configure itself to meet varying needs of the DataNodes. It is important to keep track of the current topology of the Hadoop network and track the status of the network services in Hadoop. We have also added the functionality to discover the network topology in a Hadoop cluster. This discovered topology information will be useful in performing load balancing in the network, or in making intelligent decisions in data replication in case of node failures.
Keywords :
distributed processing; network topology; pattern clustering; public domain software; resource allocation; DataNodes; Hadoop compatible framework; data replication; distributed software systems; hardware components; hardware failure detection; intelligent decisions; load balancing; master node signals; network services; network topology; node failures; open source framework; Hard disks; Hardware; IP networks; Monitoring; Network topology; Operating systems; Topology; DataNode; Hadoop; NameNode; Network Topology;
Conference_Titel :
Services in Emerging Markets (ICSEM), 2012 Third International Conference on
Conference_Location :
Mysore
Print_ISBN :
978-1-4673-5729-6
DOI :
10.1109/ICSEM.2012.16
