Hadoop Compatible Framework for Discovering Network Topology and Detecting Hardware Failures

Author

Ventrapragada, A. ; Samuel, Selvakumar ; Vidya, V.R. ; Prabha Satya Manepalli, V. ; Muralidharan, Sriram ; Rao, Smitha

Author_Institution

Int. Inst. of Inf. Technol., Bangalore, India

fYear

2012

fDate

12-15 Dec. 2012

Firstpage

58

Lastpage

64

Abstract

We have implemented a Hadoop compatible framework that helps in detection of suspected hardware failures in DataNodes within a Hadoop cluster and in signalling the nodes accordingly. Based on the status of the various hardware components of the DataNodes, the master node signals the DataNode so that appropriate actions can be taken. In a Hadoop cluster, the knowledge of the network is important for the master node to configure itself to meet varying needs of the DataNodes. It is important to keep track of the current topology of the Hadoop network and track the status of the network services in Hadoop. We have also added the functionality to discover the network topology in a Hadoop cluster. This discovered topology information will be useful in performing load balancing in the network, or in making intelligent decisions in data replication in case of node failures.

Keywords

distributed processing; network topology; pattern clustering; public domain software; resource allocation; DataNodes; Hadoop compatible framework; data replication; distributed software systems; hardware components; hardware failure detection; intelligent decisions; load balancing; master node signals; network services; network topology; node failures; open source framework; Hard disks; Hardware; IP networks; Monitoring; Network topology; Operating systems; Topology; DataNode; Hadoop; NameNode; Network Topology;

fLanguage

English

Publisher

ieee

Conference_Titel

Services in Emerging Markets (ICSEM), 2012 Third International Conference on

Conference_Location

Mysore

Print_ISBN

978-1-4673-5729-6

Type

conf

DOI

10.1109/ICSEM.2012.16

Filename

6468180