Bandwidth-aware data placement scheme for Hadoop

We are living in a data rich era. The size of the data is increasing exponentially. Social networking applications, Scientific experiments, etc. are the major contributors of Big Data. The data can be structured, semi-structured or unstructured. Big Data management solutions can be implemented in-house in the organization or it can be stored in cloud. Whether it is stored in-house or in cloud, the placement of data is very important. In general, users demand the availability of data whenever they request for it. There are many parameters that effect the data retrieval time in Hadoop. Among them, this paper pays attention to the available bandwidth. To minimize the data retrieval time, the data must be placed in a DataNode which has the maximum bandwidth. We have proposed a solution for bandwidth-aware data placement in Hadoop by periodically measuring the bandwidth between clients and DataNodes and placing the data blocks in DataNodes that have maximum end-to-end bandwidth.