Approximate Incremental Big-Data Harmonization

The needs of `big data analytics´ increasingly require IT organizations to ingest, process, and extract business insights from ever larger volumes of data that arrive far more rapidly than before, as well as from new sources such as social media, mobile devices, and sensors. However, in order to extract insights from diverse information feeds from multiple, often unrelated sources, these first need to be correlated or harmonized to a common level of granularity. We formally define this commonly arising data harmonization problem. We show how to correlate disparate data sources using map-reduce, but in an approximate and/or incremental manner as often required in practice. We motivate our techniques through a real-life enterprise data-harmonization case study for which we describe our performance results on big-data technologies, namely, Map Reduce, Hadoop and PIG.