Distributed media indexing based on MPI and MapReduce

Web-scale digital assets comprise millions or billions of documents. Due to such increase, sequential algorithms cannot cope with this data, and parallel and distributed computing become the solution of choice. MapReduce is a programming model proposed by Google for scalable data processing. MapReduce is mainly applicable for data intensive algorithms. In contrast, The message passing interface (MPI) is suitable for high performance algorithms. This paper proposes an adapted structure of MapReduce programming model using MPI for multimedia indexing. Experimental results on a large number of text (XML) excerpts related to images from the ImageNet corpus indicate that our implementation achieved good speedup compared to the sequential version and the earlier versions of MapReduce using MPI. Extensions to index large-scale multimedia collections are discussed.