A highly efficient distributed indexing system based on large cluster of commodity machines

An Information Retrieval System using centralized approach demands long time to update the web index. A highly efficient distributed indexing system operates on large & diverse datasets with optimum time consumption compared to centralized approach to update web index. In this paper, a prototype model of highly efficient distributed indexing system deployed to run on cluster of commodity machines for the creation of large index using functionality of Apache Lucene. Experimental results showed efficiency of distributed indexing process. This distributed approach helps to reduce time interval for index creation and updation, in turn keeps the index content more fresh.