Parallel indexing in a Chinese information retrieval system

The increasing data size in Chinese information-based applications renders conventional information retrieval (IR) systems unsuitable. This is because they are limited in both storage and speed. To overcome these predicaments, a parallel Chinese IR system (CIR) has been designed. It is being developed on a SIMD parallel computer, DECmpp, which is configured with 8,192 processing elements. It uses full inverted indices for retrieval. The “divide-and-conquer” principle is exercised in exploiting data parallelism in the inverted index files. The inverted indices are first partitioned into fragments. Each fragment is then assigned to an individual processing elements. Thereafter, during an index retrieval operation, all index fragments are searched in parallel. Although the principle is simple, realising the parallel indexing algorithm in a naive fashion (i.e. without considering the underlying parallel architecture) would result in poor retrieval performance. During the design of the CIR system, 3 different implementation models for parallel indexing have been considered. In this paper, qualitative evaluation of the 3 models is presented. Based on the result of the evaluation, the model that offers the best run-time performance was adopted