Role of Weighting on TDM in Improvising Performance of LSA on Text Data

In this paper, we show that the efficiency of LSA is significantly controlled by the choice of weighting algorithm applied. These weighting algorithms allocate relative importance to the document attributes (e.g. keywords) based on their occurrences in the corpus. Effects of different weighting algorithms are the central point of this paper. We experimented with various weighting algorithms to evaluate and study their effects as measured by precision and recall values. Our experiments include weighting function application on TDM (pre-weighting) in order to increase or decrease the relative importance of words based on their occurrence. We also evaluated the application of weighting functions on the projected query (post-weighting). Post-weighted keyword queries were projected on an LSA model built on pre-weighted TDM to obtain closely correlated keywords or a document (keyword collection). We have developed a prototype IR query projection tool which projects keyword queries on the LSA model to retrieve relevant keywords with a floating-point score