Enhancing Accuracy and Performance of Collaborative Filtering Algorithm by Stochastic SVD and Its MapReduce Implementation

Collaborative filtering algorithms that extract desired information from records have been widely used in data mining and information retrieval, such as recommendation systems. However, the rapidly increased data size demands more efficient and scalable algorithms and implementations. In this paper, we present a novel algorithm that utilizes stochastic singular value decomposition (SSVD) in the calculation of item-based collaborative filtering. The use of SSVD does not only provide more accurate results in terms of precision and recall, but also reduces the computational cost. The proposed algorithm was implemented using Hadoop MapReduce, which allows distributed processing of massive data stored in a distributed file system. The implementation was evaluated and compared with the recommendation systems provided in the Apache Mahout project, and a 2.53 speedup can be obtained for processing millions records. The accuracy of our algorithm is also 3 times better than the non-SVD algorithm in terms of the F1 metric, a combinative measurement of precision and recall.