DocumentCode :
1239924
Title :
General Cost Models for Evaluating Dimensionality Reduction in High-Dimensional Spaces
Author :
Lian, Xiang ; Chen, Lei
Author_Institution :
Dept. of Comput. Sci. & Eng., Hong Kong Univ. of Sci. & Technol., Kowloon, China
Volume :
21
Issue :
10
fYear :
2009
Firstpage :
1447
Lastpage :
1460
Abstract :
Similarity search usually encounters a serious problem in the high-dimensional space, known as the "curse of dimensionality". In order to speed up the retrieval efficiency, most previous approaches reduce the dimensionality of the entire data set to a fixed lower value before building indexes (referred to as global dimensionality reduction (GDR)). More recent works focus on locally reducing the dimensionality of data to different values (called the local dimensionality reduction (LDR)). In addition, random projection is proposed as an approximate dimensionality reduction (ADR) technique to answer the approximate similarity search instead of the exact one. However, so far little work has formally evaluated the effectiveness and efficiency of GDR, LDR, and ADR for the range query. Motivated by this, in this paper, we propose general cost models for evaluating the query performance over the reduced data sets by GDR, LDR, and ADR, in light of which we introduce a novel (A)LDR method, partitioning based on randomized search (PRANS). It can achieve high retrieval efficiency with the guarantee of optimality given by the formal models. Finally, a B+-tree index is constructed over the reduced partitions for fast similarity search. Extensive experiments validate the correctness of our cost models on both real and synthetic data sets and demonstrate the efficiency and effectiveness of the proposed PRANS method.
Keywords :
query processing; random processes; tree data structures; B+-tree index; approximate dimensionality reduction; general cost model; global dimensionality reduction; high-dimensional space; random projection; similarity search; High-dimensionality reduction; similarity search.;
fLanguage :
English
Journal_Title :
Knowledge and Data Engineering, IEEE Transactions on
Publisher :
ieee
ISSN :
1041-4347
Type :
jour
DOI :
10.1109/TKDE.2008.170
Filename :
4814924
Link To Document :
بازگشت