research-article

Dimension reduction for distance-based indexing

Authors:

Willard L. Miranker,

Daniel P. MirankerAuthors Info & Claims

SISAP '10: Proceedings of the Third International Conference on SImilarity Search and APplications

Pages 25 - 32

https://doi.org/10.1145/1862344.1862349

Published: 18 September 2010 Publication History

Abstract

Distance-based indexing exploits only the triangle inequality to answer similarity queries in metric spaces. Lacking of coordinate structure, mathematical tools in Rⁿ can only be applied indirectly, making it difficult for theoretical study in metric space indexing. Toward solving this problem, we formalize a "pivot space model" where data is mapped from metric space to Rⁿ, preserving all the pair wise distances under L^infin;. With this model, it can be shown that the indexing problem in metric space can be equivalently studied in Rⁿ. Further, we show the necessity of dimension reduction for Rⁿ and that the only effective form of dimension reduction is to select existing dimensions, i.e. pivot selection. The coordinate structure of Rⁿ makes the application of many mathematical tools possible. In particular, Principle Component Analysis (PCA) is incorporated into a heuristic method for pivot selection and shown to be effective over a large range of workloads. We also show that PCA can be used to reliably measure the intrinsic dimension of a metric-space.

References

[1]

}}Bentley, J. L., Multidimensional binary search trees used for associative searching. Commun.ACM, 1975. 18(9): p. 509--517.

Digital Library

[2]

}}Beyer, K. S., J. Goldstein, R. Ramakrishnan, and U. Shaft. When Is "Nearest Neighbor" Meaningful? the 7th International Conference on Database Theory. 1999: Springer-Verlag.

Digital Library

[3]

}}Bozkaya, T. and M. Ozsoyoglu, Indexing large metric spaces for similarity search queries. ACM Trans. Database Syst., 1999. 24(3): p. 361--404.

Digital Library

[4]

}}Brin, S. Near Neighbor Search in Large Metric Spaces. in the 21th International Conference on Very Large Data Bases (VLDB'95). 1995: Morgan Kaufmann Publishers Inc.

Digital Library

[5]

}}Bustos, B., G. Navarro, and E. Chavez, Pivot selection techniques for proximity searching in metric spaces. Pattern Recogn. Lett., 2003. 24(14): p. 2357--2366.

Digital Library

[6]

}}Camastra, F., Data dimensionality estimation methods: a survey. Pattern Recognition, 2003. 36(12): p. 2945--2954.

[7]

}}Chavez, E., G. Navarro, R. Baeza-Yates, and J. Marroqu, Searching in metric spaces. ACM Computing Surveys, 2001. 33(3): p. 273--321.

Digital Library

[8]

}}Ciaccia, P. and M. Patella. Bulk loading the M-tree. in 9th Australasian Database Conference (ADO'98). 1998.

[9]

}}Clarkson K. L., Nearest-neighbor searching and metric space dimensions, In: Nearest-Neighbor Methods for Learning and Vision: Theory and Practice, MIT Press, 2006, pp. 15--59

[10]

}}Guttman, A., R-trees: a dynamic index structure for spatial searching, in Proceedings of the 1984 ACM SIGMOD international conference on Management of data. 1984.

Digital Library

[11]

}}Hjaltason, G. R. and H. Samet, Index-driven similarity search in metric spaces. ACM Transactions on Database Systems (TODS), 2003. 28(4): p. 517--580.

Digital Library

[12]

}}Hochbaum, D. S. and D. B. Shmoys, A best possible heuristic for the k-center problem. Mathematics of Operational Research, 1985. 10(2): p. 180--184.

[13]

}}Kegl, B., Intrinsic dimension estimation using packing numbers. Advances in Neural Information Processing Systems, 2003. 15: p. 681--688.

[14]

}}Mao, R., W. Xu, S. Ramakrishnan, G. Nuckolls, and D. P. Miranker. On Optimizing Distance-Based Similarity Search for Biological Databases. in the 2005 IEEE Computational Systems Bioinformatics Conference (CSB 2005). 2005.

Digital Library

[15]

}}Mao, R., W. Xu, N. Singh, and D. P. Miranker, An Assessment of a Metric Space Database Index to Support Sequence Homology. International Journal on Artificial Intelligence Tools (IJAIT), 2005: p. 867--885.

[16]

}}Matousek, J., Lectures on Discrete Geometry. 2002: Springer-Verlag New York, Inc. 497.

Digital Library

[17]

}}Test suite. http://aug.csres.utexas.edu/mobios-workload/.

[18]

}}Navarro, G. Searching in Metric Spaces by Spatial Approximation. in Proceedings of the String Processing and Information Retrieval Symposium & International Workshop on Groupware. 1999: IEEE Computer Society.

Digital Library

[19]

}}Roweis, S., EM Algorithms for PCA and SPCA. Neural Information Processing Systems 10, 1997: p. 626--632.

Digital Library

[20]

}}Samet, H., Foundations of Multidimensional and Metric Data Structures. 2006, Morgan-Kaufmann.

Digital Library

[21]

}}Shaft, U. and R. Ramakrishnan. When Is Nearest Neighbors Indexable? in Tenth International Conference on Database Theory (ICDT 2005). 2005: Springer

Digital Library

[22]

}}Uhlmann, J. K., Satisfying General Proximity/Similarity Queries with Metric Trees. Information Processing Letter, 1991. 40(4): p. 175--179.

[23]

}}Weber, R., H. J. Schek, and S. Blott. A quantitative analysis and performance study for similarity-search methods in high-dimensional spaces. in International Conference on Very Large Data Bases. 1998.

Digital Library

[24]

}}Xu, W. and D. P. Miranker, A Metric Model of Amino Acid Substitution. Bioinformatics, 2004. 20(8): p. 1214--1221.

Digital Library

[25]

}}Yianilos, P. N. Data structures and algorithms for nearest neighbor search in general metric spaces. in the fourth annual ACM-SIAM Symposium on Discrete algorithms. 1993: Society for Industrial and Applied Mathematics.

Digital Library

Cited By

Marten DMeyer HHeuer A(2019)Calculating Fourier Transforms in SQLAdvances in Databases and Information Systems10.1007/978-3-030-28730-6_10(151-166)Online publication date: 13-Aug-2019
https://doi.org/10.1007/978-3-030-28730-6_10
Amato GEsuli AFalchi F(2018)A comparison of pivot selection techniques for permutation-based indexingInformation Systems10.1016/j.is.2015.01.01052:C(176-188)Online publication date: 30-Dec-2018
https://dl.acm.org/doi/10.1016/j.is.2015.01.010
Connor RVadicamo LRabitti F(2017)High-Dimensional Simplexes for Supermetric SearchSimilarity Search and Applications10.1007/978-3-319-68474-1_7(96-109)Online publication date: 28-Sep-2017
https://doi.org/10.1007/978-3-319-68474-1_7
Show More Cited By

Index Terms

Dimension reduction for distance-based indexing
1. Information systems

Recommendations

Pivot selection: Dimension reduction for distance-based indexing

Distance-based indexing exploits only the triangle inequality to answer similarity queries in metric spaces. Lacking coordinate structure, mathematical tools in R^n can only be applied indirectly, making it difficult to theoretically study metric-space ...
An eigenvalue-based pivot selection strategy for efficient indexing and searching in metric spaces

Pivots are used widely during indexing and searching in metric spaces. We maintain the distances from pivots to data objects to be indexed so the pre-computed distances can be used to prune unpromising objects during the search process. The search ...
Multifidelity Dimension Reduction via Active Subspaces

We propose a multifidelity dimension reduction method to identify a low-dimensional structure present in many engineering models. The structure of interest arises when functions vary primarily on a low-dimensional subspace of the high-dimensional input ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SISAP '10: Proceedings of the Third International Conference on SImilarity Search and APplications

September 2010

130 pages

ISBN:9781450304207

DOI:10.1145/1862344

Program Chairs:
Paolo Ciaccia
Università di Bologna
,
Marco Patella
Università di Bologna

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Bilkent University: Bilkent University
Mexican Computer Science Society

In-Cooperation

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Biological Infrastructure

Conference

SISAP '10

Sponsor:

Bilkent University

SISAP '10: 3rd International Workshop on Similarity Search and Applications

September 18 - 19, 2010

Istanbul, Turkey

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
259
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Marten DMeyer HHeuer A(2019)Calculating Fourier Transforms in SQLAdvances in Databases and Information Systems10.1007/978-3-030-28730-6_10(151-166)Online publication date: 13-Aug-2019
https://doi.org/10.1007/978-3-030-28730-6_10
Amato GEsuli AFalchi F(2018)A comparison of pivot selection techniques for permutation-based indexingInformation Systems10.1016/j.is.2015.01.01052:C(176-188)Online publication date: 30-Dec-2018
https://dl.acm.org/doi/10.1016/j.is.2015.01.010
Connor RVadicamo LRabitti F(2017)High-Dimensional Simplexes for Supermetric SearchSimilarity Search and Applications10.1007/978-3-319-68474-1_7(96-109)Online publication date: 28-Sep-2017
https://doi.org/10.1007/978-3-319-68474-1_7
Lei FWu WLi QZhang HLi PLuo QMao R(2014)Speed Up Distance-Based Similarity Query Using Multiple Threads2014 Sixth International Symposium on Parallel Architectures, Algorithms and Programming10.1109/PAAP.2014.54(215-219)Online publication date: Jul-2014
https://doi.org/10.1109/PAAP.2014.54
Li QZhang HLei FLiu GLu MMao R(2014)Excluded Middle Forest Versus Vantage Point Tree: An Analytical and Empirical ComparisonPractical Applications of Intelligent Systems10.1007/978-3-642-54927-4_41(431-437)Online publication date: 19-Jul-2014
https://doi.org/10.1007/978-3-642-54927-4_41
王海(2013)Building an Information Retrieval System: Global Indexing or Local Indexing?Software Engineering and Applications10.12677/SEA.2013.2100202:01(6-14)Online publication date: 2013
https://doi.org/10.12677/SEA.2013.21002
Amato GEsuli AFalchi F(2013)Pivot Selection Strategies for Permutation-Based Similarity SearchProceedings of the 6th International Conference on Similarity Search and Applications - Volume 819910.1007/978-3-642-41062-8_10(91-102)Online publication date: 2-Oct-2013
https://dl.acm.org/doi/10.1007/978-3-642-41062-8_10
Rui Mao Xi Liu Houjun Tang Qiuming Luo Jiaqi Chen Wenbo Wu (2011)Multivariate regression for pivot selection: A preliminary study2011 3rd Symposium on Web Society10.1109/SWS.2011.6101281(115-121)Online publication date: Oct-2011
https://doi.org/10.1109/SWS.2011.6101281

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents