research-article

Toward Improved Ranking Metrics

Authors:

Michael S. Lew,

Dionysius P. HuijsmansAuthors Info & Claims

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22, Issue 10

Pages 1132 - 1143

https://doi.org/10.1109/34.879793

Published: 01 October 2000 Publication History

Abstract

In many computer vision algorithms, a metric or similarity measure is used to determine the distance between two features. The Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise distribution is Gaussian. Based on real noise distributions measured from international test sets, we have found that the Gaussian noise distribution assumption is often invalid. This implies that other metrics, which have distributions closer to the real noise distribution, should be used. In this paper, we consider three different applications: content-based retrieval in image databases, stereo matching, and motion tracking. In each of them, we experiment with different modeling functions for the noise distribution and compute the accuracy of the methods using the corresponding distance measures. In our experiments, we compared the SSD metric, the SAD (sum of the absolute differences) metric, the Cauchy metric, and the Kullback relative information. For several algorithms from the research literature which used the SSD or SAD, we showed that greater accuracy could be obtained by using the Cauchy metric instead.

References

[1]

H. Akaike, “Information Theory and an Extension of the Maximum Likelihood Principle,” Proc. Second Int'l Symp. Information Theory, pp. 267–281, 1973.

[2]

S. Barnard and M. Fischler, “Computational Stereo,” ACM Computing Surveys, vol. 14, no. 4, pp. 553–572, 1982.

Digital Library

[3]

D.N. Bhat and S.K. Nayar, “Ordinal Measures for Image Correspondence,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 4, pp. 415–423, Apr. 1998.

Digital Library

[4]

M.J. Black, “Robust Incremental Optical Flow,” PhD thesis, Yale Univ., Sept. 1992.

Digital Library

[5]

R. Boie and I. Cox, “An Analysis of Camera Noise,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 6, pp. 671–674, June 1992.

Digital Library

[6]

I. Cox S. Hingorani and S. Rao, “A Maximum Likelihood Stereo Algorithm,” Computer Vision and Image Understanding, vol. 63, no. 3, pp. 542–567, 1996.

Digital Library

[7]

M. Flicker H. Sawhney W. Niblack J. Ashley Q. Huang B. Dom M. Gorkani J. Hafner D. Lee D. Petkovic D. Steele and P. Yanker, “Query By Image and Video Content: The QBIC System,” Computer, vol. 28, no. 9, pp. 23–32, Sept. 1995.

Digital Library

[8]

A. Fusiello V. Roberto and E. Trucco, “Efficient Stereo with Multiple Windowing,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 858–863, 1997.

Digital Library

[9]

T. Gevers and A. Smeulders, “Color-Based Object Recognition,” Pattern Recognition, vol. 32, no. 3, pp. 453–464, 1999.

[10]

W. Grimson, “Computational Experiments with a Feature Based Stereo Algorithm,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 7, no. 1, pp. 17–34, Jan. 1985.

Digital Library

[11]

J. Hafner H.S. Sawhney W. Equitz M. Flickner and W. Niblack, “Efficient Color Histogram Indexing for Quadratic Form Distance Functions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 7, pp. 729–736, July 1995.

Digital Library

[12]

F.R. Hampel E.M. Ronchetti P.J. Rousseeuw and W.A. Stahel, Robust Statistic: The Approach Based on Influence Functions. New York: John Wiley & Sons, 1986.

[13]

R. Haralick and L. Shapiro, Computer and Robot Vision II. Addison-Wesley, 1993.

Digital Library

[14]

P.J. Huber, Robust Statistic. New York: John Wiley & Sons, 1981.

[15]

D.P. Huijsmans and M.S. Lew, “Efficient Content-Based Image Retrieval in Digital Picture Collections Using Projections: (Near)Copy Locations,” Proc. 13th Int'l Conf. Pattern Recognition, vol. 3, pp. 104–108, 1996.

Digital Library

[16]

D.P. Huijsmans M.S. Lew and D. Denteneer, “Quality Measures for Interactive Image Retrieval with a Performance Evaluation of Two 3×3 Texel-Based Methods,” Lecture Notes in Computer Science, vol. 1, 311, no. 2, pp. 22–29, 1997.

Digital Library

[17]

P.M. Kelly and T.M. Cannon, “CANDID: Comparison Algorithm for Navigating Digital Image Databases,” Proc. 17th Int'l Working Conf. Scientific and Statistical Database Management, pp. 252–258, 1994.

Digital Library

[18]

P.M. Kelly T.M. Cannon and J.E. Barros, “Efficiency Issues Related to Probability Density Function Comparison,” Proc. SPIE—Storage and Retrieval for Image and Video Databases, vol. 2, 670, no. 4, pp. 42–49, 1996.

[19]

P.M. Kelly T.M. Cannon and D.R. Hush, “Query By Image Example: The CANDID Approach,” Proc. SPIE—Storage and Retrieval for Image and Video Databases, vol. 2, 420, no. 3, pp. 238–248, 1995.

[20]

S. Kullback, Information Theory and Statistics. Dover Publications, 1968.

[21]

M.S. Lew T.S. Huang and K. Wong, “Learning and Feature Selection in Stereo Matching,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 9, pp. 869–882, Sept. 1994.

Digital Library

[22]

W. Luo and H. Maitre, “Using Surface Model to Correct and Fit Disparity Data in Stereo Vision,” Proc. Int'l Conf. Pattern Recognition, vol. 1, pp. 60–64, 1990.

[23]

D. Marr and T. Poggio, “A Computational Theory of Human Stereo Vision,” Proc. Royal Soc. London, vol. 204, pp. 301–328, 1976.

[24]

J. Rissanen, “Modeling by Shortest Data Description,” Automatica, vol. 14, pp. 465–471, 1978.

Digital Library

[25]

P.J. Rousseeuw and A.M. Leroy, Robust Regression and Outlier Detection. New York: John Wiley & Sons, 1987.

Digital Library

[26]

H.S. Sawhney and J.L. Hafner, “Efficient Color Histogram Indexing,” Proc. 1994 IEEE Int'l Conf. Image Processing, vol. 2, pp. 66–70, 1994.

[27]

N. Sebe M.S. Lew and D.P. Huijsmans, “Which Ranking Metric Is Optimal? with Applications in Image Retrieval and Stereo Matching,” Proc. Int'l Conf. Pattern Recognition, pp. 265–271, 1998.

Digital Library

[28]

H.S. Stone and C.S. Li, “Image Matching by Means of Intensity and Texture Matching in the Fourier Domain,” Proc. SPIE—Electronic Imaging: Science and Technology, Jan. 1996.

[29]

M.J. Swain and D.H. Ballard, “Color Indexing,” Int'l J. Computer Vision, vol. 7, no. 1, pp. 11–32, 1991.

Digital Library

[30]

L. Tang Y. Kong L.S. Chen C. R. Lansing and T.S. Huang, “Performance Evaluation of a Facial Feature Tracking Algorithm,” Proc. NSF/ARPA Workshop: Performance vs. Methodology in Computer Vision, pp. 218–229, 1994.

Cited By

Li PXie HJiang YGe JZhang Y(2023)Neighborhood-Adaptive Multi-Cluster Ranking for Deep Metric LearningIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2022.321368033:4(1952-1965)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TCSVT.2022.3213680
Golshan HHasanzadeh R(2015)An optimized LMMSE based method for 3D MRI denoisingIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2014.234467512:4(861-870)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1109/TCBB.2014.2344675
Mao ZZhang YTian QJaimes ASebe NBoujemaa NGatica-Perez DShamma DWorring MZimmermann R(2013)What are the distance metrics for local features?Proceedings of the 21st ACM international conference on Multimedia10.1145/2502081.2502134(505-508)Online publication date: 21-Oct-2013
https://dl.acm.org/doi/10.1145/2502081.2502134
Show More Cited By

Index Terms

Toward Improved Ranking Metrics
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
    2. Machine learning algorithms
      1. Feature selection

Recommendations

Doubly Robust Estimator for Ranking Metrics with Post-Click Conversions
RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

Post-click conversion, a pre-defined action on a web service after a click, is an essential form of feedback, as it directly contributes to the final revenue and accurately captures user preferences for items, compared with the ambiguous click. However, ...
Assessing ranking metrics in top-N recommendation
Abstract
The evaluation of recommender systems is an area with unsolved questions at several levels. Choosing the appropriate evaluation metric is one of such important issues. Ranking accuracy is generally identified as a prerequisite for recommendation ...
Local geometry adaptive manifold re-ranking for shape-based 3D object retrieval
MM '12: Proceedings of the 20th ACM international conference on Multimedia

This paper proposes an improvement to Manifold Ranking algorithm used for search results ranking in the context of shape-based 3D model retrieval. Manifold Ranking algorithm by Zhou et al estimates, given a set of high-dimensional feature vectors, a ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 22, Issue 10

October 2000

145 pages

ISSN:0162-8828

Editor:
Kevin Bowyer
Univ. of South Florida, Tampa

Issue’s Table of Contents

Copyright © Copyright © 2000 IEEE. All Rights Reserved.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 October 2000

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li PXie HJiang YGe JZhang Y(2023)Neighborhood-Adaptive Multi-Cluster Ranking for Deep Metric LearningIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2022.321368033:4(1952-1965)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TCSVT.2022.3213680
Golshan HHasanzadeh R(2015)An optimized LMMSE based method for 3D MRI denoisingIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2014.234467512:4(861-870)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1109/TCBB.2014.2344675
Mao ZZhang YTian QJaimes ASebe NBoujemaa NGatica-Perez DShamma DWorring MZimmermann R(2013)What are the distance metrics for local features?Proceedings of the 21st ACM international conference on Multimedia10.1145/2502081.2502134(505-508)Online publication date: 21-Oct-2013
https://dl.acm.org/doi/10.1145/2502081.2502134
Lee CLin SLee CYang C(2012)An efficient continuous tracking system in real-time surveillance applicationJournal of Network and Computer Applications10.1016/j.jnca.2011.12.00735:3(1067-1073)Online publication date: 1-May-2012
https://dl.acm.org/doi/10.1016/j.jnca.2011.12.007
Si STao DWang MChan K(2012)Social image annotation via cross-domain subspace learningMultimedia Tools and Applications10.1007/s11042-010-0567-256:1(91-108)Online publication date: 1-Jan-2012
https://dl.acm.org/doi/10.1007/s11042-010-0567-2
Essannouni FAboutajdine D(2010)Fast frequency template matching using higher order statisticsIEEE Transactions on Image Processing10.5555/1771901.177192419:3(826-830)Online publication date: 1-Mar-2010
https://dl.acm.org/doi/10.5555/1771901.1771924
Wang MHua XHong RTang JQi GSong Y(2009)Unified video annotation via multigraph learningIEEE Transactions on Circuits and Systems for Video Technology10.5555/1641661.164167119:5(733-746)Online publication date: 1-May-2009
https://dl.acm.org/doi/10.5555/1641661.1641671
Huang ZShen HShao JZhou XCui B(2009)Bounded coordinate system indexing for real-time video clip searchACM Transactions on Information Systems10.1145/1508850.150885527:3(1-33)Online publication date: 19-May-2009
https://dl.acm.org/doi/10.1145/1508850.1508855
Wang MHua XTang JHong R(2009)Beyond distance measurementIEEE Transactions on Multimedia10.1109/TMM.2009.201291911:3(465-476)Online publication date: 1-Apr-2009
https://dl.acm.org/doi/10.1109/TMM.2009.2012919
Magalhães JCiravegna FRüger SEL Saddik AVuong SGriwodz CDel Bimbo ACandan KJaimes A(2008)Exploring multimedia in a keyword spaceProceedings of the 16th ACM international conference on Multimedia10.1145/1459359.1459374(101-110)Online publication date: 26-Oct-2008
https://dl.acm.org/doi/10.1145/1459359.1459374
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents