Article

The relationship between Precision-Recall and ROC curves

Authors:

Mark GoadrichAuthors Info & Claims

ICML '06: Proceedings of the 23rd international conference on Machine learning

Pages 233 - 240

https://doi.org/10.1145/1143844.1143874

Published: 25 June 2006 Publication History

Abstract

Receiver Operator Characteristic (ROC) curves are commonly used to present results for binary decision problems in machine learning. However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance. We show that a deep connection exists between ROC space and PR space, such that a curve dominates in ROC space if and only if it dominates in PR space. A corollary is the notion of an achievable PR curve, which has properties much like the convex hull in ROC space; we show an efficient algorithm for computing this curve. Finally, we also note differences in the two types of curves are significant for algorithm design. For example, in PR space it is incorrect to linearly interpolate between points. Furthermore, algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve.

References

[1]

Bockhorst, J., & Craven, M. (2005). Markov networks for detecting overlapping elements in sequence data. Neural Information Processing Systems 17 (NIPS). MIT Press.

[2]

Bradley, A. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145--1159.

Digital Library

[3]

Bunescu, R., Ge, R., Kate, R., Marcotte, E., Mooney, R., Ramani, A., & Wong, Y. (2004). Comparative Experiments on Learning Information Extractors for Proteins and their Interactions. Journal of Artificial Intelligence in Medicine, 139--155.

Digital Library

[4]

Cormen, T. H., Leiserson, Charles, E., & Rivest, R. L. (1990). Introduction to algorithms. MIT Press.

Digital Library

[5]

Cortes, C., & Mohri, M. (2003). AUC optimization vs. error rate minimization. Neural Information Processing Systems 15 (NIPS). MIT Press.

[6]

Davis, J., Burnside, E., Dutra, I., Page, D., Ramakrishnan, R., Costa, V. S., & Shavlik, J. (2005). View learning for statistical relational learning: With an application to mammography. Proceeding of the 19th International Joint Conference on Artificial Intelligence. Edinburgh, Scotland.

Digital Library

[7]

Drummond, C., & Holte, R. (2000). Explicitly representing expected cost: an alternative to ROC representation. Proceeding of Knowledge Discovery and Datamining (pp. 198--207).

Digital Library

[8]

Drummond, C., & Holte, R. C. (2004). What ROC curves can't do (and cost curves can). ROCAI (pp. 19--26).

[9]

Ferri, C., Flach, P., & Henrandez-Orallo, J. (2002). Learning decision trees using area under the ROC curve. Proceedings of the 19th International Conference on Machine Learning (pp. 139--146). Morgan Kaufmann.

Digital Library

[10]

Freund, Y., Iyer, R., Schapire, R., & Singer, Y. (1998). An efficient boosting algorithm for combining preferences. Proceedings of the 15th International Conference on Machine Learning (pp. 170--178). Madison, US: Morgan Kaufmann Publishers, San Francisco, US.

Digital Library

[11]

Goadrich, M., Oliphant, L., & Shavlik, J. (2004). Learning ensembles of first-order clauses for recall-precision curves: A case study in biomedical information extraction. Proceedings of the 14th International Conference on Inductive Logic Programming (ILP). Porto, Portugal.

Digital Library

[12]

Herschtal, A., & Raskutti, B. (2004). Optimising area under the ROC curve using gradient descent. Proceedings of the 21st International Conference on Machine Learning (p. 49). New York, NY, USA: ACM Press.

Digital Library

[13]

Joachims, T. (2005). A support vector method for multi-variate performance measures. Proceedings of the 22nd International Conference on Machine Learning. ACM Press.

Digital Library

[14]

Kok, S., & Domingos, P. (2005). Learning the structure of Markov Logic Networks. Proceedings of 22nd International Conference on Machine Learning (pp. 441--448). ACM Press.

Digital Library

[15]

Macskassy, S., & Provost, F. (2005). Suspicion scoring based on guilt-by-association, collective inference, and focused data access. International Conference on Intelligence Analysis.

[16]

Manning, C., & Schutze, H. (1999). Foundations of statistical natural language processing. MIT Press.

Digital Library

[17]

Prati, R., & Flach, P. (2005). ROCCER: an algorithm for rule learning based on ROC analysis. Proceeding of the 19th International Joint Conference on Artificial Intelligence. Edinburgh, Scotland.

Digital Library

[18]

Provost, F., Fawcett, T., & Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. Proceeding of the 15th International Conference on Machine Learning (pp. 445--453). Morgan Kaufmann, San Francisco, CA.

Digital Library

[19]

Raghavan, V., Bollmann, P., & Jung, G. S. (1989). A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst., 7, 205--229.

Digital Library

[20]

Singla, P., & Domingos, P. (2005). Discriminative training of Markov Logic Networks. Proceedings of the 20th National Conference on Artificial Intelligene (AAAI) (pp. 868--873). AAAI Press.

Digital Library

[21]

Srinivasan, A. (2003). The Aleph Manual Version 4. http://web.comlab.ox.ac.uk/oucl/research/areas/machlearn/Aleph/.

[22]

Yan, L., Dodier, R., Mozer, M., & Wolniewicz, R. (2003). Optimizing classifier performance via the Wilcoxon-Mann-Whitney statistics. Proceedings of the 20th International Conference on Machine Learning.

Cited By

Sang-aram CBrowaeys RSeurinck RSaeys Y(2024)Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomicseLife10.7554/eLife.88431.312Online publication date: 24-May-2024
https://doi.org/10.7554/eLife.88431.3
Sang-aram CBrowaeys RSeurinck RSaeys Y(2024)Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomicseLife10.7554/eLife.8843112Online publication date: 24-May-2024
https://doi.org/10.7554/eLife.88431
Bashizadeh MSoufizadeh PZamiri MLamei ASotoudehnejad MDaneshmand MGhodrati MIsavi EAkbarein H(2024)An Overview of Artificial Intelligence Applications in Prediction and Diagnosis of Diseases Occurrence in Veterinary Medicine: Challenges and TechniquesEltiam10.61186/eltiamj.10.2.72:10(71-87)Online publication date: 10-Mar-2024
https://doi.org/10.61186/eltiamj.10.2.7
Show More Cited By

Index Terms

The relationship between Precision-Recall and ROC curves

Recommendations

Precision-Recall-Gain curves: PR analysis done right
NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1

Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating ...
ROC curves and nonrandom data

This paper shows that ROC curves that are constructed with nonrandom data are biased.The magnitude of this bias is explored using simulations.A procedure for plotting consistent ROC curves is introduced.The presented procedure works well with simulated ...
ROC curves in cost space

ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '06: Proceedings of the 23rd international conference on Machine learning

June 2006

1154 pages

ISBN:1595933832

DOI:10.1145/1143844

Program Chairs:
William Cohen,
Andrew Moore

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

ICML '06 Paper Acceptance Rate 140 of 548 submissions, 26%;

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3,305
Total Citations
View Citations
14,912
Total Downloads

Downloads (Last 12 months)1,485
Downloads (Last 6 weeks)175

Reflects downloads up to 14 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sang-aram CBrowaeys RSeurinck RSaeys Y(2024)Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomicseLife10.7554/eLife.88431.312Online publication date: 24-May-2024
https://doi.org/10.7554/eLife.88431.3
Sang-aram CBrowaeys RSeurinck RSaeys Y(2024)Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomicseLife10.7554/eLife.8843112Online publication date: 24-May-2024
https://doi.org/10.7554/eLife.88431
Bashizadeh MSoufizadeh PZamiri MLamei ASotoudehnejad MDaneshmand MGhodrati MIsavi EAkbarein H(2024)An Overview of Artificial Intelligence Applications in Prediction and Diagnosis of Diseases Occurrence in Veterinary Medicine: Challenges and TechniquesEltiam10.61186/eltiamj.10.2.72:10(71-87)Online publication date: 10-Mar-2024
https://doi.org/10.61186/eltiamj.10.2.7
Ewart NAjeneye FGroves L(2024)Direct Antiglobulin Test in Predicting the Severity of Hyperbilirubinemia and Haemolytic Disease of the NewbornEuropean Journal of Medical and Health Research10.59324/ejmhr.2024.2(3).112:3(81-88)Online publication date: 1-May-2024
https://doi.org/10.59324/ejmhr.2024.2(3).11
BAYLAN PDEMİREL N(2024)QUANTIFYING THE IMPACT OF RISK FACTORS ON DIRECT COMPENSATION PROPERTY DAMAGE IN CANADIAN AUTOMOBILE INSURANCEKANADA OTOMOBİL SİGORTASINDA RİSK FAKTÖRLERİNİN DOĞRUDAN TAZMİN EDİLEN MADDİ HASAR ÜZERİNDEKİ ETKİSİNİN DEĞERLENDİRİLMESİNicel Bilimler Dergisi10.51541/nicel.1397941Online publication date: 17-Jan-2024
https://doi.org/10.51541/nicel.1397941
Pratham N. Surjuse Shweta G. Deogade Shantanu S. Dethe Sakshi P. Chahare Dr. Nitin Janwe (2024)Video Processing Based Tracking and Vehicle IdentificationInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-17961(415-425)Online publication date: 30-Apr-2024
https://doi.org/10.48175/IJARSCT-17961
Farooq BGanie S(2024)Deep Learning Techniques for Alzheimer's Disease DetectionDeep Learning Approaches for Early Diagnosis of Neurodegenerative Diseases10.4018/979-8-3693-1281-0.ch005(91-111)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-1281-0.ch005
Abubakar Ali Onana Oyana CSalum O(2024)Domestic Cats Facial Expression Recognition Based on Convolutional Neural NetworksInternational Journal of Engineering and Advanced Technology10.35940/ijeat.E4484.1305062413:5(45-52)Online publication date: 30-Jun-2024
https://doi.org/10.35940/ijeat.E4484.13050624
Suliman FAnayi FPackianather M(2024)Electrical Faults Analysis and Detection in Photovoltaic Arrays Based on Machine Learning ClassifiersSustainability10.3390/su1603110216:3(1102)Online publication date: 27-Jan-2024
https://doi.org/10.3390/su16031102
Andersson RBermejo-García JAgujetas RCronhjort MChilo J(2024)Smartphone IMU Sensors for Human Identification through Hip Joint Angle AnalysisSensors10.3390/s2415476924:15(4769)Online publication date: 23-Jul-2024
https://doi.org/10.3390/s24154769
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents