research-article

Small-sample precision of ROC-related estimates

Authors:

Edward R. DoughertyAuthors Info & Claims

Bioinformatics, Volume 26, Issue 6

Pages 822 - 830

https://doi.org/10.1093/bioinformatics/btq037

Published: 01 March 2010 Publication History

Abstract

Motivation: The receiver operator characteristic (ROC) curves are commonly used in biomedical applications to judge the performance of a discriminant across varying decision thresholds. The estimated ROC curve depends on the true positive rate (TPR) and false positive rate (FPR), with the key metric being the area under the curve (AUC). With small samples these rates need to be estimated from the training data, so a natural question arises: How well do the estimates of the AUC, TPR and FPR compare with the true metrics?

Results: Through a simulation study using data models and analysis of real microarray data, we show that (i) for small samples the root mean square differences of the estimated and true metrics are considerable; (ii) even for large samples, there is only weak correlation between the true and estimated metrics; and (iii) generally, there is weak regression of the true metric on the estimated metric. For classification rules, we consider linear discriminant analysis, linear support vector machine (SVM) and radial basis function SVM. For error estimation, we consider resubstitution, three kinds of cross-validation and bootstrap. Using resampling, we show the unreliability of some published ROC results.

Availability: Companion web site at http://compbio.tgen.org/paper_supp/ROC/roc.html

Contact: [email protected]

Cited By

View all

Liao HLiu QHuang ZLu KYeung CZhang Y(2022)Accumulative Time Based Ranking Method to Reputation Evaluation in Information NetworksJournal of Computer Science and Technology10.1007/s11390-021-0471-437:4(960-974)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s11390-021-0471-4
Verdoja FGrangetto M(2020)Graph Laplacian for image anomaly detectionMachine Vision and Applications10.1007/s00138-020-01059-431:1-2Online publication date: 7-Feb-2020
https://dl.acm.org/doi/10.1007/s00138-020-01059-4
Dalton L(2016)Optimal ROC-Based Classification and Performance Analysis under Bayesian Uncertainty ModelsIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2015.246596613:4(719-729)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1109/TCBB.2015.2465966
Show More Cited By

Small-sample precision of ROC-related estimates
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Small-sample error estimation for bagged classification rules
Special issue on genomic signal processing

Application of ensemble classification rules in genomics and proteomics has become increasingly common. However, the problem of error estimation for these classification rules, particularly for bagging under the small-sample settings prevalent in ...
Small Sample Error Rate Estimation for k-NN Classifiers

Small sample error rate estimators for nearest-neighbor classifiers are examined and contrasted with the same estimators for three-nearest-neighbor classifiers. The performance of the bootstrap estimators, e0 and 0.632B, is considered relative to ...
Is cross-validation valid for small-sample microarray classification?

Motivation: Microarray classification typically possesses two striking attributes: (1) classifier design and error estimation are based on remarkably small samples and (2) cross-validation error estimation is employed in the majority of the papers. ...

Comments

Information & Contributors

Information

Published In

Bioinformatics Volume 26, Issue 6

March 2010

146 pages

ISSN:1367-4803

Issue’s Table of Contents

Publisher

Oxford University Press, Inc.

United States

Publication History

Published: 01 March 2010

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Liao HLiu QHuang ZLu KYeung CZhang Y(2022)Accumulative Time Based Ranking Method to Reputation Evaluation in Information NetworksJournal of Computer Science and Technology10.1007/s11390-021-0471-437:4(960-974)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s11390-021-0471-4
Verdoja FGrangetto M(2020)Graph Laplacian for image anomaly detectionMachine Vision and Applications10.1007/s00138-020-01059-431:1-2Online publication date: 7-Feb-2020
https://dl.acm.org/doi/10.1007/s00138-020-01059-4
Dalton L(2016)Optimal ROC-Based Classification and Performance Analysis under Bayesian Uncertainty ModelsIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2015.246596613:4(719-729)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1109/TCBB.2015.2465966
Kanwal NBostanci EClark A(2016)Evaluation Method, Dataset Size or Dataset ContentJournal of Mathematical Imaging and Vision10.1007/s10851-015-0626-455:3(378-400)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1007/s10851-015-0626-4
Kim SScalzo FTelesca DHu X(2015)Ensemble of sparse classifiers for high-dimensional biological dataInternational Journal of Data Mining and Bioinformatics10.1504/IJDMB.2015.06941612:2(167-183)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1504/IJDMB.2015.069416
Wang SLi DPetrick NSahiner BLinguraru MSummers R(2015)Optimizing area under the ROC curve using semi-supervised learningPattern Recognition10.1016/j.patcog.2014.07.02548:1(276-287)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.1016/j.patcog.2014.07.025
Florez-Lopez RRamon-Jeronimo J(2015)Enhancing accuracy and interpretability of ensemble strategies in credit risk assessment. A correlated-adjusted decision forest proposalExpert Systems with Applications: An International Journal10.1016/j.eswa.2015.02.04242:13(5737-5753)Online publication date: 1-Aug-2015
https://dl.acm.org/doi/10.1016/j.eswa.2015.02.042
Barros RBasgalupp MCarvalho A(2015)Investigating fitness functions for a hyper-heuristic evolutionary algorithm in the context of balanced and imbalanced data classificationGenetic Programming and Evolvable Machines10.1007/s10710-014-9235-z16:3(241-281)Online publication date: 1-Sep-2015
https://dl.acm.org/doi/10.1007/s10710-014-9235-z
Berrar D(2014)An empirical evaluation of ranking measures with respect to robustness to noiseJournal of Artificial Intelligence Research10.5555/2655713.265572149:1(241-267)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.5555/2655713.2655721
Mu TMiwa MTsujii JAnaniadou S(2014)DISCOVERING ROBUST EMBEDDINGS IN DISSIMILARITY SPACE FOR HIGH-DIMENSIONAL LINGUISTIC FEATURESComputational Intelligence10.1111/j.1467-8640.2012.00452.x30:2(285-315)Online publication date: 1-May-2014
https://dl.acm.org/doi/10.1111/j.1467-8640.2012.00452.x
Show More Cited By

Abstract

Cited By

Recommendations

Small-sample error estimation for bagged classification rules

Small Sample Error Rate Estimation for k-NN Classifiers

Is cross-validation valid for small-sample microarray classification?

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Share

Share this Publication link

Share on social media

Affiliations