Abstract
We describe a permutation procedure used extensively in classification problems in computational biology and medical imaging. We empirically study the procedure on simulated data and real examples from neuroimaging studies and DNA microarray analysis. A theoretical analysis is also suggested to assess the asymptotic behavior of the test. An interesting observation is that concentration of the permutation procedure is controlled by a Rademacher average which also controls the concentration of empirical errors to expected errors.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bousquet, O., Elisseeff, A.: Stability and generalization. Journal Machine Learning Research 2, 499–526 (2002)
Efron, B.: The Jackknife, The Bootstrap, and Other Resampling Plans. SIAM, Philadelphia (1982)
Efron, B., Tibshirani, R.: An introduction to the bootstrap, p. 436. Chapman and Hall, Boca Raton (1993)
Fischl, B., Dale, A.M.: Measuring the thickness of the human cerebral cortex from magnetic resonance images. PNAS 26, 11050–11055 (2000)
Fischl, B., Sereno, M.I., Tootell, R.B.H., Dale, A.M.: High-resolution intersubject averaging and a coordinate system for the cortical surface. Human Brain Mapping 8, 262–284 (1999)
Golland, P., Fischl, B.: Permutation tests for classification: Towards statistical significance in image-based studies. In: Taylor, C.J., Noble, J.A. (eds.) IPMI 2003. LNCS, vol. 2732, pp. 330–341. Springer, Heidelberg (2003)
Golub, T.R., Slonim, D., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J.P., Coller, H., Loh, M.L., Downing, J.R., Caligiuri, M.A., Bloomfield, C.D., Lander, E.S.: Molecular Classification of Cancer: Class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)
Good, P.: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypothesis. Springer, Heidelberg (1994)
Hsing, T., Attoor, S., Dougherty, E.: Relation between permutation-test p values and classifier error estimates. Machine Learning 52, 11–30 (2003)
Kendall, M.G.: The treatment of ties in ranking problems. Biometrika 33, 239–251 (1945)
Kutin, S., Niyogi, P.: Almost-everywhere algorithmic stability and generalization error. Technical report TR-2002-03, University of Chicago (2002)
Mukherjee, S., Golland, P., Panchenko, D.: Permutation tests for classification. AI Memo 2003-019, Massachusetts Institute of Technology (2003)
Mukherjee, S., Tamayo, P., Rogers, S., Rifkin, R., Engle, A., Campbell, C., Golub, T.R., Mesirov, J.P.: Estimating dataset size requirements for classifying dna microarray data. Journal Computational Biology 10(2), 119–142 (2003)
Nichols, T.E., Holmes, A.P.: Nonparametric permutation tests for functional neuroimaging: A primer with examples. Human Brain Mapping 15, 1–25 (2001)
Pomeroy, S., Tamayo, P., Gaasenbeek, M., Sturlia, L., Angelo, M., Kim, Y.H., McLaughlin, M.E., Goumnerova, L.C., Black, P.M., Lauand, C., Lau, J.C., Allen, J.C., Zagzag, D., Olson, M.M., Curran, T., Wetmore, C., Biegel, J.A., Poggio, T., Mukherjee, S., Rifkin, R., Califano, A., Stolovitzky, G., Louis, D.N., Mesirov, J.P., Lander, E.S., Golub, T.R.: Prediction of embryonal tumor outcome based on gene expression. Nature 415, 436–442 (2002)
Slonim, D., Tamayo, P., Mesirov, J.P., Golub, T.R., Lander, E.: Class prediction and discovery using gene expression data. In: Proceedings of the Fourth Annual Conference on Computational Molecular Biology (RECOMB), pp. 263–272 (2000)
Talagrand, M.: Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l’I.H.E.S. 81, 73–205 (1995)
van der Vaart, A., Wellner, J.: Weak convergence and Empirical Processes With Applications to Statistics. Springer, Heidelberg (1996)
Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, Chichester (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Golland, P., Liang, F., Mukherjee, S., Panchenko, D. (2005). Permutation Tests for Classification. In: Auer, P., Meir, R. (eds) Learning Theory. COLT 2005. Lecture Notes in Computer Science(), vol 3559. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11503415_34
Download citation
DOI: https://doi.org/10.1007/11503415_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26556-6
Online ISBN: 978-3-540-31892-7
eBook Packages: Computer ScienceComputer Science (R0)