Abstract
A fundamental approach in neuroscience research is to test hypotheses based on neuropsychological and behavioral measures, i.e., whether certain factors (e.g., related to life events) are associated with an outcome (e.g., depression). In recent years, deep learning has become a potential alternative approach for conducting such analyses by predicting an outcome from a collection of factors and identifying the most “informative” ones driving the prediction. However, this approach has had limited impact as its findings are not linked to statistical significance of factors supporting hypotheses. In this article, we proposed a flexible and scalable approach based on the concept of permutation testing that integrates hypothesis testing into the data-driven deep learning analysis. We apply our approach to the yearly self-reported assessments of 621 adolescent participants of the National Consortium of Alcohol and Neurodevelopment in Adolescence (NCANDA) to predict negative valence, a symptom of major depressive disorder according to the NIMH Research Domain Criteria (RDoC). Our method successfully identifies categories of risk factors that further explain the symptom.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Gurd, J., Kischka, U., Marshall, J.C., Halligan, P.W.: Handbook of Clinical Neuropsychology, 2nd edn. Oxford University Press, Oxford (2010)
Blakesley, R., et al.: Comparisons of methods for multiple hypothesis testing in neuropsychological research. Neuropsychology 23, 255–264 (2009)
Kang, M., et al.: Prediction of cognitive impairment via deep learning trained with multi-center neuropsychological test data. BMC Med. Inform. Decis. Mak. 19 (2019)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Golland, P., Liang, F., Mukherjee, S., Panchenko, D.: Permutation tests for classification. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS (LNAI), vol. 3559, pp. 501–515. Springer, Heidelberg (2005). https://doi.org/10.1007/11503415_34
Ojala, M., Garriga, G.C.: Permutation tests for studying classifier performance. J. Mach. Learn. Res. 11(6) (2010)
Fisher, A., Rudin, C., Dominici, F.: All models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res. 20(177), 1–81 (2019)
Frank, E., Witten, I.H.: Using a permutation test for attribute selection in decision trees. In: Shavlik, J.W. (ed.) Proceedings of the Fifteenth International Conference on Machine Learning, pp. 152–160. Morgan Kaufmann (1998)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Altmann, A., Toloşi, L., Sander, O., Lengauer, T.: Permutation importance: a corrected feature importance measure. Bioinformatics 26(10), 1340–1347 (2010)
Ishwaran, H., Lu, M.: Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival. Stat. Med. 38(4), 558–582 (2019)
Mi, X., Zou, B., Zou, F., Hu, J.: Permutation-based identification of important biomarkers for complex diseases via machine learning models. Nat. Commun. 12(1), 1–12 (2021)
Insel, T., et al.: Research Domain Criteria (RDoC): toward a new classification framework for research on mental disorders (2010)
Pohl, K.M., et al.: The NCANDA_PUBLIC_6Y_REDCAP_V01 data release of the national consortium on alcohol and neurodevelopment in adolescence (NCANDA) (2021). https://doi.org/10.7303/syn25606546
Brown, S.A., et al.: The national consortium on alcohol and neurodevelopment in adolescence (NCANDA): a multisite study of adolescent development and substance use. J. Stud. Alcohol Drugs 76(6), 895–908 (2015)
Gioia, G.A., Isquith, P.K., Guy, S.C., Kenworthy, L.: Test review behavior rating inventory of executive function. Child Neuropsychol. 6(3), 235–238 (2000)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates Inc. (2019)
Brodersen, K.H., Ong, C.S., Stephan, K.E., Buhmann, J.M.: The balanced accuracy and its posterior distribution. In: 2010 20th International Conference on Pattern Recognition, pp. 3121–3124. IEEE (2010)
Suhara, Y., Xu, Y., Pentland, A.: DeepMood: forecasting depressed mood based on self-reported histories via recurrent neural networks, pp. 715–724 (2017)
Rottenberg, J., Gotlib, I.H.: Socioemotional functioning in depression. In: Mood Disorders: A Handbook of Science and Practice, pp. 61–77 (2004)
Klinger-Koenig, J., Hertel, J., Terock, J., Voelzke, H., Van der Auwera, S., Grabe, H.J.: Predicting physical and mental health symptoms: additive and interactive effects of difficulty identifying feelings, neuroticism and extraversion. J. Psychosom. Res. 115, 14–23 (2018)
Watson, D., Stasik, S.M., Ellickson-Larew, S., Stanton, K.: Extraversion and psychopathology: a facet-level analysis. J. Abnorm. Psychol. 124(2), 432 (2015)
De Venter, M., Demyttenaere, K., Bruffaerts, R.: The relationship between adverse childhood experiences and mental health in adulthood. A systematic literature review. Tijdschr. Psychiatr. 55(4), 259–268 (2013)
Kendler, K.S., Bulik, C.M., Silberg, J., Hettema, J.M., Myers, J., Prescott, C.A.: Childhood sexual abuse and adult psychiatric and substance use disorders in women: an epidemiological and cotwin control analysis. Arch. Gen. Psychiatry 57(10), 953–959 (2000)
Gotlib, I.H., Joormann, J.: Cognition and depression: current status and future directions. Annu. Rev. Clin. Psychol. 6, 285–312 (2010)
Lamblin, M., Murawski, C., Whittle, S., Fornito, A.: Social connectedness, mental health and the adolescent brain. Neurosci. Biobehav. Rev. 80, 57–68 (2017)
Ueno, K.: The effects of friendship networks on adolescent depressive symptoms. Soc. Sci. Res. 34(3), 484–510 (2005)
Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Shapley, L.S.: A value for n-person games, contributions to the theory of games, vol. 2, pp. 307–317 (1953)
Gosling, S.D., Rentfrow, P.J., Swann, W.B., Jr.: A very brief measure of the Big-Five personality domains. J. Res. Pers. 37(6), 504–528 (2003)
Connor-Smith, J.K., Compas, B.E., Wadsworth, M.E., Thomsen, A.H., Saltzman, H.: Responses to stress in adolescence: measurement of coping and involuntary stress responses. J. Consult. Clin. Psychol. 68(6), 976 (2000)
Cyders, M.A., Smith, G.T., Spillane, N.S., Fischer, S., Annus, A.M., Peterson, C.: Integration of impulsivity and positive mood to predict risky behavior: development and validation of a measure of positive urgency. Psychol. Assess. 19(1), 107 (2007)
Masten, A.S., Neemann, J., Andenas, S.: Life events and adjustment in adolescents: the significance of event independence, desirability, and chronicity. J. Res. Adolesc. 4(1), 71–97 (1994)
Bernstein, D.P., et al.: Initial reliability and validity of a new retrospective measure of child abuse and neglect. Am. J. Psychiatry 152, 1535–1537 (1994)
Alzueta, E., et al.: Risk for depression tripled during the COVID-19 pandemic in emerging adults followed for the last 8 years. Psychol. Med. 1–8 (2021)
Acknowledgements
This study was in part supported by the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA) by means of research grants from the National Institute on Alcohol Abuse and Alcoholism (NIAAA) AA021697 (PI: KMP) and AA028840 (PI: QZ). The research was also supported by the Stanford Human-Centered Artificial Intelligence (HAI) Google Cloud Credit (PI: KMP).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Paschali, M., Zhao, Q., Adeli, E., Pohl, K.M. (2022). Bridging the Gap Between Deep Learning and Hypothesis-Driven Analysis via Permutation Testing. In: Rekik, I., Adeli, E., Park, S.H., Cintas, C. (eds) Predictive Intelligence in Medicine. PRIME 2022. Lecture Notes in Computer Science, vol 13564. Springer, Cham. https://doi.org/10.1007/978-3-031-16919-9_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-16919-9_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16918-2
Online ISBN: 978-3-031-16919-9
eBook Packages: Computer ScienceComputer Science (R0)
