research-article

The Power of Speech in the Wild: Discriminative Power of Daily Voice Diaries in Understanding Auditory Verbal Hallucinations Using Deep Learning

Authors:

Ayesha Chander,

Serguei Pakhomov,

Andrew CampbellAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 7, Issue 3

Article No.: 133, Pages 1 - 29

https://doi.org/10.1145/3610890

Published: 27 September 2023 Publication History

Abstract

Mobile phone sensing is increasingly being used in clinical research studies to assess a variety of mental health conditions (e.g., depression, psychosis). However, in-the-wild speech analysis -- beyond conversation detecting -- is a missing component of these mobile sensing platforms and studies. We augment an existing mobile sensing platform with a daily voice diary to assess and predict the severity of auditory verbal hallucinations (i.e., hearing sounds or voices in the absence of any speaker), a condition that affects people with and without psychiatric or neurological diagnoses. We collect 4809 audio diaries from N=384 subjects over a one-month-long study period. We investigate the performance of various deep-learning architectures using different combinations of sensor behavioral streams (e.g., voice, sleep, mobility, phone usage, etc.) and show the discriminative power of solely using audio recordings of speech as well as automatically generated transcripts of the recordings; specifically, our deep learning model achieves a weighted f-1 score of 0.78 solely from daily voice diaries. Our results surprisingly indicate that a simple periodic voice diary combined with deep learning is sufficient enough of a signal to assess complex psychiatric symptoms (e.g., auditory verbal hallucinations) collected from people in the wild as they go about their daily lives.

References

[1]

Daniel A Adler, Dror Ben-Zeev, Vincent WS Tseng, John M Kane, Rachel Brian, Andrew T Campbell, Marta Hauser, Emily A Scherer, and Tanzeem Choudhury. 2020. Predicting early warning signs of psychotic relapse from passive sensing data: an approach using encoder-decoder neural networks. JMIR mHealth and uHealth 8, 8 (2020), e19962.

[2]

Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning. PMLR, 173--182.

[3]

Nancy C Andreasen and Michael Flaum. 1991. Schizophrenia: the characteristic symptoms. Schizophrenia bulletin 17, 1 (1991), 27--49.

[4]

Nancy C Andreasen and William M Grove. 1986. Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophrenia bulletin 12, 3 (1986), 348--359.

[5]

American Psychiatric Association. 2013. Diagnostic and statistical manual of mental disorders: DSM-5. Vol. 5. American psychiatric association Washington, DC.

[6]

Min Hane Aung, Mark Matthews, and Tanzeem Choudhury. 2017. Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies. Depression and anxiety 34, 7 (2017), 603--609.

[7]

Yi Ji Bae, Midan Shim, and Won Hee Lee. 2021. Schizophrenia detection using machine learning approach from social media content. Sensors 21, 17 (2021), 5924.

[8]

Jakob E Bardram and Aleksandar Matic. 2020. A decade of ubiquitous computing research in mental health. IEEE Pervasive Computing 19, 1 (2020), 62--72.

[9]

Robert H Belmaker and Galila Agam. 2008. Major depressive disorder. New England Journal of Medicine 358, 1 (2008), 55--68.

[10]

Dror Ben-Zeev, Rachel Brian, Rui Wang, Weichen Wang, Andrew T Campbell, Min SH Aung, Michael Merrill, Vincent WS Tseng, Tanzeem Choudhury, Marta Hauser, et al. 2017. CrossCheck: Integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatric rehabilitation journal 40, 3 (2017), 266.

[11]

Dror Ben-Zeev, Emily A Scherer, Rui Wang, Haiyi Xie, and Andrew T Campbell. 2015. Next-generation psychiatric assessment: Using smartphone sensors to monitor behavior and mental health. Psychiatric rehabilitation journal 38, 3 (2015), 218.

[12]

Dror Ben-Zeev, Rui Wang, Saeed Abdullah, Rachel Brian, Emily A Scherer, Lisa A Mistler, Marta Hauser, John M Kane, Andrew Campbell, and Tanzeem Choudhury. 2016. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatric services 67, 5 (2016), 558--561.

[13]

Josef Bless, Runar Smelror, Ingrid Agartz, and Kenneth Hugdahl. 2017. SA110. Using a Smartphone App to Assess Auditory Hallucinations in Adolescent Schizophrenia: Is This the Way to go for Better Control Over Voices? Schizophrenia bulletin 43, Suppl 1 (2017), S152.

[14]

Mehdi Boukhechba, Yu Huang, Philip Chow, Karl Fua, Bethany A Teachman, and Laura E Barnes. 2017. Monitoring social anxiety from mobility and communication patterns. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. 749--753.

Digital Library

[15]

Vera Brink, Catheleine van Driel, Saliha El Bouhaddani, Klaas J Wardenaar, Lieke van Domburgh, Barbara Schaefer, Marije van Beilen, Agna A Bartels-Velthuis, and Wim Veling. 2020. Spontaneous discontinuation of distressing auditory verbal hallucinations in a school-based sample of adolescents: a longitudinal study. European child & adolescent psychiatry 29 (2020), 777--790.

[16]

Xiao Chang, Yi-Bin Xi, Long-Biao Cui, Hua-Ning Wang, Jin-Bo Sun, Yuan-Qiang Zhu, Peng Huang, Guusje Collin, Kang Liu, Min Xi, et al. 2015. Distinct inter-hemispheric dysconnectivity in schizophrenia patients with and without auditory verbal hallucinations. Scientific Reports 5, 1 (2015), 1--12.

[17]

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785--794.

Digital Library

[18]

Xingui Chen, Gong-Jun Ji, Chunyan Zhu, Xiaomeng Bai, Lu Wang, Kongliang He, Yaxiang Gao, Longxiang Tao, Fengqiong Yu, Yanghua Tian, et al. 2019. Neural correlates of auditory verbal hallucinations in schizophrenia and the therapeutic response to theta-burst transcranial magnetic stimulation. Schizophrenia bulletin 45, 2 (2019), 474--483.

[19]

Zhenyu Chen, Mu Lin, Fanglin Chen, Nicholas D Lane, Giuseppe Cardone, Rui Wang, Tianxing Li, Yiqiang Chen, Tanzeem Choudhury, and Andrew T Campbell. 2013. Unobtrusive sleep monitoring using smartphones. In Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare. ICST (Institute for Computer Sciences, Social-Informatics and ..., 145--152.

Digital Library

[20]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).

[21]

Michael A Cohn, Matthias R Mehl, and James W Pennebaker. 2004. Linguistic markers of psychological change surrounding September 11, 2001. Psychological science 15, 10 (2004), 687--693.

[22]

Cheryl M Corcoran, Facundo Carrillo, Diego Fernández-Slezak, Gillinder Bedi, Casimir Klim, Daniel C Javitt, Carrie E Bearden, and Guillermo A Cecchi. 2018. Prediction of psychosis across protocols and risk cohorts using automated language analysis. World Psychiatry 17, 1 (2018), 67--75.

[23]

H Corona-Hernández, SG Brederoo, JN de Boer, and IEC Sommer. 2022. A data-driven linguistic characterization of hallucinated voices in clinical and non-clinical voice-hearers. Schizophrenia Research 241 (2022), 210--217.

[24]

Benjamin Sage Crosier, Rachel Marie Brian, and Dror Ben-Zeev. 2016. Using Facebook to reach people who experience auditory hallucinations. Journal of medical Internet research 18, 6 (2016), e160.

[25]

Nicholas Cummins, Alice Baird, and Bjoern W Schuller. 2018. Speech analysis for health: Current state-of-the-art and the increasing impact of deep learning. Methods 151 (2018), 41--54.

[26]

Bruce N Cuthbert et al. 2014. The RDoC framework: continuing commentary. World Psychiatry 13, 2 (2014), 196.

[27]

Kirstin Daalman, Marco PM Boks, Kelly MJ Diederen, Antoin D de Weijer, Jan Dirk Blom, René S Kahn, and Iris EC Sommer. 2011. The same or different? A phenomenological comparison of auditory verbal hallucinations in healthy and psychotic individuals. The Journal of clinical psychiatry 72, 3 (2011), 0--0.

[28]

K Daalman, IEC Sommer, EM Derks, and ER Peters. 2013. Cognitive biases and auditory verbal hallucinations in healthy and clinical individuals. Psychological Medicine 43, 11 (2013), 2339--2347.

[29]

Saskia de Leede-Smith and Emma Barkus. 2013. A comprehensive review of auditory verbal hallucinations: lifetime prevalence, correlates and mechanisms in healthy and clinical individuals. Frontiers in human neuroscience 7 (2013), 367.

[30]

Philippe Delespaul, Marten devries, and Jim van Os. 2002. Determinants of occurrence and recovery from hallucinations in daily life. Social psychiatry and psychiatric epidemiology 37, 3 (2002), 97--104.

[31]

Sasha Deutsch-Link. 2016. Language in schizophrenia: What we can learn from quantitative text analysis. 2047 (2016).

[32]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[33]

Clement Donde, David Luck, Stephanie Grot, David I Leitman, Jerome Brunelin, and Frederic Haesebaert. 2017. Tone-matching ability in patients with schizophrenia: A systematic review and meta-analysis. Schizophrenia Research 181 (2017), 94--99.

[34]

Roisin Doyle, Niall Turner, Felicity Fanning, Daria Brennan, Laoise Renwick, Elizabeth Lawlor, and Mary Clarke. 2014. First-episode psychosis and disengagement from treatment: a systematic review. Psychiatric Services 65, 5 (2014), 603--611.

[35]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, Vol. 96. 226--231.

Digital Library

[36]

Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. 2015. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE transactions on affective computing 7, 2 (2015), 190--202.

[37]

Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia. 1459--1462.

Digital Library

[38]

Florian Eyben, Martin Wöllmer, and Björn Schuller. 2020. openSMILE. https://github.com/audeering/opensmile.

[39]

Denzil Ferreira, Vassilis Kostakos, and Anind K Dey. 2015. AWARE: mobile context instrumentation framework. Frontiers in ICT 2 (2015), 6.

[40]

Judith M Ford. 2016. Studying auditory verbal hallucinations using the RDoC framework. Psychophysiology 53, 3 (2016), 298--304.

[41]

William I Fraser, Kathleen M King, Philip Thomas, and Robert E Kendell. 1986. The diagnosis of schizophrenia by language analysis. The British Journal of Psychiatry 148, 3 (1986), 275--278.

[42]

Daniel Freeman and Philippa A Garety. 2003. Connecting neurosis and psychosis: the direct influence of emotion on delusions and hallucinations. Behaviour research and therapy 41, 8 (2003), 923--947.

[43]

Christopher D Frith and D John Done. 1988. Towards a neuropsychology of schizophrenia. The British Journal of Psychiatry 153, 4 (1988), 437--443.

[44]

Kelvin MT Fung, Hector WH Tsang, and Patrick W Corrigan. 2008. Self-stigma of people with schizophrenia as predictor of their adherence to psychosocial treatment. Psychiatric rehabilitation journal 32, 2 (2008), 95.

[45]

Google Activity Recognition Api. 2019. Google Activity Recognition Api. https://developers.google.com/android/reference/com/google/android/gms/location/ActivityRecognitionClient.

[46]

Petra C Gronholm, Graham Thornicroft, Kristin R Laurens, and Sara Evans-Lacko. 2017. Mental health-related stigma and pathways to care for people at risk of psychotic disorders or experiencing first-episode psychosis: a systematic review. Psychological medicine 47, 11 (2017), 1867--1879.

[47]

Agnes Grünerbl, Amir Muaremi, Venet Osmani, Gernot Bahle, Stefan Oehler, Gerhard Tröster, Oscar Mayora, Christian Haring, and Paul Lukowicz. 2014. Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE journal of biomedical and health informatics 19, 1 (2014), 140--148.

[48]

Gillian Haddock, J McCarron, N Tarrier, and EB Faragher. 1999. Scales to measure dimensions of hallucinations and delusions: the psychotic symptom rating scales (PSYRATS). Psychological medicine 29, 4 (1999), 879--889.

[49]

S Hartley, G Haddock, D Vasconcelos e Sa, R Emsley, and C Barrowclough. 2014. An experience sampling study of worry and rumination in psychosis. Psychological Medicine 44, 8 (2014), 1605--1614.

[50]

Nik Wahidah Hashim, Mitch Wilkes, Ronald Salomon, Jared Meggs, and Daniel J France. 2017. Evaluation of voice acoustics as predictors of clinical depression scores. Journal of Voice 31, 2 (2017), 256--e1.

[51]

Karl Herholz, Alexander Thiel, Klaus Wienhard, Uwe Pietrzyk, H-M Von Stockhausen, Hans Karbe, J Kessler, Thomas Bruckbauer, Marco Halber, and W-D Heiss. 1996. Individual functional anatomy of verb generation. Neuroimage 3, 3 (1996), 185--194.

[52]

RE Hoffman, M Varanko, J Gilmore, and AL Mishara. 2008. Experiential features used by patients with schizophrenia to differentiate 'voices' from ordinary verbal thought. Psychological medicine 38, 8 (2008), 1167--1176.

[53]

Ralph E Hoffman. 1986. Verbal hallucinations and language production processes in schizophrenia. Behavioral and Brain Sciences 9, 3 (1986), 503--517.

[54]

Ralph E Hoffman. 2007. A social deafferentation hypothesis for induction of active schizophrenia. Schizophrenia bulletin 33, 5 (2007), 1066--1070.

[55]

Daniel C Javitt. 2009. When doors of perception close: bottom-up models of disrupted cognition in schizophrenia. Annual review of clinical psychology 5 (2009), 249--275.

[56]

Daniel C Javitt and Robert A Sweet. 2015. Auditory dysfunction in schizophrenia: integrating clinical and basic features. Nature Reviews Neuroscience 16, 9 (2015), 535--550.

[57]

Louise C. Johns, Mary Cannon, Nicola Singleton, Robin M. Murray, Michael Farrell, Traolach Brugha, Paul Bebbington, Rachel Jenkins, and Howard Meltzer. 2004. Prevalence and correlates of self-reported psychotic symptoms in the British population. British Journal of Psychiatry 185, 4 (Oct. 2004), 298--305. https://doi.org/10.1192/bjp.185.4.298

[58]

Louise C Johns, Kristiina Kompus, Melissa Connell, Clara Humpston, Tania M Lincoln, Eleanor Longden, Antonio Preti, Ben Alderson-Day, Johanna C Badcock, Matteo Cella, et al. 2014. Auditory verbal hallucinations in persons with and without a need for care. Schizophrenia bulletin 40, Suppl_4 (2014), S255--S264.

[59]

Louise C Johns, James Y Nazroo, Paul Bebbington, and Elizabeth Kuipers. 2002. Occurrence of hallucinatory experiences in a community sample and ethnic variations. The British Journal of Psychiatry 180, 2 (2002), 174--178.

[60]

Ewa Kacewicz, James W Pennebaker, Matthew Davis, Moongee Jeon, and Arthur C Graesser. 2013. Pronoun use reflects standings in social hierarchies. Journal of Language and Social Psychology (2013), 0261927X13502654.

[61]

Se Hyun Kim, Hee Yeon Jung, Samuel S Hwang, Jae Seung Chang, Yeni Kim, Yong Min Ahn, and Yong Sik Kim. 2010. The usefulness of a self-report questionnaire measuring auditory verbal hallucinations. Progress in Neuro-Psychopharmacology and Biological Psychiatry 34, 6 (2010), 968--973.

[62]

David Kimhy, Melanie M Wall, Marie C Hansen, Julia Vakhrusheva, C Jean Choi, Philippe Delespaul, Nicholas Tarrier, Richard P Sloan, and Dolores Malaspina. 2017. Autonomic Regulation and Auditory Hallucinations in Individuals With Schizophrenia: An Experience Sampling Study. Schizophrenia Bulletin 43, 4 (Feb. 2017), 754--763. https://doi.org/10.1093/schbul/sbw219

[63]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[64]

Mrinal Kumar, Mark Dredze, Glen Coppersmith, and Munmun De Choudhury. 2015. Detecting changes in suicide content manifested in social media following celebrity suicides. In Proceedings of the 26th ACM Conference on Hypertext & Social Media. ACM, 85--94.

Digital Library

[65]

Gina R Kuperberg, Philip K McGuire, Edward T Bullmore, Michael J Brammer, Sophie Rabe-Hesketh, Ian C Wright, David J Lythgoe, Steven CR Williams, and Anthony S David. 2000. Common and distinct neural substrates for pragmatic, semantic, and syntactic processing of spoken sentences: an fMRI study. Journal of Cognitive Neuroscience 12, 2 (2000), 321--341.

Digital Library

[66]

Frank Larøi, Iris E Sommer, Jan Dirk Blom, Charles Fernyhough, Dominic H Ffytche, Kenneth Hugdahl, Louise C Johns, Simon McCarthy-Jones, Antonio Preti, Andrea Raballo, et al. 2012. The characteristic features of auditory verbal hallucinations in clinical and nonclinical groups: state-of-the-art overview and future directions. Schizophrenia bulletin 38, 4 (2012), 724--733.

[67]

Josephine Lau, Benjamin Zimmerman, and Florian Schaub. 2018. Alexa, are you listening? Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1--31.

Digital Library

[68]

Belinda R Lennox, S Bert, G Park, Peter B Jones, and Peter G Morris. 1999. Spatial and temporal mapping of neural activity associated with auditory hallucinations. The Lancet 353, 9153 (1999), 644.

[69]

Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 (2017).

[70]

Tania M Lincoln, Winfried Rief, Stefan Westermann, Michael Ziegler, Marie-Luise Kesting, Eva Heibach, and Stephanie Mehl. 2014. Who stays, who benefits? Predicting dropout and change in cognitive behaviour therapy for psychosis. Psychiatry Research 216, 2 (2014), 198--205.

[71]

Hong Lu, Denise Frauendorfer, Mashfiqui Rabbi, Marianne Schmid Mast, Gokul T Chittaranjan, Andrew T Campbell, Daniel Gatica-Perez, and Tanzeem Choudhury. 2012. Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM conference on ubiquitous computing. 351--360.

Digital Library

[72]

Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765--4774. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

[73]

Masking and padding with Keras. 2021. Masking and padding with Keras. https://www.tensorflow.org/guide/keras/masking_and_padding.

[74]

John McGrath, Sukanta Saha, Joy Welham, Ossama El Saadi, Clare MacCauley, and David Chant. 2004. A systematic review of the incidence of schizophrenia: the distribution of rates and the influence of sex, urbanicity, migrant status and methodology. BMC medicine 2, 1 (2004), 1--22.

[75]

Colette M McKay, Donna M Headlam, and David L Copolov. 2000. Central auditory processing in patients with auditory hallucinations. American Journal of Psychiatry 157, 5 (2000), 759--766.

[76]

Neil M McLachlan, Dougal S Phillips, Susan L Rossell, and Sarah J Wilson. 2013. Auditory processing and hallucinations in schizophrenia. Schizophrenia research 150, 2-3 (2013), 380--385.

[77]

Emiliano Miluzzo, Nicholas D Lane, Shane B Eisenman, and Andrew T Campbell. 2007. CenceMe--injecting sensing presence into social networking applications. In European Conference on Smart Sensing and Context. Springer, 1--28.

[78]

Kyle S Minor, Beshaun J Davis, Matthew P Marggraf, Lauren Luther, and Megan L Robbins. 2018. Words matter: Implementing the electronically activated recorder in schizotypy. Personality Disorders: Theory, Research, and Treatment 9, 2 (2018), 133.

[79]

Shayan Mirjafari, Kizito Masaba, Ted Grover, Weichen Wang, Pino Audia, Andrew T Campbell, Nitesh V Chawla, Vedant Das Swain, Munmun De Choudhury, Anind K Dey, et al. 2019. Differentiating higher and lower job performers in the workplace using mobile sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 2 (2019), 1--24.

Digital Library

[80]

David C Mohr, Mi Zhang, and Stephen M Schueller. 2017. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual review of clinical psychology 13 (2017), 23--47.

[81]

Rodney Morice and Don McNicol. 1986. Language changes in schizophrenia: a limited replication. Schizophrenia Bulletin 12, 2 (1986), 239--251.

[82]

Isaac Moshe, Yannik Terhorst, Kennedy Opoku Asare, Lasse Bosse Sander, Denzil Ferreira, Harald Baumeister, David C Mohr, and Laura Pulkki-Råback. 2021. Predicting Symptoms of Depression and Anxiety Using Smartphone and Wearable Data. Frontiers in psychiatry 12 (2021).

[83]

Amir Muaremi, Franz Gravenhorst, Agnes Grünerbl, Bert Arnrich, and Gerhard Tröster. 2014. Assessing bipolar episodes using speech cues derived from phone calls. In Pervasive Computing Paradigms for Mental Health: 4th International Symposium, MindCare 2014, Tokyo, Japan, May 8-9, 2014, Revised Selected Papers 4. Springer, 103--114.

[84]

Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin 29, 5 (2003), 665--675.

[85]

Stefanie Nickels, Matthew D Edwards, Sarah F Poole, Dale Winter, Jessica Gronsbell, Bella Rozenkrants, David P Miller, Mathias Fleck, Alan McLean, Bret Peterson, et al. 2021. Toward a Mobile Platform for Real-world Digital Measurement of Depression: User-Centered Design, Data Quality, and Behavioral and Clinical Modeling. JMIR mental health 8, 8 (2021), e27589.

[86]

Jukka-Pekka Onnela, Caleb Dixon, Keary Griffin, Tucker Jaenicke, Leila Minowada, Sean Esterkin, Alvin Siu, Josh Zagorsky, and Eli Jones. 2021. Beiwe: A data collection platform for high-throughput digital phenotyping. Journal of Open Source Software 6, 68 (2021), 3417.

[87]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).

[88]

Paola Pedrelli, Szymon Fedor, Asma Ghandeharioun, Esther Howe, Dawn F Ionescu, Darian Bhathena, Lauren B Fisher, Cristina Cusin, Maren Nyer, Albert Yeung, et al. 2020. Monitoring changes in depression severity using wearable and mobile sensors. Frontiers in psychiatry 11 (2020), 1413.

[89]

James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. UT Faculty/Researcher Works (2015).

[90]

James W Pennebaker, Cindy K Chung, Joey Frazee, Gary M Lavergne, and David I Beaver. 2014. When small words foretell academic success: The case of college admissions essays. PloS one 9, 12 (2014), e115844.

[91]

James W Pennebaker, Matthias R Mehl, and Kate G Niederhoffer. 2003. Psychological aspects of natural language use: Our words, our selves. Annual review of psychology 54, 1 (2003), 547--577.

[92]

Viliam Rapcan, Shona D'Arcy, Sherlyn Yeap, Natasha Afzal, Jogin Thakore, and Richard B Reilly. 2010. Acoustic and temporal analysis of speech: A potential biomarker for schizophrenia. Medical engineering & physics 32, 9 (2010), 1074--1079.

[93]

Benjamin Rolland, Ali Amad, Emmanuel Poulet, Régis Bordet, Alexandre Vignaud, Rémy Bation, Christine Delmaire, Pierre Thomas, Olivier Cottencin, and Renaud Jardri. 2015. Resting-state functional connectivity of the nucleus accumbens in auditory and visual hallucinations in schizophrenia. Schizophrenia bulletin 41, 1 (2015), 291--299.

[94]

Matthia Sabatelli, Venet Osmani, Oscar Mayora, Agnes Gruenerbl, and Paul Lukowicz. 2014. Correlation of significant places with self-reported state of bipolar disorder patients. In 2014 4th International Conference on Wireless Mobile Communication and Healthcare-Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH). IEEE, 116--119.

[95]

Norihiro Sadato, Yoshiharu Yonekura, Hiroki Yamada, Satoshi Nakamura, Atsuo Waki, and Yasushi Ishii. 1998. Activation patterns of covert word generation detected by fMRI: comparison with 3D PET. Journal of computer assisted tomography 22, 6 (1998), 945--952.

[96]

Sohrab Saeb, Emily G Lattie, Konrad P Kording, and David C Mohr. 2017. Mobile phone detection of semantic location and its relationship to depression and anxiety. JMIR mHealth and uHealth 5, 8 (2017), e7297.

[97]

Sohrab Saeb, Mi Zhang, Christopher J Karr, Stephen M Schueller, Marya E Corden, Konrad P Kording, and David C Mohr. 2015. Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study. Journal of medical Internet research 17, 7 (2015).

[98]

Koustuv Saha, Ted Grover, Stephen M Mattingly, Vedant Das Swain, Pranshu Gupta, Gonzalo J Martinez, Pablo Robles-Granda, Gloria Mark, Aaron Striegel, and Munmun De Choudhury. 2021. Person-Centered Predictions of Psychological Constructs with Social Media Contextualized by Multimodal Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--32.

Digital Library

[99]

Shekhar Saxena, Graham Thornicroft, Martin Knapp, and Harvey Whiteford. 2007. Resources for mental health: scarcity, inequity, and inefficiency. The lancet 370, 9590 (2007), 878--889.

[100]

Terrence J Sejnowski. 2018. The deep learning revolution. MIT press.

[101]

Bryony Sheaves, Paul E Bebbington, Guy M Goodwin, Paul J Harrison, Colin A Espie, Russell G Foster, and Daniel Freeman. 2016. Insomnia and hallucinations in the general population: findings from the 2000 and 2007 British Psychiatric Morbidity Surveys. Psychiatry Research 241 (2016), 141--146.

[102]

Jessica Helen Silver, Marcus Lewton, and Heledd Wyn Lewis. 2023. Mediators of negative content and voice-related distress in a diverse sample of clinical and non-clinical voice-hearers. British Journal of Clinical Psychology 62, 1 (2023), 96--111.

[103]

Robert R Sinclair and Janelle H Cheung. 2016. Money matters: Recommendations for financial stress research in occupational health psychology. Stress and Health 32, 3 (2016), 181--193.

[104]

Runar Elle Smelror, Josef Johann Bless, Kenneth Hugdahl, and Ingrid Agartz. 2019. Feasibility and Acceptability of Using a Mobile Phone App for Characterizing Auditory Verbal Hallucinations in Adolescents With Early-Onset Psychosis: Exploratory Study. JMIR Formative Research 3, 2 (May 2019), e13882. https://doi.org/10.2196/13882

[105]

Iris EC Sommer, Kirstin Daalman, Thomas Rietkerk, Kelly M Diederen, Steven Bakker, Jaap Wijkstra, and Marco PM Boks. 2010. Healthy individuals with auditory verbal hallucinations; who are they? Psychiatric assessments of a selected sample of 103 subjects. Schizophrenia bulletin 36, 3 (2010), 633--641.

[106]

Iris EC Sommer, Kelly MJ Diederen, Jan-Dirk Blom, Anne Willems, Leila Kushan, Karin Slotema, Marco PM Boks, Kirstin Daalman, Hans W Hoek, Sebastiaan FW Neggers, et al. 2008. Auditory verbal hallucinations predominantly activate the right inferior frontal area. Brain 131, 12 (2008), 3169--3177.

[107]

M Stephane, S Barton, and NN Boutros. 2001. Auditory verbal hallucinations and dysfunction of the neural substrates of speech. Schizophrenia research 50, 1-2 (2001), 61--78.

[108]

Rael D Strous, Nelson Cowan, Walter Ritter, and Daniel C Javitt. 1995. Auditory sensory (" echoic") memory dysfunction in schizophrenia. The American journal of psychiatry (1995).

[109]

Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29, 1 (2010), 24--54.

[110]

A. Y. Tien. 1991. Distribution of hallucinations in the population. Social Psychiatry and Psychiatric Epidemiology 26, 6 (1991), 287--292. https://doi.org/10.1007/bf00789221

[111]

Vincent W-S Tseng, Akane Sano, Dror Ben-Zeev, Rachel Brian, Andrew T Campbell, Marta Hauser, John M Kane, Emily A Scherer, Rui Wang, Weichen Wang, et al. 2020. Using behavioral rhythms and multi-task learning to predict fine-grained symptoms of schizophrenia. Scientific reports 10, 1 (2020), 1--17.

[112]

Rachel Tucker, John Farhall, Neil Thomas, Christopher Groot, and Susan L Rossell. 2013. An examination of auditory processing and affective prosody in relatives of patients with auditory hallucinations. Frontiers in Human Neuroscience 7 (2013), 531.

[113]

Ryan J Van Lieshout and Joel O Goldberg. 2007. Quantifying self-reports of auditory verbal hallucinations in persons with psychosis. Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement 39, 1 (2007), 73.

[114]

Fabian Wahle, Tobias Kowatsch, Elgar Fleisch, Michael Rufer, Steffi Weidt, et al. 2016. Mobile sensing and support for people with depression: a pilot trial in the wild. JMIR mHealth and uHealth 4, 3 (2016), e5960.

[115]

Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell. 2014. StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. 3--14.

Digital Library

[116]

Rui Wang, Min S. H. Aung, Saeed Abdullah, Rachel Brian, Andrew T. Campbell, Tanzeem Choudhury, Martan Hauser, John Kane, Michael Merrill, Emily A. Scherer, and Vincent W. S. Tseng. 2016. CrossCheck: Toward passive sensing and detection of mental health changes in people with schizophrenia. (2016).

[117]

Rui Wang, Weichen Wang, Min SH Aung, Dror Ben-Zeev, Rachel Brian, Andrew T Campbell, Tanzeem Choudhury, Marta Hauser, John Kane, Emily A Scherer, et al. 2017. Predicting symptom trajectories of schizophrenia using mobile sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--24.

Digital Library

[118]

Rui Wang, Weichen Wang, Alex DaSilva, Jeremy F Huckins, William M Kelley, Todd F Heatherton, and Andrew T Campbell. 2018. Tracking depression dynamics in college students using mobile phone and wearable sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--26.

Digital Library

[119]

Weichen Wang, Gabriella M Harari, Rui Wang, Sandrine R Müller, Shayan Mirjafari, Kizito Masaba, and Andrew T Campbell. 2018. Sensing behavioral change over time: Using within-person variability features from mobile sensing to predict personality traits. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--21.

Digital Library

[120]

Weichen Wang, Shayan Mirjafari, Gabriella Harari, Dror Ben-Zeev, Rachel Brian, Tanzeem Choudhury, Marta Hauser, John Kane, Kizito Masaba, Subigya Nepal, et al. 2020. Social sensing: assessing social functioning of patients living with schizophrenia using mobile phone sensing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--15.

Digital Library

[121]

Weichen Wang, Subigya Nepal, Jeremy F. Huckins, Lessley Hernandez, Vlado Vojdanovski, Dante Mack, Jane Plomp, Arvind Pillai, Mikio Obuchi, Alex daSilva, Eilis Murphy, Elin Hedlund, Courtney Rogers, Meghan Meyer, and Andrew Campbell. 2022. First-Gen Lens: Assessing Mental Health of First-Generation Students across Their First Year at College Using Mobile Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 2, Article 95 (jul 2022), 32 pages. https://doi.org/10.1145/3543194

Digital Library

[122]

Flavie Waters, Daniel Collerton, Dominic H Ffytche, Renaud Jardri, Delphine Pins, Robert Dudley, Jan Dirk Blom, Urs Peter Mosimann, Frank Eperjesi, Stephen Ford, et al. 2014. Visual hallucinations in the psychosis spectrum and comparative information from neurodegenerative disorders and eye disease. Schizophrenia bulletin 40, Suppl_4 (2014), S233--S245.

[123]

Danny Wyatt, Tanzeem Choudhury, and Jeff A Bilmes. 2007. Conversation detection and speaker segmentation in privacy-sensitive situated speech data. In INTERSPEECH. 586--589.

[124]

Danny Wyatt, Tanzeem Choudhury, Jeff A Bilmes, and Henry A Kautz. 2007. A Privacy-Sensitive Approach to Modeling Multi-Person Conversations. In IJCAI, Vol. 7. 1769--1775.

Digital Library

[125]

Danny Wyatt, Tanzeem Choudhury, and Henry Kautz. 2007. Capturing spontaneous conversation and social dynamics: A privacy-sensitive data collection effort. In Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on, Vol. 4. IEEE, IV--213.

[126]

Weizhe Xu, Jake Portanova, Ayesha Chander, Dror Ben-Zeev, and Trevor Cohen. 2020. The Centroid Cannot Hold: Comparing Sequential and Global Estimates of Coherence as Indicators of Formal Thought Disorder. In AMIA Annual Symposium Proceedings, Vol. 2020. American Medical Informatics Association, 1315.

[127]

Weizhe Xu, Weichen Wang, Jake Portanova, Ayesha Chander, Andrew Campbell, Serguei Pakhomov, Dror Ben-Zeev, and Trevor Cohen. 2022. Fully Automated Detection of Formal Thought Disorder with Time-series Augmented Representations for Detection of Incoherent Speech (TARDIS). Journal of Biomedical Informatics (2022), 103998.

Cited By

Sahu NGupta SLone H(2024)Wearable Technology Insights: Unveiling Physiological Responses During Three Different Socially Anxious ActivitiesACM Journal on Computing and Sustainable Societies10.1145/36636712:2(1-23)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3663671
Wang JFeng YKumbhar GWang GYan QJin QFerrier RXiong JLi TOkoshi TKo JLiKamWa R(2024)SoilCares: Towards Low-cost Soil Macronutrients and Moisture Monitoring Using RF-VNIR SensingProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661868(196-209)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661868
Liu CDong ZHuang LYan WWang XFang DChen X(2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643512
Show More Cited By

Index Terms

The Power of Speech in the Wild: Discriminative Power of Daily Voice Diaries in Understanding Auditory Verbal Hallucinations Using Deep Learning
1. Applied computing
  1. Life and medical sciences
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Speech communication in the wild
EACL '12: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

Much of what we know about speech perception comes from laboratory studies with clean, canonical speech, ideal listeners and artificial tasks. But how do interlocutors manage to communicate effectively in the seemingly less-than-ideal conditions of ...
Kernel power flow orientation coefficients for noise-robust speech recognition

Noise-robustness has become a crucial parameter in Automatic Speech Recognition (ASR) systems today with their increased use in noise-filled real-world environments. One way to address this issue is to develop features that are innately noise-robust. ...
Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Dysarthria is a motor speech disorder that causes inability to control and coordinate one or more articulators. This makes it difficult for a dysarthric speaker to utter certain speech sound units, thereby producing poorly articulated, slurred, and ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 7, Issue 3

September 2023

1734 pages

EISSN:2474-9567

DOI:10.1145/3626192

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2023

Published in IMWUT Volume 7, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Institute of Mental Health (NIMH)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
234
Total Downloads

Downloads (Last 12 months)234
Downloads (Last 6 weeks)19

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sahu NGupta SLone H(2024)Wearable Technology Insights: Unveiling Physiological Responses During Three Different Socially Anxious ActivitiesACM Journal on Computing and Sustainable Societies10.1145/36636712:2(1-23)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3663671
Wang JFeng YKumbhar GWang GYan QJin QFerrier RXiong JLi TOkoshi TKo JLiKamWa R(2024)SoilCares: Towards Low-cost Soil Macronutrients and Moisture Monitoring Using RF-VNIR SensingProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661868(196-209)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661868
Liu CDong ZHuang LYan WWang XFang DChen X(2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643512
Chang ZZhang FXiong JChen WZhang DGanesan DLane NShi W(2024)MSense: Boosting Wireless Sensing Capability Under Motion InterferenceProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649350(108-123)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3636534.3649350
Zhang XZhang DXie YWu DLi YZhang D(2024)WaffleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314587:4(1-29)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631458
Zhang DZhang XXie YZhang FWang XLi YZhang D(2024)LoCalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314367:4(1-27)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631436
Yang YRen LChen CHu BZhang ZLi XShen YZhu KJi JZhang YNi YWu JWang QWu JSun LTao YWang G(2024)SnapInflatables: Designing Inflatables with Snap-through Instability for Responsive InteractionProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642933(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642933
Mäder AMeegahapola LGatica-Perez D(2024)Learning About Social Context From Smartphone Data: Generalization Across Countries and Daily Life MomentsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642444(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642444
Hohman FKery MRen DMoritz D(2024)Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning ExperiencesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642109(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642109
Arun ASaruwatari SShah SBharadia DEskicioglu RHuang PPatwari N(2023)XRLoc: Accurate UWB Localization to Realize XR DeploymentsProceedings of the 21st ACM Conference on Embedded Networked Sensor Systems10.1145/3625687.3625810(459-473)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3625687.3625810
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents