BDI-Sen: A Sentence Dataset for Clinical Symptoms of Depression
Pages 2996 - 3006
Abstract
People tend to consider social platforms as convenient media for expressing their concerns and emotional struggles. With their widespread use, researchers could access and analyze user-generated content related to mental states. Computational models that exploit that data show promising results in detecting at-risk users based on engineered features or deep learning models. However, recent works revealed that these approaches have a limited capacity for generalization and interpretation when considering clinical settings. Grounding the models' decisions on clinical and recognized symptoms can help to overcome these limitations. In this paper, we introduce BDI-Sen, a symptom-annotated sentence dataset for depressive disorder. BDI-Sen covers all the symptoms present in the Beck Depression Inventory-II (BDI-II), a reliable questionnaire used for detecting and measuring depression. The annotations in the collection reflect whether a statement about the specific symptom is informative (i.e., exposes traces about the individual's state regarding that symptom). We thoroughly analyze this resource and explore linguistic style, emotional attribution, and other psycholinguistic markers. Additionally, we conduct a series of experiments investigating the utility of BDI-Sen for various tasks, including the detection and severity classification of symptoms. We also examine their generalization when considering symptoms from other mental diseases. BDI-Sen may aid the development of future models that consider trustworthy and valuable depression markers.
Supplemental Material
MP4 File
This video presents BDI-Sen, a symptom-annotated dataset for depression that includes manually labelled sentences addressing the 21 BDI-II symptoms. By leveraging the eRisk2019 collections as data source, our dataset provides binary relevance labels for the BDI-II symptoms and weak labels regarding their severity level. We designed a retrieval phase to filter-out candidate sentences based on the descriptions of the BDI-II elements, and three experts decided the actual relevance of the candidates. We explored this resource, revealing linguistic and emotional differences among the symptoms. Moreover, we performed two main experiments with state-of-the-art models trained solely on BDI-Sen: symptom detection and symptom severity classification, including an extensive error analysis for both tasks. The good generalization ability of our models further underlines the usefulness of BDI-Sen as a resource for developing robust mental health detection models.
- Download
- 71.84 MB
References
[1]
Nazanin Andalibi, Pinar Ozturk, and Andrea Forte. 2017. Sensitive Self-Disclosures, Responses, and Social Support on Instagram: The Case of #Depression. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 1485--1500. https://doi.org/10.1145/2998181.2998243
[2]
Aaron T Beck, Robert A Steer, Roberta Ball, and William F Ranieri. 1996. Comparison of Beck Depression Inventories-IA and-II in psychiatric outpatients. Journal of personality assessment, Vol. 67, 3 (1996), 588--597.
[3]
Fidel Cacheda, Diego Fernandez, Francisco J Novoa, Victor Carneiro, et al. 2019. Early detection of depression: social network analysis and random forest techniques. Journal of medical Internet research, Vol. 21, 6 (2019), e12554.
[4]
Amy Callahan and Kay Inckle. 2012. Cybertherapy or psychobabble? A mixed methods study of online emotional support. British Journal of Guidance & Counselling, Vol. 40, 3 (2012), 261--278.
[5]
Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, and Nazli Goharian. 2018. SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 1485--1497.
[6]
Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying Mental Health Signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Baltimore, Maryland, USA, 51--60. https://doi.org/10.3115/v1/W14-3207
[7]
Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine. 2018. Natural language processing of social media as screening for suicide risk. Biomedical informatics insights, Vol. 10 (2018), 1178222618792860.
[8]
Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a. Social Media as a Measurement Tool of Depression in Populations (WebSci '13). Association for Computing Machinery, New York, NY, USA, 47--56. https://doi.org/10.1145/2464464.2464480
[9]
Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013b. Predicting depression via social media. In Proceedings of the international AAAI conference on web and social media, Vol. 7. 128--137.
[10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[11]
Adam G Dunn, Kenneth D Mandl, and Enrico Coiera. 2018. Social media interventions for precision public health: promises and risks. NPJ digital medicine, Vol. 1, 1 (2018), 47.
[12]
Sindhu Kiranmai Ernala, Michael L. Birnbaum, Kristin A. Candan, Asra F. Rizvi, William A. Sterling, John M. Kane, and Munmun De Choudhury. 2019. Methodological Gaps in Predicting Mental Health States from Social Media: Triangulating Diagnostic Signals. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--16. https://doi.org/10.1145/3290605.3300364
[13]
Sam Fletcher, Md Zahidul Islam, et al. 2018. Comparing sets of patterns with the Jaccard index. Australasian Journal of Information Systems, Vol. 22 (2018).
[14]
Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017. Detecting depression and mental illness on social media: an integrative review. Current Opinion in Behavioral Sciences, Vol. 18 (2017), 43--49.
[15]
Aron Halfin. 2007. Depression: the benefits of early and appropriate treatment. American Journal of Managed Care, Vol. 13, 4 (2007), S92.
[16]
Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2020. Do Models of Mental Health Based on Social Media Data Generalize?. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3774--3788. https://doi.org/10.18653/v1/2020.findings-emnlp.337
[17]
Rebecca A Hayes, Caleb T Carr, and Donghee Yvette Wohn. 2016. One click, many meanings: Interpreting paralinguistic digital affordances in social media. Journal of Broadcasting & Electronic Media, Vol. 60, 1 (2016), 171--187.
[18]
Hsin-Yi Huang. 2016. Examining the beneficial effects of individual's self-disclosure on the social network site. Computers in human behavior, Vol. 57 (2016), 122--132.
[19]
Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 7184--7190. https://aclanthology.org/2022.lrec-1.778
[20]
Payam Karisani and Eugene Agichtein. 2018. Did you really just have a heart attack? Towards robust detection of personal health mentions in social media. In Proceedings of the 2018 World Wide Web Conference. 137--146.
[21]
Sylvia Deidre Kauer, Cheryl Mangan, and Lena Sanci. 2014. Do online mental health services improve help-seeking for young people? A systematic review. Journal of medical Internet research, Vol. 16, 3 (2014), e3103.
[22]
Kurt Kroenke, Robert L Spitzer, and Janet BW Williams. 2001. The PHQ-9: validity of a brief depression severity measure. Journal of general internal medicine, Vol. 16, 9 (2001), 606--613.
[23]
Lourdes Lasa, Jose L Ayuso-Mateos, Jose L Vázquez-Barquero, FJ Diez-Manrique, and Christopher F Dowrick. 2000. The use of the Beck Depression Inventory to screen for depression in the general population: a preliminary analysis. Journal of affective disorders, Vol. 57, 1--3 (2000), 261--265.
[24]
Chung-Ying Lin, Peyman Namdar, Mark D Griffiths, and Amir H Pakpour. 2021. Mediated roles of generalized trust and perceived social support in the effects of problematic social media use on mental health: A cross-sectional study. Health Expectations, Vol. 24, 1 (2021), 165--173.
[25]
Huijie Lin, Jia Jia, Quan Guo, Yuanyuan Xue, Qi Li, Jie Huang, Lianhong Cai, and Ling Feng. 2014. User-level psychological stress detection from social media using deep neural network. In Proceedings of the 22nd ACM international conference on Multimedia. 507--516.
[26]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. https://doi.org/10.48550/ARXIV.1907.11692
[27]
David E Losada and Fabio Crestani. 2016. A test collection for research on depression and language use. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Évora, Portugal, September 5-8, 2016, Proceedings 7. Springer, 28--39.
[28]
David E Losada, Fabio Crestani, and Javier Parapar. 2019. Overview of erisk 2019 early risk prediction on the internet. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 340--357.
[29]
David E Losada, Fabio Crestani, and Javier Parapar. 2020. eRisk 2020: Self-harm and depression challenges. In European Conference on Information Retrieval. Springer, 557--563.
[30]
Kate Loveys, Jonathan Torrez, Alex Fine, Glen Moriarty, and Glen Coppersmith. 2018. Cross-cultural differences in language markers of depression online. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. Association for Computational Linguistics, New Orleans, LA, 78--87. https://doi.org/10.18653/v1/W18-0608
[31]
Mufan Luo and Jeffrey T Hancock. 2020. Self-disclosure and social media: motivations, mechanisms and psychological well-being. Current opinion in psychology, Vol. 31 (2020), 110--115.
[32]
Sean MacAvaney, Bart Desmet, Arman Cohan, Luca Soldaini, Andrew Yates, Ayah Zirikly, and Nazli Goharian. 2018. RSDD-Time: Temporal Annotation of Self-Reported Mental Health Diagnoses. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. Association for Computational Linguistics, New Orleans, LA, 168--173. https://doi.org/10.18653/v1/W18-0618
[33]
Anna Metzler and Herbert Scheithauer. 2017. The long-term benefits of positive self-presentation via profile pictures, number of friends and the initiation of relationships on Facebook for adolescents' self-esteem and the initiation of offline relationships. Frontiers in psychology, Vol. 8 (2017), 1981.
[34]
Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word--emotion association lexicon. Computational intelligence, Vol. 29, 3 (2013), 436--465.
[35]
C Brendan Montano. 1994. Recognition and treatment of depression in a primary care setting. The Journal of clinical psychiatry (1994).
[36]
Danielle Mowery, Hilary Smith, Tyler Cheney, Greg Stoddard, Glen Coppersmith, Craig Bryan, Mike Conway, et al. 2017. Understanding depressive symptoms and psychosocial stressors on Twitter: a corpus-based study. Journal of medical Internet research, Vol. 19, 2 (2017), e6895.
[37]
Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022. Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 8446--8459. https://doi.org/10.18653/v1/2022.acl-long.578
[38]
Cardwell C Nuckols and Cardwell C Nuckols. 2013. The diagnostic and statistical manual of mental disorders,(DSM-5). Philadelphia: American Psychiatric Association (2013).
[39]
Ahmed Husseini Orabi, Prasadith Buddhitha, Mahmoud Husseini Orabi, and Diana Inkpen. 2018. Deep learning for depression detection of twitter users. In Proceedings of the fifth workshop on computational linguistics and clinical psychology: from keyboard to clinic. 88--97.
[40]
Rosa María Ortega-Mendoza, Delia Irazú Hernández-Farías, Manuel Montes y Gómez, and Luis Villaseñor-Pineda. 2022. Revealing traces of depression through personal statements analysis in social media. Artificial Intelligence in Medicine, Vol. 123 (2022), 102202. https://doi.org/10.1016/j.artmed.2021.102202
[41]
Javier Parapar, Patricia Martín-Rodilla, David E Losada, and Fabio Crestani. 2021. Overview of eRisk 2021: Early risk prediction on the internet. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 324--344.
[42]
Javier Parapar, Patricia Martín-Rodilla, David E Losada, and Fabio Crestani. 2023. eRisk 2023: Depression, Pathological Gambling, and Eating Disorder Challenges. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2--6, 2023, Proceedings, Part III. Springer, 585--592.
[43]
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, Vol. 71, 2001 (2001), 2001.
[44]
A Picardi, I Lega, L Tarsitani, M Caredda, G Matteucci, MP Zerella, R Miglio, A Gigantesco, M Cerbo, A Gaddini, et al. 2016. A randomised controlled trial of the effectiveness of a program for early detection and treatment of depression in primary care. Journal of affective disorders, Vol. 198 (2016), 96--101.
[45]
Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion. Elsevier, 3--33.
[46]
Anxo Pérez, Javier Parapar, and Álvaro Barreiro. 2022. Automatic depression score estimation with word embedding models. Artificial Intelligence in Medicine, Vol. 132 (2022), 102380. https://doi.org/10.1016/j.artmed.2022.102380
[47]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, Vol. 21, 1 (2020), 5485--5551.
[48]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
[49]
Esteban André s Ríssola, Mohammad Aliannejadi, and Fabio Crestani. 2020. Beyond Modelling: Understanding Mental Disorders in Online Social Media. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12035), Joemon M. Jose, Emine Yilmaz, Jo a o Magalh a es, Pablo Castells, Nicola Ferro, Mário J. Silva, and Flá vio Martins (Eds.). Springer, 296--310. https://doi.org/10.1007/978-3-030-45439-5_20
[50]
Esteban A. Ríssola, Mohammad Aliannejadi, and Fabio Crestani. 2022. Mental disorders on online social media through the lens of language and behaviour: Analysis and visualisation. Information Processing & Management, Vol. 59, 3 (2022), 102890. https://doi.org/10.1016/j.ipm.2022.102890
[51]
Mihye Seo, Jinhee Kim, and Hyeseung Yang. 2016. Frequent interaction and fast feedback predict perceived social support: Using crawled and self-reported data of Facebook users. Journal of Computer-Mediated Communication, Vol. 21, 4 (2016), 282--297.
[52]
Guangyao Shen, Jia Jia, Liqiang Nie, Fuli Feng, Cunjun Zhang, Tianrui Hu, Tat-Seng Chua, Wenwu Zhu, et al. 2017. Depression detection via harvesting social media: A multimodal dictionary learning solution. In IJCAI. 3838--3844.
[53]
Hong-Han Shuai, Chih-Ya Shen, De-Nian Yang, Yi-Feng Lan, Wang-Chien Lee, Philip S Yu, and Ming-Syan Chen. 2016. Mining online social data for detecting social network mental disorders. In Proceedings of the 25th International Conference on World Wide Web. 275--285.
[54]
Ruba Skaik and Diana Inkpen. 2020. Using social media for mental health surveillance: a review. ACM Computing Surveys (CSUR), Vol. 53, 6 (2020), 1--31.
[55]
Marcel Trotzek, Sven Koitka, and Christoph Friedrich. 2018. Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences. IEEE Transactions on Knowledge and Data Engineering, Vol. 32 (04 2018), 588--601. https://doi.org/10.1109/TKDE.2018.2885515
[56]
Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med, Vol. 37, 5 (2005), 360--363.
[57]
Colin G Walsh, Beenish Chaudhry, Prerna Dua, Kenneth W Goodman, Bonnie Kaplan, Ramakanth Kavuluru, Anthony Solomonides, and Vignesh Subbian. 2020. Stigma, biomarkers, and algorithmic bias: recommendations for precision behavioral health with artificial intelligence. JAMIA open, Vol. 3, 1 (2020), 9--15.
[58]
Chia-chen Yang, Sean M Holden, and Mollie DK Carter. 2017. Emerging adults' social media self-presentation and identity development at college transition: Mindfulness as a moderator. Journal of Applied Developmental Psychology, Vol. 52 (2017), 212--221.
[59]
Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and Self-Harm Risk Assessment in Online Forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 2968--2978. https://doi.org/10.18653/v1/D17--1322
[60]
Zhiling Zhang, Siyuan Chen, Mengyue Wu, and Kenny Q. Zhu. 2022a. Psychiatric Scale Guided Risky Post Screening for Early Detection of Depression. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, Luc De Raedt (Ed.). ijcai.org, 5220--5226. https://doi.org/10.24963/ijcai.2022/725
[61]
Zhiling Zhang, Siyuan Chen, Mengyue Wu, and Kenny Q. Zhu. 2022b. Symptom Identification for Interpretable Detection of Multiple Mental Disorders on Social Media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 9970--9985. https://aclanthology.org/2022.emnlp-main.677
[62]
Ayah Zirikly, Dana Atzil-Slonim, Maria Liakata, Steven Bedrick, Bart Desmet, Molly Ireland, Andrew Lee, Sean MacAvaney, Matthew Purver, Rebecca Resnik, and Andrew Yates (Eds.). 2022. Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology. Association for Computational Linguistics, Seattle, USA. https://aclanthology.org/2022.clpsych-1.0
[63]
Hamad Zogan, Imran Razzak, Shoaib Jameel, and Guandong Xu. 2021. DepressionNet: Learning Multi-Modalities with User Post Summarization for Depression Detection on Social Media. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 133--142. https://doi.org/10.1145/3404835.3462938
Index Terms
- BDI-Sen: A Sentence Dataset for Clinical Symptoms of Depression
Recommendations
SenseMood: Depression Detection on Social Media
ICMR '20: Proceedings of the 2020 International Conference on Multimedia RetrievalMore than 300 million people have been affected by depression all over the world. Due to the medical equipment and knowledge limitations, most of them are not diagnosed at the early stages. Recent work attempts to use social media to detect depression ...
Depression detection on online social network with multivariate time series feature of user depressive symptoms
AbstractIn recent years, depression has attracted worldwide attention because of its prevalence and great risk for suicide. Existing studies have confirmed the feasibility of depression detection on online social networks. Most existing ...
Highlights- Build a dataset with complete and time-consecutive user tweeting history.
- ...
Harnessing emotions for depression detection
AbstractHuman emotions using textual cues, speech patterns, and facial expressions can give insight into their mental state. Although there are several uni-modal datasets for emotion recognition, there are very few labeled datasets for multi-modal ...
Comments
Information & Contributors
Information
Published In
July 2023
3567 pages
ISBN:9781450394086
DOI:10.1145/3539618
- General Chairs:
- Hsin-Hsi Chen,
- Wei-Jou (Edward) Duh,
- Hen-Hsen Huang,
- Program Chairs:
- Makoto P. Kato,
- Josiane Mothe,
- Barbara Poblete
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 July 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Xunta de Galicia
- Ministerio de Ciencia e Innovación
Conference
SIGIR '23
Sponsor:
SIGIR '23: The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 23 - 27, 2023
Taipei, Taiwan
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 411Total Downloads
- Downloads (Last 12 months)362
- Downloads (Last 6 weeks)18
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in