Intent aware data augmentation by leveraging generative AI for stress detection in social media texts

Background and motivation

The well-being of individuals and their ability to function effectively in life depends on their mental health, which encompasses emotional, psychological, and social aspects (American Psychological Association, 2019). Stress, a natural reaction to challenges, becomes problematic when it exceeds a certain level, affecting our physical, emotional, and behavioral responses (American Psychological Association, 2019). According to the Stress in America survey (2022), around 27% of adults experience debilitating stress (American Psychological Association, 2022), while 74% encounter overwhelming stress situations (Mental Health Foundation, 2018). Daily stress overload is a significant burden for young adults aged 18 to 34 (American Psychological Association, 2022). In their 2018 report, the Mental Health Foundation emphasized the link between stress, depression (51%), and heightened anxiety (61%), while also highlighting alarming rates of self-harm (16%) and suicidal thoughts (32%) among individuals dealing with stress (Mental Health Foundation, 2018). Generation Z’s elevated stress levels signal a concerning trend (American Psychological Association, 2020).

The urgency to comprehend and deal with stress is emphasized by its profound effects on individual and societal well-being. Understanding and measuring stress involves a range of factors, like self-report surveys and observational studies (Garg, 2023; Ramirez-Esparza et al., 2021; Rude, Gortner & Pennebaker, 2004). In today’s digital age, platforms such as Reddit are crucial, providing spaces for self-expression and the sharing of daily experiences (Statista, 2022). Social media enables adolescents to form communities, maintain friendships, and access social support (Mental Health Foundation, 2018; U.S. Surgeon General Advisory, 2023).

By employing NLP techniques, social media platforms can detect stress patterns in user-generated posts (Statista, 2022; U.S. Surgeon General Advisory, 2023).

Text classification for stress analysis

Numerous studies have investigated stress classification through NLP and machine learning, employing various models and approaches. Researchers have utilized classical supervised techniques (e.g., support vector machines, logistic regression, naive Bayes, multi-layer perceptrons, decision trees) as well as deep models like convolutional neural networks (CNNs) and gated recurrent units (GRUs) (Turcan & McKeown, 2019; Inamdar et al., 2023; Febriansyah et al., 2023; Selvadass, Bruntha & Priyadharsini, 2022).

Another set of approaches involves fine-tuning pre-trained, large language models, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), RoBERTA (Liu et al., 2019), and their variants like Mental BERT (Ji et al., 2022), Mental RoBERTA (Ji et al., 2022) and PHS-BERT (Naseem et al., 2022). Additionally, integration of linguistic features has shown promise for stress detection, with notable results (Ilias, Mouzakitis & Askounis, 2023; Wang et al., 2023). For the DREADDIT dataset, Kumar, Trueman & Cambria (2022) achieved an F1 score of 83.90 using RoBERTa along with linguistic inquiry and word count (LIWC) features. Yang, Zhang & Ananiadou (2022) introduced KC-NET, incorporating context-aware post-encoding, knowledge-aware dot product attention, and supervised contrastive learning, achieving an impressive F1 score of 83.50 on the same dataset.

Comparative evaluation of Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4 in zero-shot prompting, few-shot prompting, and instruction fine-tuning revealed that GPT provides the highest accuracy in zero-shot prompting, while FLAN-T5 excels in few-shot prompting (Xu et al., 2023). Yang et al. (2023) comprehensively compared language model (LLM) performances, highlighting the effectiveness of ChatGPT with expert-written few-shot examples. The best zero-shot classification performance using ChatGPT is an F1 score of 73% (Lamichhane, 2023). Some of the prompts used in literature for zero-shot classification using ChatGPT can be seen in Table 3. The varying performance highlights the dependence of zero-shot classification on query prompts.

In summary, the literature reveals a diverse array of approaches ranging from classical supervised techniques to advanced pre-trained language models, each demonstrating varying degrees of success in stress classification tasks. Despite the superior performance of domain-specific models, RoBERTa turns out to be the best-performing general text classification model (Kumar, Trueman & Cambria, 2022).

Data augmentation

Widely used token-level augmentation techniques involve preserving syntax and semantic meaning through well-designed local modifications. The most popular technique is EDA (Wei & Zou, 2019) which includes random operations such as deletion, insertion, replacement, and swapping. An AEDA only performs random insertion of punctuation marks and promises better performance (Karimi, Rossi & Prati, 2021). Language models (LMs), including Conditional BERT (Wu et al., 2019) and paraphrasing techniques like round-trip or back translation (Edunov et al., 2018; Xie et al., 2020; Corbeil & Ghadivel, 2020), play a crucial role in data augmentation for NLP tasks. Some research explores end-to-end models for paraphrasing with additional syntactic information and latent variables (Kumar et al., 2019; Prakash et al., 2016; Iyyer et al., 2018; Gupta et al., 2018).

Unconditional data augmentation models like generative adversarial networks (GANs) (Goodfellow et al., 2014) and variational autoencoders (VAEs) (Kingma & Welling, 2013), along with graph-based methods (Chen, Ji & Evans, 2020), are also explored in the literature. Conditional generation methods, utilizing pre-trained language models like GPT-2 (Anaby-Tavor et al., 2020) and T5 (Lee et al., 2021), generate extra text conditioned on labels for data augmentation.

Recent efforts leverage Generative AI, such as ChatGPT, proposing novel techniques (Yoo et al., 2021; Dai et al., 2023). Yoo et al. (2021) suggests a technique using large-scale language models to generate realistic text samples, incorporating soft-labels for knowledge distillation. AugGPT (Dai et al., 2023) rephrases each sentence in training samples into multiple conceptually similar but semantically different samples for downstream model training.

Challenges and opportunities

The Dreaddit dataset leverages Reddit posts for stress classification, but its limited size challenges language model performance, therefore, data augmentation is crucial. Unconditional data augmentation techniques (Goodfellow et al., 2014; Kingma & Welling, 2013) generate random texts but lack contextual understanding essential for stress detection. Token-level augmentations benefit simpler tasks more, hindering performance in complex ones (Chen et al., 2023). EDA (Wei & Zou, 2019) improves with machine learning techniques (Ansari, Garg & Saxena, 2021), but limited benefits are observed for BERT and RoBERTa (Feng et al., 2021). In mental health classification on Reddit, RoBERTa and EDA yield minimal enhancement, attributed to EDA’s efficacy in smaller datasets (Murarka, Radhakrishnan & Ravichandran, 2021). Data noising (Wei & Zou, 2019; Karimi, Rossi & Prati, 2021) is unsuitable for social media noise. Sentence-based augmentation is also ineffective for Reddit due to large paragraphs (Ansari, Garg & Saxena, 2021). Local changes yield structurally similar sentences, making token or sentence-level techniques ineffective for RoBERTa in stress classification (Anaby-Tavor et al., 2020). Despite challenges, generative AI shows promise for superior data augmentation (Yoo et al., 2021; Dai et al., 2023), particularly with RoBERTa. Further research in this direction can overcome mental health classification limitations.

Model

Our baseline stress detection model, RoBERTa (Liu et al., 2019), optimizes BERT pretraining with enhanced performance, denoising autoencoding, and robust hyperparameter tuning. Chosen for its proven efficacy on mental health datasets, RoBERTa serves as a general yet effective baseline, substantiated by literature and memory considerations.

Dataset

Major mental health studies (Garg, 2023; Ansari, Garg & Saxena, 2021) predominantly utilize Reddit data. Benchmark datasets for stress detection include the Stress Annotated Dataset (SAD) (Mauriello et al., 2021) and DREADDIT (Turcan & McKeown, 2019).

SAD (Mauriello et al., 2021) comprises 6,850 SMS-like sentences categorized into nine stressor categories, offering daily stressor classifications and a collection of sentences derived from stress management literature, live chatbot conversations, crowdsourcing, and web scraping.

DREADDIT (Turcan & McKeown, 2019) presents a text corpus of 190,000 posts from five Reddit community categories, with 3.5K labeled segments for stress identification extracted from 3,000 posts. Covering ten subreddits related to abuse, social issues, anxiety, PTSD, and financial matters from January 2017 to November 2018, the dataset emphasizes expressions of stress with a negative attitude. This aligns with our stress classification research, emphasizing understanding user intent. Annotators focused on identifying both stress and a negative attitude in posts. The training set comprises 2,838 data points (51.6% labeled as stressful), and the test set has 715 data points (52.4% labeled as stressful).

Given the relatively small size of the dataset, data augmentation techniques are deemed necessary to enhance its robustness and effectiveness in stress classification.

Baseline augmentation techniques

Proposed augmentation techniques

In this section, we introduce a novel approach to data augmentation, termed 1-shot intent-aware data augmentation, leveraging generative AI models, specifically ChatGPT (GPT 3.5). Unlike traditional token-level augmentation, our methodology delves deeper into understanding user posts at a semantic level. An overview of our proposed method can be seen in Fig. 1.

Model training

Our baseline model is RoBERTa (Liu et al., 2019), chosen for enhanced training efficiency. Training utilized early stopping with key parameters, including a batch size of 8, learning rate of 1 × 10⁻⁶, Adam optimizer, and binary cross-entropy loss. The dataset was already divided into training and test sets. We then split the training data into training and validation sets with a ratio of 80:20, while keeping the test split unchanged. Data splits and experiments were conducted on an NVIDIA GeForce RTX 2080 Ti using the PyTorch framework.

Model evaluation metrics

Our stress detection model is evaluated using precision, recall, F1 score, and accuracy, tailored to the binary classification (Stressed or not Stressed).

Implementation of RoBERTa and data preprocessing

During the implementation of the baseline, a notable disparity in the performance of RoBERTa using raw and preprocessed data was observed. The employed data preprocessing techniques included:

1. Converting data to lowercase

2. Removing punctuation marks

3. Removing stop words and frequent words

4. Removing extra spaces and emojis

5. Removing words and digits containing digits

6. Removing URL links and HTML tags

7. Stemming and lemmatization

8. Expanding contractions

Examples of posts, both before and after preprocessing, are presented in Table 4.

We compared the performance of RoBERTa with and without preprocessing, evaluating accuracy, F1 score, recall, and precision (see Table 5). Corresponding confusion matrices are shown in Fig. 2. Results indicate that preprocessing tends to increase the number of posts labeled as stressed, possibly due to an enhanced focus on stressors rather than intent.

The investigation into data preprocessing revealed that RoBERTa performed commendably even without preprocessing. However, preprocessing led to a loss of contextual information.

Zero-shot stress classification with ChatGPT

For a better comparison we also performed zero-shot stress classification using ChatGPT. In contrast to the zero-shot classification results reported in the literature, we achieved an accuracy of 74.96 and an F1 score of 73.91 with precision and recall being 78.25 and 74.39 respectively using the following prompt:

Even though we achieved results better than that reported in literature, ChatGPT still falls short from RoBERTa in stress classification.

1-shot intent-aware data augmentation

The comparison of experimental results utilizing 1-shot intent-aware data augmentation with state-of-the-art techniques such as Back Translation (BT), EDA, and AEDA individually and in various combinations is presented in Table 6. ‘Concatenated’ in the table indicates that the augmentation techniques were applied to data and then the data was concatenated. While ‘both’ indicates that back translation was done on data and then the other augmentation technique was applied on both original and back translated texts.

It is noteworthy that intent-aware augmentation markedly enhances the model performance by 3%, with opposite intent augmentation achieving an F1 score of 82% and same intent augmentation yielding an F1 score of 83%. BT and AEDA do not improve the RoBERTa’s performance, while EDA causes further degradation.

Size of augmented data

Exploration of different augmentation sizes for our proposed 1-shot intent-aware augmentation technique (same and opposite intent) was conducted where we gradually increased the size of augmented data and observed its effect on the performance of the model as shown in Fig. 3. The observed improvement in model results was noted, but a saturation point in data augmentation effectiveness was identified, emphasizing the need for real-time data beyond a certain threshold.

Statistical analysis of augmented data

We conducted a comprehensive statistical analysis, employing T-test, P-value, and BLEU score to assess the impact of data augmentation on model performance. The statistical values for our proposed technique, compared with state-of-the-art methods, are presented in Table 7.