Journal of Personality Assessment ISSN: 0022-3891 (Print) 1532-7752 (Online) Journal homepage: https://www.tandfonline.com/loi/hjpa20 Dimensionality, Reliability, Invariance, and Validity of the Multidimensional Social Anxiety Response Inventory–21 (MSARI–21) James Deller, Jessica Perrotte, Katherine Wainwright, Joshua Brunsman & Augustine Osman To cite this article: James Deller, Jessica Perrotte, Katherine Wainwright, Joshua Brunsman & Augustine Osman (2019): Dimensionality, Reliability, Invariance, and Validity of the Multidimensional Social Anxiety Response Inventory–21 (MSARI–21), Journal of Personality Assessment To link to this article: https://doi.org/10.1080/00223891.2019.1569529 Published online: 25 Mar 2019. Submit your article to this journal View Crossmark data Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=hjpa20 JOURNAL OF PERSONALITY ASSESSMENT https://doi.org/10.1080/00223891.2019.1569529 Dimensionality, Reliability, Invariance, and Validity of the Multidimensional Social Anxiety Response Inventory–21 (MSARI–21) James Deller , Jessica Perrotte , Katherine Wainwright, Joshua Brunsman, and Augustine Osman Department of Psychology, The University of Texas at San Antonio ABSTRACT ARTICLE HISTORY Research shows that social anxiety disorder (SAD) is prevalent in the United States, and could interfere with many aspects of a person’s life. Although numerous psychological instruments have been developed to measure presence and intensity levels of social anxiety, these instruments fail to capture the range of responses individuals use to mitigate the negative affect associated with the anxiety, namely alcohol use and anger distress. Recently, the Multidimensional Social Anxiety Response Inventory–21 (MSARI–21) was developed to address this limitation and increase our understanding of the complexity of social anxiety. We expand on this work by evaluating the psychometric properties of the instrument, using a combination of exploratory structural equation and bifactor modeling, and item response techniques. Across 2 studies, data indicated the presence of a strong, 3-factor structure (i.e., anger distress, alcohol reliance, and social avoidance), strong internal consistency, and evidence of both convergent and discriminant validity. In addition, results showed that the MSARI–21 multidimensional structure was invariant across gender. We conclude that the MSARI–21 is a valid and valuable tool for assessing individuals’ responses to social anxiety, and that future research should evaluate the instrument within other samples to ensure its utility across clinical and subclinical populations. Received 30 April 2018 Accepted 2 December 2018 Social anxiety disorder (SAD) is a psychological disorder in which individuals experience feelings of intense fear or anxiety in social situations, largely due to their expectations of being (negatively) evaluated by other people.1 In addition to intense negative emotional states, SAD can also interfere with an individual by adversely influencing their social, professional, and recreational activities (e.g., Buckner & Heimberg, 2010; Lochner et al., 2003). Unfortunately, SAD is one of the most common psychological disorders within the United States (Kessler, Berglund, et al., 2005; Kessler, Chiu, Demler, & Walters, 2005). Given the widespread nature of the disorder and the negative impacts on functioning, SAD has been the focus of attention for clinicians and researchers alike. For example, the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM–5; American Psychiatric Association, 2013) made a number of definitional changes to SAD (B€ ogels et al., 2010) to aid in clinical research and diagnoses. In addition, several instruments have been developed to better assess the symptoms individuals experience, and identify the social situations or states that could exacerbate the disorder. Examples include the Brief Fear of Negative Evaluation Scale (BFNE; Leary, 1983), the Liebowitz Social Anxiety Scale (LSAS; Liebowitz, 1987), and the Social Phobia and Anxiety Inventory (SPAI; Turner, Beidel, Dancu, & Stanley, 1989). CONTACT James Deller 78249, USA. jamescdeller@gmail.com Although existing self-report measures of social anxiety provide important screening or assessment information, it is important to note that these instruments often assess different aspects of social anxiety (Modini, Abbott, & Hunt, 2015). For example, many emphasize the degree of distress felt from social anxiety (e.g., the Social Phobia and Anxiety Inventory [SPAI–23]; Roberson-Nay, Strong, Nay, Beidel, & Turner, 2007) rather than the responses to that distress. Those that measure behavioral responses to social anxiety (e.g., the Subtle Avoidance Frequency Examination [SAFE]; Cuming et al., 2009) typically capture only a single dimension of social anxiety (i.e., frequency of avoidance or safety behaviors). To understand fully the nature of the social anxiety construct, researchers and clinicians must investigate a wider range of emotional and behavioral responses linked to social anxiety across empirically supported dimensions of the construct. That is, individuals with social anxiety might respond in ways other than these safety behaviors or social avoidance, such as reacting with anger or turning to alcohol to cope. Although such behaviors have been linked to social anxiety within the broader literature, they have largely been ignored by current social anxiety measures. To address this gap, Osman, Freedenthal, Acosta, and Pirani (2015) recently developed the Multidimensional Social Anxiety Response Department of Psychology, University of Texas at San Antonio, One UTSA Circle, San Antonio, TX ß 2019 Taylor & Francis Group, LLC 1 We indicated the “negative” in parentheses because of the ongoing discussion in the extant literature that social anxiety should be reconceptualized to consider anxiety-related responses to positive comments (see, e.g., Skocic, Jackson, & Hulbert, 2015). 2 DELLER ET AL. Inventory–21 (MSARI–21) to better assess and understand the reasons individuals with social anxiety behave in such ways in response to being observed, criticized, or judged in various social situations. In this project, we build on this research through an examination of the psychometric properties of the MSARI–21 in two new nonclinical samples. In the next section, we briefly review the literature on the common affective and behavioral responses to social anxiety that informed the development of the instrument and provide a summary of steps in the construction of the instrument (more detailed analyses are reported in the unpublished manual; Osman et al., 2015). Social anxiety and social avoidance One of the most common ways people with social anxiety might respond to potentially distressing social situations is to avoid them altogether. For example, people who experience marked social anxiety might refuse to attend a social event or only make a brief appearance and leave early to manage their social anxiety symptoms. Indeed, social avoidance has been so closely linked to SAD that it is used in the diagnoses and assessment of the disorder per the DSM–5 (American Psychiatric Association, 2013). Several existing measures examining social avoidance, such as the LSAS (Liebowitz, 1987) and the Social Avoidance and Distress Scale (SADS; Watson & Friend, 1969) include measures of social avoidance to determine severity of social anxiety. Although avoiding potentially stressful social situations might help prevent experiencing SAD symptoms, such behaviors are maladaptive and can have significant negative consequences. For example, many career paths require extensive social interaction situations (e.g., making presentations at formal or informal meetings at a workplace). People who experience social anxiety and consequently avoid these events or situations could be limiting their job opportunities or their career advancements (Wittchen & Beloch, 1996). Additionally, such behaviors might lead to other mental health difficulties. In support of this, Moitra, Herbert, and Forman (2008) found that avoidance behavior was a significant mediating factor between SAD and the development of major depressive disorder. Social anxiety and alcohol use Individuals who present with social anxiety might also try to cope with their fear or anxiety by turning to the use of illicit or licit substances (in particular, alcohol). Among researchers, the notion that alcohol is used to reduce tension associated with anxiety and serve as a “social lubricant” is not new (see Critchlow, 1986). For those who experience social anxiety, using alcohol to cope with social situations could be a desirable strategy for a number of reasons. As suggested by a recent model of social anxiety and substance use (Buckner, Heimberg, Ecker, & Vinci, 2013), alcohol might increase positive affect when confronted with a social situation, attenuate an individual’s fear of negative appraisals from others, or help facilitate social interactions. Note, however, that research does not necessarily suggest that increased alcohol use is a hallmark of increased social anxiety (e.g., Ham, Bacon, Carrigan, Zamboanga, & Casner, 2016; Ham, Zamboanga, Olthuis, Casner, & Bui, 2010). Rather, research indicates social anxiety might be related to alcohol use through both positive and negative alcohol expectancies (Ham et al., 2016). Ham and colleagues (2010) reported that students who endorsed greater amounts of social anxiety did not engage in as much drinking as students who reported less social anxiety. However, students with social anxiety had stronger expectations that alcohol use would reduce tension compared to students who were not as socially anxious (Ham et al., 2010). Similarly, Buckner and Heimberg (2010) found people who endorsed high levels of social anxiety did not differ in their reports of drinking (in either quantity or frequency) compared to individuals with lower levels of social anxiety, but were more likely to report using alcohol as an aid for social situations. It is important to note that although people who present with social anxiety might not necessarily be drinking more, SAD is related to an increased prevalence of alcohol use disorder (AUD; Morris, Stewart, & Ham, 2005). Indeed, over a 14-year longitudinal study, individuals with SAD were more likely to develop an alcohol dependence condition compared to those without SAD (Buckner et al., 2008). Social anxiety and anger distress Research indicates people with anxiety disorders might also be more prone to reacting with feelings of anger or aggression. Moscovitch, McCabe, Antony, Rocca, and Swinson (2008) found that individuals with panic disorders and those with social phobia disorders were more likely to experience feelings of anger compared to control groups. Similarly, another study showed that individuals who met the criteria for generalized anxiety disorder (GAD) reported higher levels of trait anger and hostility compared to those who did not (Desch^enes, Dugas, Fracalanza, & Koerner, 2012). Desch^enes and colleagues also found that the measures of anger and hostility were significant predictors of the overall severity of the anxiety symptoms. Regarding social anxiety specifically, individuals with social anxiety have been found to spend greater portions of their day feeling angry compared to those without social anxiety (Kashdan & Collins, 2010). In addition to having higher levels of state and trait anger, individuals with social anxiety also display an increased tendency to react with anger to negative events (e.g., being criticized) as well as without direct provocation (Erwin, Heimberg, Schneier, & Liebowitz, 2003). The link between anger and social anxiety might be due to individuals who present with social anxiety using less effective strategies to regulate their emotions. Research shows that individuals with social anxiety tend to engage in rumination and suppression strategies to regulate their emotional states (e.g., Blalock, Kashdan, & Farmer, 2016; Kocovski, Endler, Rector, & Flett, 2005). Both of these regulation strategies have been found to be ineffective at regulating anger, typically resulting in the anger being maintained or decreasing overall MSARI–21 PSYCHOMETRICS positive affect (e.g., Gross, 2002). This is consistent with Erwin et al. (2003), who found that people with SAD were more likely to engage in suppression strategies to control their anger and were more likely to display higher state and trait anger compared to nonclinical controls. Brief overview of the development of the MSARI–21 Osman and colleagues (2015) noted that one major limitation of most existing self-report instruments is the use of the essential symptoms of social anxiety symptoms (i.e., intensity of fear or anxiety about being criticized or judged). In addition, other instruments tend to be composed of specific situations (e.g., participating in a small group event and asking an attractive person of the opposite sex for a date) that elicit anxiety-related responses (e.g., avoidance) or assess a relatively narrow range of responses (e.g., safety behaviors to minimize or conceal symptoms). Accordingly, Osman and colleagues (2015) noted the need for a measure that would assess several content-specific responses and the thoughts behind these responses within individuals with SAD. Although a battery of existing psychological instruments could theoretically be developed to investigate such responses and their underlying causes, Osman and colleagues (2015) sought to develop a more concise and specific measure to minimize participant burden and fatigue. Osman and colleagues (2015) conceptualized several dimensions (i.e., social avoidance, anger distress, and alcohol reliance) for enhancing our understanding of the functioning of individuals who present with SAD. Specifically, social avoidance refers to a general tendency to avoid social situations. Anger distress refers to a general tendency to experience internalized anger within social situations. Finally, alcohol reliance refers to an individual’s tendency to use alcohol as a means to alter his or her behaviors and control anxiety in social situations. The authors highlighted the importance of assessing these affective and behavioral responses of social anxiety and the underlying thoughts behind these responses, asserting such a measure would be invaluable to the understanding of the nature of SAD. Because maladaptive responses to SAD can exacerbate the disorder or contribute to the development of other psychological problems (e.g., substance abuse, loneliness, depression), a measure that can quickly identify and assess content-specific responses and underlying thoughts could be instrumental in SAD treatment. Following the conventional qualitative steps (i.e., generating items from multiple sources, reviewing items for content specificity, clarity, and relevancy), Osman et al. (2015) developed the MSARI–21 across two studies, examining the factor structure of the instrument as well as convergent and discriminant validity.2 2 The analyses for each of these studies are available in detail in an unpublished manual (Osman et al., 2015). Results from these studies provided strong empirical support for a three-factor model (i.e., anger distress, alcohol reliance, and social avoidance), and each factor was comprised of seven items. 3 In one of the pilot investigations, undergraduate students (N ¼ 15), who obtained high scores on the Social Anxiety Life Interference (SALI) scale of the Social Anxiety and Depression Life Interference Inventory (SADLI; see Garcia, Acosta, & Osman, 2016; Osman, Bagge, Freedenthal, Gutierrez, & Emmerich, 2011) contributed to (a) generating items, and (b) evaluating the content relevancy of the final version of the instrument (see Osman et al., 2015). Overview of objectives and planned analyses The goal of this project was to use modern statistical techniques to further examine the psychometric properties of the MSARI–21 instrument. Specifically, we identified the following objectives to guide this project: 1. 2. 3. 4. Examine confirmatory data for the multidimensional structure of the MSARI–21 items (Study 1; N ¼ 629). Reexamine evidence of internal consistency reliability for scores on the MSARI–21 scales by calculating internal consistency reliability estimates using alternative estimation methods (Study 1; N ¼ 629). Examine measurement invariance across male and female study participants for the items within each MSARI–21 scale (Study 1). Establish evidence of convergent and discriminant validity estimates for the MSARI–21 scale scores using empirically grounded measures available within the literature (Study 2). For the first objective, we adopted exploratory structural equation modeling (ESEM) and bifactor modeling techniques. An advantage of ESEM is that it incorporates aspects from both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) to overcome inherent limitations in either technique. That is, unlike the conventional CFA strategy, ESEM allows items to load freely across factors and thus is less restrictive and allows for interpretable item-factor loadings or factor intercorrelations (Marsh, Morin, Parker, & Kaur, 2014). We conducted ESEM with Mplus version 7.4 (Muthen & Muthen, 1998–2011), comparing the fit of two models (a unidimensional model and the proposed three-factor oblique model) using a robust maximum likelihood estimator (MLR) and oblique geomin rotation. In addition to v2 values, we also assessed model fit using the following indexes and suggested cutoffs: comparative fit index (CFI) and Tucker–Lewis Index (TLI), values  .95 (Schermelleh-Engel, Moosbrugger, & M€ uller, 2003); root mean square error of approximation (RMSEA), values  .08 (Browne & Cudeck, 1993); and standardized root mean square residual (SRMR), values  .08 (Hu & Bentler, 1999). We also estimated an ESEM bifactor model, in which items are specified to load on their respective factors (i.e., specified subscales) as well as on a “general” factor (i.e., an underlying factor of “general social anxiety”). This procedure allowed us to calculate two additional indexes to interpret the dimensionality of the instrument: the explained common 4 DELLER ET AL. variance (ECV) and the percentage of uncontaminated correlations (PUC). The ECV refers to the proportion of common variance that is due to the underlying general factor relative to the common variance explained by the group factors (Reise, 2012). PUC refers to the ratio of the number of “uncontaminated” matrix correlations (i.e., correlations due solely to the general factor) to total number of unique correlations. High ECV and PUC values suggest stronger influence of the general factor, and thus suggest unidimensionality, whereas lower values suggest multidimensionality. Reise, Scheines, Widaman, and Haviland (2013) suggested using an ECV benchmark of .60 when PUC values are less than .80. The second objective served to extend previous work with the MSARI–21 by calculating internal consistency reliability estimates for the scale scores. Although Cronbach’s coefficient a (Cronbach, 1951) is typically used for this purpose, it might often be an inaccurate estimate due to unrealistic assumptions. That is, coefficient a can underestimate reliability when the assumption of tau-equivalence is violated, and overestimate when the assumption of uncorrelated error variances is violated (Raykov & Marcoulides, 2015). Due to these potential issues, we calculated point and interval estimates for coefficient x (McDonald, 1999), which does not hold these assumptions. This measure of internal consistency reliability was calculated using the R package ‘coefficient alpha’ (Zhang & Yuan, 2016). For the third objective, we assessed measurement invariance across male and female participants. To ensure items on the MSARI–21 scales have the same interpretation across these groups, we conducted differential item functioning (DIF) analyses using Item Response Theory for Patient Reported Outcomes (IRTPRO) Version 4.2 for each of the three scales (Cai, Thissen, & du Toit, 2011). This approach allowed us to calculate the DIF (v 2 cja) and fit statistics (S – v2) for each item for both males and females. In Study 2, we examined evidence for convergent and discriminant validity estimates of the MSARI–21 scale scores by assessing the associations between the MSARI–21 individual scale scores and scores on well-established self-report measures of drinking motives, anger expression, social phobia and anxiety, and other psychiatric symptoms found within the broader literature. We predicted that the MSARI–21 scale scores would be moderately to highly correlated with scores on measures of similar constructs. To this end, we used SPSS version 22 and R software (Version 3.3.3) to examine the pattern of associations between scores on the Anger Distress, Alcohol Reliance, and Social Avoidance scales of the MSARI–21 to scores on other empirically established measures of expressions of anger, drinking behaviors, and social anxiety. Study 1 Method Participants Following institutional review board (IRB) approval, the SONA-Qualtrics Web platform was used to administer the questionnaires to undergraduate students at a large Southwestern state university. The questionnaire could be completed within 30 to 35 min. Accordingly, of the initial pool of 640 participants, data for individuals (n ¼ 11) completing the study within 5 min or less were not included in the current analyses. The final sample with complete itemlevel data included 629 participants (408 females, 221 males; M ¼ 19.78 years, SD ¼ 3.44 years). The self-reported ethnic background of this sample was 47.7% Hispanic, 28.1% Caucasian or White, 7.6% African American, 9.4% Asian, 4.9% biracial or multiethnic, and 2.2% Middle Eastern. Procedure Measures Multidimensional Social Anxiety Response Inventory. The MSARI–21 (Osman et al., 2015) is a 21item measure that assesses the affective and behavior responses and underlying reasoning for these responses within individuals with SAD. Participants read over several reactions to social evaluations or interactions and rate how well each describes their own thoughts, feelings, or behaviors, using a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). The measure consists of three sevenitem scales: Anger Distress, Alcohol Reliance, and Social Avoidance. The Anger Distress scale assesses the extent to which the individual responds to socially anxious situations by experiencing internalized anger in response to being evaluated (negatively or positively) and specific reasons for this behavior (e.g., feeling judged by others). The Alcohol Reliance scale assesses the extent to which the individual uses alcohol to cope with feelings of social anxiety and underlying reasons (e.g., alcohol improves self-presentation). The Social Avoidance scale assesses avoidance and specific reasons for avoiding social situations or interactions (e.g., uncertainty about being called upon to say or do something). For copyright reasons, permission was granted for reproducing only abbreviated items. Results ESEM and bifactor models Fit indexes for the tested models are shown in Table 1. The one-factor model showed very poor fit for the sample data, v2(189) ¼ 4,678.71, p < .001; CFI ¼ 0.38, TLI ¼ 0.31, RMSEA ¼ 0.19, CI [.19, .20], SRMR ¼ .21. However, the three-factor oblique model showed good fit, v2(150) ¼ 391.46, p < .001; CFI ¼ 0.97, TLI ¼ 0.95; RMSEA ¼ 0.05, 90% CI [.04, .06], SRMR ¼ .02. In addition, as shown in Table 2, scale items in this model showed strong loadings on their predicted factors, with minimal cross-loadings. The bifactor model, v2(132) ¼ 272.57, p < .001; CFI ¼ 0.98, TLI ¼ 0.97, RMSEA ¼ 0.04, 90% CI [.03, .05], SRMR ¼ .02, also demonstrated good fit to the sample data. As noted previously, the bifactor model allows the researcher to calculate bifactor-specific indexes (ECV ¼ .38, PUC ¼ .70). Taken as a whole, these results provide further evidence for the multidimensionality of the MSARI–21, and that each of the three MSARI–21 PSYCHOMETRICS Table 1. Model fit statistics for Study 1 and Study 2. 90% CI for RMSEA Study 1 One-factor Three-factor Bifactor Study 2 Three-factor v2 df CFI TLI SRMR RMSEA LL UL 4678.71 391.46 272.57 189 150 132 .38 .97 .98 .31 .95 .97 .21 .02 .02 .19 .05 .04 .19 .04 .03 .20 .06 .05 491.37 186 .92 .91 .05 .08 .07 .09 Note. CFI ¼ comparative fit index; TLI ¼ Tucker–Lewis Index; SRMR ¼ standardized root mean square residual; RMSEA ¼ root mean square error of approximation; LL ¼ lower limit; UL ¼ upper limit. p < .001. Table 2. Standardized factor loadings for Study 1. Three-factor model Item One-factor model AngD 1 2 6 9 14 16 19 5 8 10 12 17 20 21 3 4 7 11 13 15 18 .41 .44 .39 .51 .52 .53 .49 .42 .37 .39 .44 .47 .38 .46 .73 .72 .81 .76 .81 .77 .80 .67 .72 .74 .78 .80 .83 .79 .01 .01 .01 .00 .01 .01 .00 .01 .12 .04 .22 .00 .04 .01 AlcR .04 .05 .06 .05 .05 .01 .03 .82 .89 .92 .85 .64 .91 .91 .03 .04 .03 .01 .01 .02 .00 Bifactor model SocA General AngD .04 .01 .12 .01 .05 .01 .01 .05 .04 .02 .06 .18 .03 .05 .78 .82 .85 .64 .88 .80 .86 .22 .17 .24 .28 .21 .25 .22 .82 .85 .90 .88 .72 .89 .93 .24 .28 .28 .31 .32 .28 .30 AlcR SocA .64 .69 .70 .75 .76 .78 .74 .30 .35 .13 .08 .18 .01 .02 .76 .79 .79 .59 .82 .74 .79 Note. N ¼ 629. AngD ¼ Anger Distress; AlcR ¼ Alcohol Reliance; SocA ¼ Social Avoidance. Factor loadings > .40 are shown in bold. scales should be scored separately rather than combined into a total score. Internal consistency reliability Given the evidence for multidimensionality, we calculated the internal consistency reliability estimates and related 95% confidence intervals for each scale score of the MSARI–21. As shown in Table 3, all three-scale scores demonstrated excellent estimates of internal consistency reliability. Measurement invariance To determine if items within each scale would be useful across important demographic groups, we examined item functioning across males and females for each scale separately. The complete DIF and fit statistics are displayed in Table 4. To assess adequacy of the discrimination parameters, we used the range of values recommended by Baker and Kim (2004). Within the Anger Distress scale, we found that the discrimination parameters were very high for males, ranging from 2.01 (Item 1) to 3.79 (Item 14). Similarly, the discrimination parameters were high to very high for 5 Table 3. Descriptive statistics and reliability estimates for Study 1 and Study 2. M SD x 95% CI Study 1 (N ¼ 629) Alcohol Reliance Social Avoidance Anger Distress Scale 12.91 14.03 16.53 6.95 6.69 6.53 .95 .93 .90 [.94, .96] [.92, .94] [.89, .92] Study 2 (N ¼ 250) Alcohol Reliance Social Avoidance Anger Distress 10.78 13.18 15.56 5.98 6.67 6.58 .94 .92 .90 [.92, .96] [.90, .94] [.89, .91] Note. x ¼ coefficient x. females, ranging from 1.63 (Item 1) to 3.55 (Item 16). Furthermore, the comparative analysis found no evidence of DIF between the groups for the Anger Distress scale items. Within the Alcohol Reliance scale, the discrimination parameters for the items were very high for both males and females, ranging from 2.59 (Item 17) to 5.73 (Item 21) and 2.77 (Item 17) to 6.38 (Item 10), respectively. The analysis revealed mixed evidence for Item 10 (“It is helpful to me to have a drink containing alcohol before a social event”), as the DIF statistic approached significance for this single item. Further, the item-level diagnostic statistics for this item were significant for females, suggesting that this item could be performing differently between genders. However, we decided to retain this item for three reasons. First, the DIF only approached significance; second, retaining or dropping a single item would not substantially affect the overall score of the Alcohol Reliance scale; finally, we opted to retain the item to maintain the content validity of the scale. Within the Social Avoidance scale, the discrimination parameters for the items were very high for both males, ranging from 2.20 (Item 4) to 3.81 (Item 13) and females, ranging from 2.21 (Item 17) to 3.45 (Item 7). Similar to the Anger Distress scale, there was no evidence of DIF for any of the Social Avoidance scale items. Study 2 Method Participants and procedure The final sample for Study 2 was 250 undergraduate participants (161 female, 89 male; M ¼ 19.74 years, SD ¼ 4.29 years), after removing data for 8 participants who completed the questionnaires within 5 min or less. The self-reported ethnic background information for this sample was 54.4% Hispanic, 18.8% Caucasian or White, 12.0% African American, 8.4% Asian, 3.2% biracial or multiethnic, 1.2% Middle Eastern, 1.2% other ethnicities, and 0.8% American Indian or Indigenous. All participants provided informed consent and completed several questionnaires including the MSARI–21, a demographic questionnaire, and several concurrent instruments discussed in the measures subsection. Measures Drinking Motives Questionnaire, Revised. The Drinking Motives Questionnaire, Revised (DMQ–R; Cooper, 1994) is a 6 DELLER ET AL. Table 4. Fit statistics for differential item functioning analysis. Fit statistics a Femalesb Males Scale Anger distress Alcohol reliance Social avoidance Item 1 2 6 9 14 16 19 5 8 10 12 17 20 21 3 4 7 11 13 15 18 a 2.01 2.42 2.06 3.05 3.79 2.98 3.24 3.14 4.75 5.53 3.93 2.59 5.34 5.73 2.27 2.20 3.44 2.75 3.81 3.02 2.96 SE 0.24 0.28 0.25 0.36 0.50 0.36 0.43 0.37 0.60 0.76 0.48 0.35 0.72 0.83 0.27 0.27 0.45 0.37 0.52 0.39 0.37 2 S–v 50.74 50.83 66.29 42.08 38.39 43.24 32.99 39.27 43.93 38.50 54.02 44.12 39.87 36.73 37.80 48.94 29.06 62.62 24.09 34.99 40.35 df 39 37 46 34 30 35 29 38 33 28 32 30 24 25 36 38 27 35 26 29 33 p .10 .06 .03 .16 .14 .16 .28 .41 .09 .09 .01 .05 .02 .06 .39 .11 .36 .01 .57 .20 .18 a 1.63 1.96 1.85 2.53 2.80 3.55 2.71 3.80 4.37 6.38 5.02 2.77 5.37 8.86 2.57 2.62 3.45 2.21 3.59 2.85 3.34 SE 0.30 0.34 0.33 0.44 0.50 0.60 0.48 0.36 0.42 0.74 0.54 0.31 0.59 1.40 0.22 0.23 0.33 0.20 0.36 0.26 0.32 2 S–v 69.67 68.95 69.53 58.73 39.65 44.25 54.28 48.43 56.25 43.47 49.14 57.34 38.48 20.80 41.96 52.64 52.85 104.72 35.00 53.74 44.67 DIF statistics df 60 57 61 50 46 42 42 43 33 27 31 34 30 22 44 47 40 48 35 41 40 p .18 .13 .21 .19 .73 .38 .10 .26 .01 .02 .02 .01 .14 .53 .56 .26 .08 .00 .47 .09 .28 v2cja 5.7 4.8 4.2 6.9 2.7 2.3 1.8 2.5 3.3 11.3 1.8 0.5 0.5 1.0 2.3 1.0 2.0 1.3 1.5 0.8 7.5 p .22 .32 .38 .14 .62 .69 .78 .65 .50 .02 .78 .97 .97 .91 .68 .91 .73 .87 .83 .94 .11 Note. DIF ¼ differential item functioning; a ¼ discrimination parameter; SE ¼ standard error; S – v2 ¼ item fit statistics; df ¼ degrees of freedom. Values shown in bold represent items where significant DIF was potentially detected. a n ¼ 221. bn ¼ 408. 20-item instrument that assesses different motives individuals might have for using alcohol. Participants read over the different reasons people might drink, and indicate how often each is a factor in their own drinking behavior, using a 5point scale ranging from 1 (almost never) to 5 (almost always). The instrument is composed of four five-item subscales, tapping into different drinking motives (social, coping, enhancement, and conformity). Social motives include using alcohol as a way to celebrate, or improve special occasions or social events. Coping motives include using drinking as a way to ignore or reduce one’s negative thoughts and feelings. Enhancement motives include drinking to enhance a positive mood. Conformity motives include using alcohol as a means to “fit in” with others as well as to avoid negative social repercussions such as ridicule and rejection. State–Trait Anger Expression Inventory. The 32-item Anger Expression scale of the State–Trait Anger Expression Inventory (STAXI-2; Spielberger, 1999) assessed how participants experience, express, and control their anger. This scale is composed of four eight-item subscales: the ExpressionOut (AX-O), Anger Expression-In (AX-I), Anger ControlOut (AC-O), and Anger Control-In (AC-I). For each, participants indicate how often they engage in specific reactions or behaviors when they feel angry, using a 4-point scale ranging from 1 (almost never) to 4 (almost always). The AX-O subscale captures the extent to which individuals express their anger through outward behaviors, such as slamming doors or yelling at those around them. The AX-I subscale measures the extent to which individuals try to “bottle up” their anger. The AC-O subscale measures how often individuals attempt to control their anger by actively monitoring and controlling their behaviors. The AC-I subscale captures the extent to which individuals attempt to control their anger by making themselves calm down or “cool off.” Scores on these subscales are also used to calculate an overall anger expression index (AX index) for each individual. Social Phobia and Anxiety Inventory. The SPAI–23 (Roberson-Nay et al., 2007) is a 23-item measure of social (16 items) and agoraphobic anxiety (7 items). Items on the agoraphobic scale assess anxiety that arises from perceptions of the setting (e.g., open spaces, public transportation) rather than interactions with others. Participants report how often they experience various thoughts or feelings when responding to the questionnaire items, using a 5-point scale ranging from 0 (never) to 4 (always). Scores on the agoraphobia subscale can be subtracted from scores on the social anxiety scale to provide a more accurate measure of SAD (Turner, Beidel, & Dancu, 1996). Symptom Assessment 45 Inventory. The Symptom Assessment 45 Inventory (SA–45; Maruish, Bershadsky, & Goldstein, 1998) is a multidimensional checklist of psychiatric symptoms. The instrument asks individuals to rate how much they were bothered by each symptom over the last week, using a 5-point scale ranging from 1 (not at all) to 5 (extremely). It is composed of nine five-item subscales: anxiety (e.g., “Feeling tense or keyed up”), depression (e.g., “Feeling no interest in things”), hostility (e.g., “Shouting or throwing things”), interpersonal sensitivity (e.g., “Feeling that people are unfriendly or dislike you”), obsessive–compulsive disorder (e.g., “Having to check and double-check what you do”), paranoid ideation (e.g., “Feeling that most people cannot be trusted”), phobic anxiety (e.g., “Feeling afraid in open spaces or on the streets”), psychoticism (e.g., “Hearing voices that other people do not hear”), and somatization (e.g., “Soreness of your muscles”). Scores on each MSARI–21 PSYCHOMETRICS 7 Table 5. Standardized factor loadings for Study 2. MSARI–21 Abbreviated item 1. I get mad easily … in front of other people. 2. I get intensely angry … when I say or do something. 6. I feel like telling people off … comments about me. 9. I feel intensely annoyed … comments about what I say or do. 14. My anger toward people … be quite intense. 16. I experience intense feelings … failings. 19. I feel the urge … in a social situation. 5. Drinking alcohol helps … talking with people. 8. Drinking alcohol before … comfortably with other people. 10. It is helpful to me … before a social event. 12. I have better control … contains alcohol. 17. I am only able … a drink containing alcohol. 20. Having a drink … my presentation or performance. 21. It usually works best … social gathering. 3. I avoid social events because … be expected to do or say. 4. I avoid social get-togethers because … myself to other people. 7. I often find ways … about my true feelings. 11. I avoid social encounters because … in front of other people. 13. I avoid social … when I am around people. 15. I avoid social events … other people. 18. I often wiggle my way … other people. Anger Distress Alcohol Reliance Social Avoidance .73 .80 .61 .68 .74 .74 .77 .76 .88 .92 .78 .63 .88 .89 .74 .78 .80 .69 .86 .79 .84 Note. N ¼ 250. All factor loadings significant at p < .001; v2 (186) ¼ 91.37, p < .001; comparative fit index ¼ 0.92; MSARI–21 ¼ Multidimensional Social Anxiety Response Inventory–21; Tucker–Lewis Index ¼ 0.91, root mean square error of approximation ¼ 0.08, 90% CI [.07, .09], standardized root mean square residual ¼ 0.05. subscale are summed to create a total score. The subscales of the SA–45 were used to examine evidence of convergent and discriminant validity for the MSARI–21 scale scores. Results Item factor loadings and fit indexes of the three-factor solution for the MSARI–21 in Study 2 are reported in Table 5 Convergent and discriminant validity The final aim was to examine convergent and discriminant validity of the MSARI–21. Specifically, we examined the correlations between scores on each MSARI–21 scale and scores on established, widely used instruments that assess the same or similar constructs. We expected moderate to high (r  .40) correlations between participants’ scores on (a) the Anger Distress scale with the STAXI–2 anger expression (AX-I, AX-O, and AX Index) and the SA–45 hostility scale scores; (b) the Alcohol Reliance scale with the DMQ–R subscale scores related to improving mood or social events (Enhancement and Social) and reducing negative thoughts (Coping); and (c) the Social Avoidance scale with scores on the SPAI–23 (social phobia and difference scores) and SA–45 phobic anxiety scale scores. For discriminant validity, we predicted weak or low (r < .40) correlations between scores on the MSARI–21 scales and scores on measures of dissimilar constructs. In addition, we assessed discriminant validity using dependent correlational analyses. The complete correlations are presented in Table 6. Anger Distress. We first examined the correlations between scores on the MSARI–21 Anger Distress scale and scores on the STAXI–2 measure. As expected, the Anger Distress scale had moderate positive correlations with scores on the Anger Index (r ¼ .49) and the AX-I (r ¼ .44) scale and a weak negative relationship with the AC-O (r ¼ –.34) and AC-I (r ¼ –.27). However, the Anger Distress scale had a weak positive relationship with scores on the AX-O scale (r ¼ .34). In addition, we also calculated dependent correlation tests (Steiger, 1980) to better examine evidence of discriminant validity for the Anger Distress scale. The Anger Distress scale was found to be more strongly related to the AX-I scale score than the AC-I scale score, t(249) ¼ 8.95, p < .001. Scores on the Anger Distress scale were also found to be more strongly associated with AX-O scale scores than the AC-I scale scores, t(249) ¼ 6.49, p < .001, and more related to Anger Index than AC-I scale scores, t(249) ¼ 7.04, p < .001. Finally, the SA–45 features a hostility scale score that was also found to be more strongly related to the Anger Distress scale score when compared to the AC-I scale score, t(249) ¼ 8.84, p < .001. Alcohol Reliance. Next, we examined the correlations between scores on the MSARI–21 Alcohol Reliance scale and scores on the DMQ–R. The MSARI–21 Alcohol Reliance scale score was highly correlated with the DMQ–R Social Motives scale scores (r ¼ .62), Coping Motives scale scores (r ¼ .64), and Enhancement Motives scale scores (r ¼ .56), all ps < .05. The Conformity Motives scale scores of the DMQ–R were weakly correlated with scores on the Alcohol Reliance scale (r ¼ .37). These findings support our hypotheses and provide evidence for convergent validity. Again, the dependent correlation test was used to help establish evidence for discriminant validity. This test showed that the MSARI–21 Alcohol Reliance scale score was more related to the DMQ–R Social Motives scale score, t(249) ¼ 4.85, p < .001, the Coping Motives scale score, t(249) ¼ 5.5, p < .001, and the Enhancement Motives scale score, t(249) ¼ 3.29, p ¼ .001, when compared to the Conformity Motives scale score. Social Avoidance. We also examined the correlations between scores on the MSARI–21 Social Avoidance scale and scores on the SPAI–45. As predicted, the Social 8 DELLER ET AL. Table 6. Correlations between MSARI–21 scales and concurrent measures. MSARI–21 Measures Scales STAXI–2 Anger Distress Alcohol Reliance Social Avoidance AX-O AX-I AC-O AC-I AX Index .34 .44** .34 .27 .49** .08 .18 .04 .03 .09 .04 .43** .09 .09 .24 Social Phobia Agoraphobia Difference Score .39 .24 .36 .24 .14 .22 .72** .39 .70** Hostility Phobic Anxiety .50** .32 .10 .14 .17 .49** Social Coping Enhancement Conformity .06 .27 .08 .14 .62** .64** .56** .37 .03 .12 .02 .12 SPAI–23 SA–45 DMQ–R Note. MSARI–21 ¼ Multidimensional Social Anxiety Response Inventory–21; STAXI–2 ¼ State–Trait Anger Expression Inventory; AX-O ¼ Anger ExpressionOut; AX-I ¼ Anger Expression-In; AC-O ¼ Anger Control-Out; AC-I ¼ Anger Control-In; AX Index ¼ Anger Expression Index; SPAI–23 ¼ Social Phobia and Anxiety Inventory; SA–45 ¼ Symptom Assessment; DMQ–R ¼ Drinking Motives  Questionnaire, Revised. Correlations  .40 are shown in bold.  p < .05. p < .01. Avoidance scale score was found to be highly correlated with the SPAI–45 Social Phobia scale score (r ¼ .72) and the SA–45 phobic anxiety scale score (r ¼ .49) and weakly correlated with the Agoraphobia scale score (r ¼ .39). Additionally, the difference scale score (a more accurate assessment of social phobia) was found to be highly correlated (r ¼ .70) with the MSARI–21 Social Avoidance scale score. To establish discriminant validity, Steiger’s (1980) dependent correlation test showed that the Social Avoidance scale score was more related to the Social Phobia scale score than to the Agoraphobia scale score, t(249) ¼ 7.79, p < .001. Finally, we examined the correlations between the Anger Distress and Alcohol Reliance scales with scores on the SPAI–23. Because these dimensions were conceptualized as responses to social anxiety, we expected the scores to be positively correlated. Scores on the Anger Distress scale were positively correlated with Social Phobia scale scores (r ¼ .39). Similarly, scores on the Alcohol Reliance scale were positively correlated with Social Phobia scores (r ¼ .24). General discussion Although individuals with social anxiety tend to avoid social situations, they might respond in other ways, namely reacting with internalized expressions of anger or relying on alcohol. However, current instruments have largely ignored these potential response tendencies. Osman and colleagues (2015) developed the MSARI–21 to better examine these behavioral responses and increase our understanding of social anxiety as a construct. This project built on this development by assessing the multidimensional nature of this new instrument, calculating estimates of internal reliability, assessing measurement invariance across genders, and assessing convergent and discriminant validity. In Study 1, we examined the multidimensional nature of the instrument using structural equation modeling (SEM) and bifactor modeling approaches. Results from the SEM and bifactor model revealed strong evidence for the multidimensionality of the instrument, suggesting that scores on each of the three scales should be calculated independently rather than summed to create an overall total inventory score. The analyses in Study 1 also revealed that each of the three MSARI–21 scale scores had adequate estimates of internal reliability for use in research and clinical settings (all estimates  .90). The third objective of the study was to investigate measurement invariance of the MSARI–21 instrument. Because of the reported gender differences in the presentation of social anxiety (both SAD and endorsement of social anxiety symptoms; Ranta et al., 2007; Xu et al., 2012), Study 1 assessed invariance across males and females using a DIF technique. These analyses revealed a potential issue for only a single item (Item 10 from the Alcohol Reliance scale) across all three scales. That is, the DIF statistics approached significance for only 1 of the 21 items of the MSARI–21. Given that Item 10 merely approached statistical significance, and dropping this item would not substantially alter the scores on the Alcohol Reliance scale, we recommend retaining this item in future investigations. The results of Study 1 provide strong evidence for the utility of the MSARI–21 across gender. However, research suggests that experiences and displays of social anxiety might be influenced by factors associated with race, ethnicity, or both (e.g., Hoffman, Asnaani, & Hinton, 2010; Okazaki, Liu, Longworth, & Minn, 2002). To provide evidence that the MSARI–21 is a robust measure for all racial, ethnic, and cultural groups, future validation analyses (e.g., factorial invariance via CFA, DIF) should be conducted across these different groups as well. In Study 2, we examined the structure of the MSARI–21 within a new sample and assessed the relationships between scores on the MSARI–21 scales and scores on established self-report measures of related and unrelated constructs. In addition to support for the three-factor structure of the measure, we found evidence for the convergent validity of the MSARI–21 scale scores. That is, scores on the Anger Distress scale were moderately to strongly positively related to other measures of anger and hostility (and negatively related to measures of anger control). Similarly, scores on the Alcohol Reliance scale (which assesses the extent to which individuals use alcohol to better cope with social situations) were positively related to drinking alcohol to improve mood or reduce negative thoughts and feelings. Scores on the Social Avoidance scale (which assesses the extent to which individuals attempt to avoid negative social situations altogether) were strongly related to scores on measures of social phobia. Study 2 also found a positive relationship (r ¼ .39) between Anger Distress scores and Social Phobia scores on the SPAI–23. This is in line with Breen and Kashdan (2011), who found a positive relationship between social anxiety and state anger (r ¼ .25). Although this project found a stronger relationship between anger and social anxiety, this MSARI–21 PSYCHOMETRICS is to be expected given that the Anger Distress scale items assess responses specific to social anxiety. Our findings of a positive correlation (r ¼ .24) between Alcohol Reliance scale scores and Social Phobia are also in line with Schry and White (2013), who found a positive relationship between social anxiety and both positive and negative alcohol expectancies (r ¼ .15 and r ¼ .16, respectively). Limitations and future research directions Several limitations must be considered when interpreting the findings of this project. For example, it is important to note that some characteristics of the sample could limit the generalizability of these findings to other groups. For Studies 1 and 2, we had data from relatively young (M age < 21), nonclinical, predominantly Hispanic college student samples. Although symptomatic experiences of social anxiety are generally common in this age group, clinical diagnoses of SAD are far less so (see Grant et al., 2005; Purdon, Antony, Monteiro, & Swinson, 2001). Thus, there is a need to extend the methodologies adopted here to data obtained from clinical inpatient and outpatient samples. In addition, there is still a need to examine invariance testing of the MSARI–21 at the scale level. Future studies could investigate this using other measurement invariance techniques, such as multiplegroup CFA. Another potential limitation relates to the nature of the MSARI–21 instrument. As the instrument is designed to assess both the responses to social anxiety and the underlying reasons behind these responses, the items could be fairly complex. For example, it is possible that participants might endorse the response (“I avoid social situations”) without endorsing the underlying reason (“because I am uncertain how to present myself”) and might experience some confusion on how best to respond. This could affect the results and reliability of the instrument. However, such complexity might be necessary given the instrument’s purpose of assessing the responses and underlying reasoning for these responses within individuals. In addition, the high reliability estimates found in Study 1 (all estimates  .90) might suggest that participants are not experiencing a high degree of uncertainty. Previous research has linked endorsement of social anxiety to adverse alcohol-related outcomes within nonclinical samples of college students (Gilles, Turk, & Fresco, 2006). Furthermore, research demonstrates that subclinical levels of anxiety can adversely influence the performance of an individual such as on working memory tasks (see Moran, 2016). Given the extensive cognitive demands placed on college-age students, future investigations might examine the extent to which scores on this scale differentiate the responses of students who present with a moderate to high level of social anxiety symptoms and those less influenced by the process of being evaluated in social situations (e.g., giving a presentation or working in a group). Future research should also expand on validating scores on this measure in a number of ways. First, as this project collected cross-sectional data, we did not examine evidence 9 for test–retest reliability or stability of the MSARI–21 scale scores over time. Data collected at multiple time points to more thoroughly assess for these measures of consistency would be beneficial to the further validation of the measure. Additionally, although this study found evidence of convergent validity with the SPAI–23, future research should also compare scores on the MSARI–21 with other measures of social anxiety, including behavioral assessments from observers. Funding Jessica Perrotte was supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under Award Number F31AA026477. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. ORCID James Deller http://orcid.org/0000-0001-7374-7621 http://orcid.org/0000-0002-4091-7820 Jessica Perrotte References American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Arlington, VA: American Psychiatric Publishing. doi:10.1176/appi.books.9780890425596 Baker, F. B., & Kim, S. H. (Eds.). (2004). Item Response Theory: Parameter estimation techniques. Boca Raton, FL: Taylor and Francis Group Blalock, D. V., Kashdan, T. B., & Farmer, A. S. (2016). Trait and daily emotion regulation in social anxiety disorder. Cognitive Therapy and Research, 40, 416–425. doi:10.1007/s10608-015-9739-8 B€ ogels, S. M., Alden, L., Beidel, D. C., Clark, L. A., Pine, D. S., Stein, M. B., & Voncken, M. (2010). Social anxiety disorder: Questions and answers for the DSM V. Depression and Anxiety, 27, 168–189. doi:10.1002/da.20670 Breen, W. E., & Kashdan, T. B. (2011). Anger suppression after imagined rejection among individuals with social anxiety. Journal of Anxiety Disorders, 25, 879–887. doi:10.1016/j.janxdis.2011.04.009 Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136 162). Newbury Park, CA: Sage doi: 10.1177/0049124192021002005 Buckner, J. D., & Heimberg, R. G. (2010). Drinking behaviors in social situations account for alcohol-related problems among socially anxious individuals. Psychology of Addictive Behaviors, 24, 640–648. doi: 10.1037/a0020968 Buckner, J. D., Heimberg, R. G., Ecker, A. H., & Vinci, C. (2013). A biopsychosocial model of social anxiety and substance use. Depression and Anxiety, 30, 276–284. doi:10.1002/da.22032 Buckner, J. D., Schmidt, N. B., Lang, A. R., Small, J. W., Schlauch, R. C., & Lewinsohn, P. M. (2008). Specificity of social anxiety disorder as a risk factor for alcohol and cannabis dependence. Journal of Psychiatric Research, 42, 230–239. doi:10.1016/j.jpsychires.2007. 01.02 Cai, L., Thissen, D., & du Toit, S. (2011). IRTPRO 4.2 [Computer software]. Lincolnwood: Scientific Software International. 10 DELLER ET AL. Cooper, M. L. (1994). Motivations for alcohol use among adolescents: Development and validation of a four-factor model. Psychological Assessment, 6, 117 128. doi:10.1037/1040-3590.6.2.117 Critchlow, B. (1986). The powers of John Barleycorn: Beliefs about the effects of alcohol on social behavior. American Psychologist, 41, 751–764. doi:10.1037/0003-066X.41.7.751 Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. doi:10.1007/BF02310555 Cuming, S., Rapee, R. M., Kemp, N., Abbot, M. J., Peters, L., & Gaston, J. E. (2009). A self-report measure of subtle avoidance and safety behaviors relevant to social anxiety: Development and psychometric properties. Journal of Anxiety Disorders, 23, 879–883. doi: 10.1016/j.janxdis.2009.05.002 Desch^enes, S. S., Dugas, M. J., Fracalanza, K., & Koerner, N. (2012). The role of anger in generalized anxiety disorder. Cognitive Behaviour Therapy, 41, 261–271. doi:1080/16506073.2012.666564 Erwin, B. A., Heimberg, R. G., Schneier, F. R., & Liebowitz, M. R. (2003). Anger experience and expression in social anxiety disorder: Pretreatment profile and predictors of attrition and response to cognitive-behavioral treatment. Behavior Therapy, 34, 331–350. doi: 10.1016/S0005-7894(03)80004-7 Garcia, A. F., Acosta, M., & Osman, A. (2016). Using a recently developed self-report instrument to assess social anxiety life interference in individuals with co-occurring depression: A known-groups analysis. Dual diagnosis (Foster City), 2, 1–4. doi:10.21767/24725048.100016 Gilles, D. M., Turk, C. L., & Fresco, D. M. (2006). Social anxiety, alcohol expectancies, and self-efficacy as predictors of heavy drinking in college students. Addictive Behaviors, 31, 388–398. doi:10.1016/ j.addbeh.2005.05.020 Grant, B. F., Hasin, D. S., Blanco, C., Stinson, F. S., Chou, S. P., Goldstein, R. B., … & Huang, B. (2005). The epidemiology of social anxiety disorder in the United States: Results from the National Epidemiologic Survey on Alcohol and Related Conditions. The Journal of Clinical Psychiatry, 66, 135–1361. doi:10.4088/ JCP.v66n1102 Gross, J. J. (2002). Emotion regulation: Affective, cognitive, and social consequences. Psychophysiology, 39, 281–291. doi:10.1017/ S0048577201393198 Ham, L. S., Bacon, A. K., Carrigan, M. H., Zamboanga, B. L., & Casner, H. G. (2016). Social anxiety and alcohol use: The role of alcohol expectancies about social outcomes. Addiction Research & Theory, 24, 9–16. doi:10.3109/16066359.2015.1036242 Ham, L. S., Zamboanga, B. L., Olthuis, J. V., Casner, H. G., & Bui, N. (2010). No fear, just relax and play: Social anxiety, alcohol expectancies, and drinking games among college students. Journal of American College Health, 58(5), 473–479. doi:10.1080/ 07448480903540531 Hoffman, S. G., Asnaani, M. A., & Hinton, D. E. (2010). Cultural aspects in social anxiety and social anxiety disorder. Depression and Anxiety, 12, 1117–1127. doi:10.1002/da.20759 Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118 Kashdan, T. B., & Collins, R. L. (2010). Social anxiety and the experience of positive emotion and anger in everyday life: An ecological momentary assessment approach. Anxiety, Stress, & Coping, 23(3), 259–272. doi:10.1080/10615800802641950 Kessler, R. C., Berglund, P., Demler, O., Jin, R., Merikangas, K. R., Walters, E. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 593–602. doi: 10.1001/archpsyc.62.6.593 Kessler, R. C., Chiu, W. T., Demler, O., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617–627. doi:10.1001/archpsyc.62.6.617 Kocovski, N. L., Endler, N. S., Rector, N. A., & Flett, G. L. (2005). Ruminative coping and post-event processing in social anxiety. Behaviour Research and Therapy, 43, 971–984. doi:10.1016/ j.brat.2004.06.015 Leary, M. R. (1983). A brief version of the Fear of Negative Evaluation Scale. Personality and Social Psychology Bulletin, 9, 371–375. doi: 10.1177/0146167283093007 Liebowitz, M. R. (1987). Social phobia. Modern Problems of Pharmacopsychiatry, 22, 141–173. doi:10.1159/000414022 Lochner, C., Mogotsi, M., du Toit, P. L., Kaminer, D., Niehaus, D. J., & Stein, D. J. (2003). Quality of life in anxiety disorders: A comparison of obsessive-compulsive disorder, social anxiety disorder, and panic disorder. Psychopathology, 36, 255–262. doi:10.1159/000073451 Marsh, H. W., Morin, A. J. S., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110. doi:10.1146/annurevclinpsy-032813-153700 Maruish, M. E., Bershadsky, B., & Goldstein, L. (1998). Reliability and validity of the SA-45: Further evidence from a primary care setting. Assessment, 5, 407–419. doi:10.1177/107319119800500410 McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates. doi:10.1111/j.2044-8317.1981. tb00621.x Modini, M., Abbott, M. J., & Hunt, C. (2015). A systematic review of the psychometric properties of trait social anxiety self-report measures. Journal of Psychopathology and Behavioral Assessment, 37, 645–662. doi:10.1037/a0020968 Moitra, E., Herbert, J. D., & Forman, E. M. (2008). Behavioral avoidance mediates the relationship between anxiety and depressive symptoms among social anxiety disorder patients. Journal of Anxiety Disorders, 22, 1205–1213. doi:10.1016/j.janxdis.2008.01.002 Moran, T. P. (2016). Anxiety and working memory capacity: A metaanalysis and narrative review. Psychological Bulletin, 142, 831–864. doi:10.1037/bul0000051 Morris, E. P., Stewart, S. H., & Ham, L. S. (2005). The relationship between social anxiety disorder and alcohol use disorders: A critical review. Clinical Psychology Review, 25, 734–760. doi:10.1016/ j.cpr.2005.05.004 Moscovitch, D. A., McCabe, R. E., Antony, M. M., Rocca, L., & Swinson, R. P. (2008). Anger experience and expression across the anxiety disorders. Depression and Anxiety, 25, 107–113. doi:10.1002/ da.20280 Muthen, L. K., & Muthen, B. O. (1998-2011). Mplus User’s Guide. (6th ed.). Los Angeles, CA: Muthen & Muthen. Okazaki, S., Liu, J. F., Longworth, S. L., & Minn, J. Y. (2002). Asian American – White American differences in expressions of social anxiety: A replication and extension. Cultural Diversity and Ethnic Minority Psychology, 8, 234–247. doi:10.1037/1099-9809.8.3.234 Osman, A., Bagge, C. L., Freedenthal, S., Gutierrez, P. M., & Emmerich, A. (2011). Development and evaluation of the Social Anxiety and Depression Life Interference—24 (SADLI-24) Inventory. Journal of Clinical Psychology, 67, 82–98. doi:10.1002/ jclp.20728 Osman, A., Freedenthal, S., Acosta, M., & Pirani, S. (2015). The Multidimensional Social Anxiety Response Inventory-21 (MSARI-21): Development and Preliminary Validation in Non-Clinical Samples. Unpublished manual. Department of Psychology, the University of Texas at San Antonio. Purdon, C., Antony, M., Monteiro, S., & Swinson, R. P. (2001). Social anxiety in college students. Anxiety Disorders, 15, 203–215. doi: 10.1016/S0887-6185(01)00059-7 Ranta, K., Kaltiala-Heino, R., Koivisto, A., Tuomisto, M. T., Pelkonen, M., & Marttunen, M. (2007). Age and gender differences in social anxiety symptoms during adolescence. The Social Phobia Inventory (SPIN) as a measure. Psychiatry Research, 153, 261–270. doi:10.1016/ j.psychres.2006.12.006 Raykov, T., & Marcoulides, G. A. (2015). A direct latent variable modeling based method for point and interval estimation of coefficient alpha. Educational and Psychological Measurement, 75, 146–156. doi: 10.1177/0013164414526039 MSARI–21 PSYCHOMETRICS Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696. doi:10.1080/ 00273171.2012.715555 Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73, 5–26. doi:10.1177/0013164412449831 Roberson-Nay, R., Strong, D. R., Nay, W. T., Beidel, D. C., & Turner, S. M. (2007). Development of an abbreviated Social Phobia and Anxiety Inventory (SPAI) using item response theory: The SPAI-23. Psychological Assessment, 19, 133–145. doi:10.1037/1040-3590.19. 1.133 Schermelleh-Engel, K., Moosbrugger, H., & M€ uller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23–74. Schry, A. R., & White, S. W. (2013). Understanding the relationship between social anxiety and alcohol use in college students: A metaanalysis. Addictive Behaviors, 38, 2690–2706. doi:10.1016/j.addbeh. 2013.06.014 Skocic, S., Jackson, H., & Hulbert, C. (2015). Beyond DSM-5: An alternative approach to assessing social anxiety disorder. Journal of Anxiety Disorders, 30, 8–15. doi:10.1016/j.janxdis.2014.006 Spielberger, C. D. (1999). Manual for the State Trait Anger Expression Inventory-2 (STAXI-2). Odessa, FL: Pyscholgoical Assessment Resources. 11 Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251. doi:10.1037//00332909.87.2.245 Turner, S. M., Beidel, D. C., Dancu, C. V., & Stanley, M. A. (1989). An empirically derived inventory to measure social fears and anxiety: The Social Phobia and Anxiety Inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 35–40. doi: 10.1037/1040-3590.1.1.35 Turner, S. M., Beidel, D. C., & Dancu, C. V. (1996). Social Phobia & Anxiety Inventory: Manual. Toronto, ON: Multi-Health Systems. Watson, D., & Friend, R. (1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448–457. doi: 10.1037/h0027806 Wittchen, H. U., & Beloch, E. (1996). The impact of social phobia on quality of life. International Clinical Psychopharmacology, 11, 15–23. doi:10.1016/S0006-3223(97)8746-X Xu, Y., Schneier, F., Heimberg, R. G., Princisvalle, K., Liebowitz, M. R., Wang, S., & Blanco, C. (2012). Gender differences in social anxiety disorder: Results from the national epidemiologic sample on alcohol and related conditions. Journal of Anxiety Disorders, 26, 12–19. doi: 10.1016/j.janxdis.2011.08.006 Zhang, Z., & Yuan, K. H. (2016). Robust coefficients alpha and omega and confidence intervals with outlying observations and missing data: methods and software. Educational and Psychological Measurement, 76, 387–411. doi:10.1177/0013164415594658