Automated Coding of Qualitative Interviews with Latent Semantic Analysis Ingo Feinerer Fridolin Wild Vienna University of Economics and Business Administration {h0125130|fridolin.wild}@wu-wien.ac.at Abstract: Coding and analysing qualitative interviews is one of several core techniques used in marketing research. Qualitative methods offer valuable information hardly gained by standard quantitative methods since open-ended questions and interviews provide deeper insight into customer demands. The main disadvantages of qualitative methods are their inherent subjectivity and their high costs. We tackle this problem by applying latent semantic analysis (LSA) in a fully automated way on transcripts of interviews and we propose two algorithms based on LSA. We evaluate the algorithms against two separate real-life cases taken from the automobile industry and from the Austrian mobile phone market. Thereby, we compare the machine results against marketing expert judgements and show that the algorithms proposed provide perfect reliability with appropriate validity in automated coding and textual analysis. 1 Introduction The proportion of qualitative methods in market and opinion research has risen continuously in the last years. Common qualitative approaches in industry target methods with a high trade-off between quality and quantity, as effective qualitative research is costly. Effective qualitative methods which can be implemented and evaluated within shorter time frames tend to predominate: in Germany, for example, the share of depth interviews decreased from 66 % in 1995 to 28 % in 2005, whereas the share of expert interviews rose from 11 % to 53 % (Arb06). At the same time, empirical qualitative research methods face the problem of inherent subjectivity when conducted manually. Typically, few researchers conduct, interpret, and analyse the latent structure in the empirical material collected, with an ever-increasing rate of errors the more interviews they have to process and with subjective differences grounding in their varying interpretations. Choosing representative and trained interviewers can circumvent this only to a limited amount. Nevertheless qualitative research has several advantages. Closed-ended questions only shift the subjectivity towards the question creation by asking along a battery of predefined items thus preventing spontaneous associations. Qualitative research therefor is and has to be an integral part of empirical research, especially used for hypothesis building. In this contribution, we propose a technique based on latent semantic analysis (LSA) that tries to reduce the costs of analysis by providing an automation of analysis substeps and tries to reduce the subjectivity problem inherent to manual analysis. To be more precise, we investigate, how LSA can be applied to analyse the transcripts of interviews in order to automate the coding of interview transcripts into higher level concepts to be investigated. With an automated coding instrument at hand, transcripts can be investigated in more depth, as exploratory examination can be performed with significantly lower efforts compared to manual coding. We therefore will present two algorithms with which we try to extract brand associations automatically from the interview texts which we then will evaluate against results generated by human coding experts. The rest of this paper is structured as follows. First, we present the methodology applied in this contribution, thereby describing the data-sets used and the overall approach deployed for evaluating reliability and validity of the algorithms we propose. Second, we describe the two algorithms against the background of the latent semantic analysis theory when used to extract brand associations. Third, we report on the results of the automated analysis and discuss the findings in relation to their validity and reliability. In the last section, we conclude with a review on the applicability of the proposed method and identify research problems to be worked on in future investigations. 2 Methodology In order to evaluate the applicability of the two algorithms we propose that base on LSA, i.e. the headcount and the termcount algorithm, we deploy to data-sets collected from qualitative interviews that were conducted manually by marketing experts. One of these data-sets stems from a study conducted in Austria about brands of the Austrian mobile phone market. These interviews were conducted and analysed by hand through Marketmind, an Austrian-based marketing company. The other dataset was originally contributed by (SSOF98) and was conducted in the automobile sector investigating brand relationsships in the US and in Germany. The mobile phone data-set consists of 969 interviews searching for associations to five mobile phone companies based in Austria. The interview questions aimed at activating brand associations, e.g., “Which image do you perceive if you consider brand X?” or “Please imagine brand Z. What do you associate?”. Each interview consisted of up to 10 answers, mainly short sentences or phrases. Both the questions and answers are in German. The five mobile phone companies are A1 consisting of 134 interviews, T-Mobile consisting of 125 interviews, One consisting of 120 interviews, Tele.ring consisting of 122 interviews and Drei consisting of 468 interviews. The second dataset we will consider consists of 24 in depth interviews on brand perception of Mercedes in USA and Germany. Each interview is composed of 64 questions to trigger emotions and associations for the brand Mercedes, e.g., “If I buy a Mercedes I have a good feeling because . . . ” “Please characterise a typical Mercedes driver!” All answers are available in German. The answers to all 64 questions in one interview have lengths between about 3500 and about 11500 words, most interviews float around a length of 4000 words. The transcripts contain both questions and answers. We stripped the questions and extracted the subset of answers fitting to questions related to the brand Mercedes. To evaluate the effectiveness of the algorithms we propose in this contribution, we investigate their reliability and validity. Concerning reliability (irrelevant whether inter-rater-, test-retest- or parallel-formsreliability) the LSA method has a very high reliability by construction. The LSA procedure is clearly defined in the literature, and is deterministically implemented. This means the LSA methods yields exactly the same results every time for the same set of parameters. This means our approach can help to deal with reliability problems as presented by (RLR+ 05): they explain how brand attitude data which is typically collected through free choice attitude questions lacks reliability. Of course we cannot control what interviewees respond but we can guarantee that no subjectivity is introduced in the analysis of the transcripts. The second aspect is validity. We perform external validity tests by checking how similar our automatically produced results are to results aggregated manually by human marketing experts with classical qualitative methods. Since the human scores are not perfectly reliable (you may find an interesting discussion on intercoder reliability by (LSDB02)) it is clear that we cannot expect perfect external validity when comparing to human scores, as humans among themselves also do not perfectly correlate in their judgements. We expect the algorithms to provide near-human results with high correlations in order to provide evidence to be valid. 3 Algorithms Applying Latent Semantic Analysis (LSA) for text-mining has a long-standing tradition since its invention in the late eighties of the last century. Dumais provides a comprehensive overview on LSA and its various application settings including = Figure 1: Singular Value Decomposition. = Figure 2: Truncated SVD. applications for assessing semantic relations in human memory based on text collections (Dum03). The basic idea behind LSA is that texts contain a semantic structure which, however, is obscured by variations in the wording, and that this structure of meaning can be (partially) unveiled by calculating conceptual indices derived through a truncated singular value decomposition. Comparing both texts and terms on the basis of the resulting lower-dimensional space thereby form the basic working principle of analytic applications using LSA (for a tutorial cf. (WS06)). The various LSA application processes share a common core: typically first a document-term matrix M (a so-called ‘textmatrix’) is constructed from a given collection of texts denoting the number of occurrences of a specific term (the rows) for each document (the columns). Using a singular value decomposition (SVD), this matrix is decomposed into three partial matrices (the so-called ‘latent-semantic space’, see Figure 1). Thereby T constitutes the left-singular vectors, D the right-singular vectors being both orthonormal and a diagonal matrix S which contains the singular values, so that M = T SD T . By truncating these matrices to a relatively small number of singular values, the document-term matrix M can be restructured in such a way that it reflects only those k common dimensions that account for the greatest share of its underlying variance (cf. (VBRGK06)): Mk = Tk Sk DkT (see Figure 2). In (WSSN05), the authors investigate recommendations on how to identify an ideal number of singular values k to be kept. To keep additional documents from influencing a previously calculated latent-semantic space and to reduce computational costs, new texts can be folded-in after the singular value decomposition (cf. (WS06)): first, a textmatrix V of the same format as M is constructed from the texts. Second, the textmatrix is mapped into the existing factor structure and subsequently reflected back to its textmatrix format: V ′ = Tk Sk (V T Tk Sk−1 ), see = Figure 3: Folding-in of additional texts. Figure 3. Based on this general LSA framework, we tested several different methods of how to extract brand associations which mainly differed in how the input corpus is compiled from the available material and how similarities between brand names, constructs, and interview transcripts are measured. The following section will describe the two algorithms that emerged from our experiments in detail. The first algorithm works on data-sets that contain only responses to one question. In such a data-set, the answers are extracted person by person, each statement in a different file. For the headcount method, the data-set is split into subsets separating the responses by brands and per person: each file contains only the response of one interviewee to one particular brand name (which can be filtered easily from the transcripts if the questions and answers are ordered according to the different brands). One data-set thus only contains the answers provided by the interviewees to one specific brand. For the second approach, especially for larger texts, the interviews are split into questions and answers, each of them written into a separate text file, additionally separated by interviewees. As the interviews are significantly greater in length compared to the input seed-terms from the textual representatives of the coding scheme, the headcount algorithm cannot be easily applied to such a corpus: The correlation between any document and the pseudo-document of the seed-terms has to be expected to be rather low as the documents cover a greater domain and thus are not as sparse as the pseudo-document of the seed-terms. Therefore, we developed a second algorithm, which we call termcount, that deploys the full-transcripts of the larger interviews and uses the similarity between two pseudo-document vectors created by folding the brand name and the seed-terms of the corresponding construct into the latent-semantic space constructed over the interview transcript documents. 3.1 Algorithm 1: Headcount A vector space V n can be defined containing vectors of term frequencies within R (see Equation 1). The term frequencies fn can be real numbers as a textvector can also be weighted using common termweighting measures such as inverse document frequencies. V n = {(f1 , f2 , . . . , fn ) | f1 , f2 , . . . , fn ∈ R} (1) n Within this vector space V a multiset M can be declared which contains the document collection, also called corpus (Equation 2). This multiset M can also be represented in matrix format, thus giving a textmatrix as described above. M = {d | d ∈ V n } (2) The multiset M has to satisfy the condition that for every two elements of the set, there is a term label function t which yields the same character-set (Equation 3). ∀v, w ∈ M : t(v) = t(w) (3) The term label function relates every vector element of the vectors within M with its corresponding term label. This multiset M is generated by a) constructing a textmatrix over the input corpus which contains the response of exactly one interviewee per document, b) erecting a latent-semantic space over this matrix, and c) converting this latent-semantic space back to a textmatrix representation format to allow for the more convenient application of operations (these steps a–c are described in more detail in Section 3 above). This output textmatrix contains now a representation of the training documents as termvectors again and is identical with the multiset M as defined above in Equation 2. Stopwords (i.e., highly frequent terms like ‘she’, ‘a’ and ‘was’) are filtered from these texts, the minimum word length is set to two, thus excluding all terms with less than two characters. The documents are read into memory linearly. After they have been read into memory, however, they are converted to a bag of words textmatrix representation, where the order of terms is no longer of importance. The number of first k singular values to be selected for the truncation is calculated as described in (WSSN05). A coding scheme CS is a set of constructs to be investigated, each construct containing seed-terms representing the construct (Equations 4, 5): CS = {c | c ∈ V n } (4) ∀c, q ∈ CS × M : t(c) = t(q) (5) such that This c is created by folding the terms from the text representation of the coding scheme into the latent-semantic space, thereby generating a (pseudo-)document vector of the same format as the set M . The headcount algorithm hc compares every element c of a coding scheme prepared in advance by the analysts with every existing document d in M and maps this comparison to a natural number (Equation 6). hc : M × c 7→ N0 (6) Again c and d satisfy the condition that they have the same format, as requested in Equation 7. ∀d ∈ M : t(d) = t(c) (7) This natural number indicates the percent share of how many persons (compared to the overall number of interviewees = documents) have mentioned terms strongly associated to the seed-terms contained in the particular coding scheme element. hc = {d | (cor (c, d) > l) ∧ (d ∈ M )} · 100 M (8) Therefore, simply the cardinality of the set created by filtering those elements from M that correlate with the pseudo-document containing the seed-terms for the coding scheme element has to be divided by the cardinality of M (see Equation 8). For more convenient interpretation by providing natural numbers as output, this fraction was normalised to the number of documents (=number of interviewees) and converted to percentages. The output of hc thus represents the percentage of the share of persons. The threshold used as limit l in our experiments was found out to yield best results when set to l = .5, which is also in the range of good correlation of the correlation measure applied. In our case, we used Pearson’s product moment correlation coefficient, other measures, however, are valid as well and produce similar results. To evaluate the validity of the headcount algorithm, we compare the machine extracted association strengths with the previously established human classifications (external validation). See Section 4 for the results. We call this method ‘headcount’, as it basically analyses the transcripts on a personby-person basis. 3.2 Algorithm 2: Termcount The second algorithm differs in so far as the association strength is measured directly via similarities in the textmatrix. A pseudo-document containing the brand name is compared against a pseudo-document containing the constituting seedterms of a coding construct. Seed terms thereby are usually very prominent terms able to stipulate associations of the concept under investigation. For example, a coding construct ‘security’ could be composed of the term set (‘secure’, ‘safe’, ‘stability’). We call this second method ‘termcount’, as it directly measures association strengths between the brand name and a seed-vocabulary. The document vectors c and b represent pseudo-document vectors folded into the existing space, b containing a frequency of 1 at the position of the term of the brand name (e.g., ‘Mercedes’) and c containing frequencies of 1 at the positions of the terms representing the variable under investigation (e.g., (‘secure′ = 1) ∧ (‘save′ = 1) ∧ ‘stability ′, see Equation 9). c ∈ V n, b ∈ V n (9) Again c and b satisfy the condition that they have the same format as all existing documents d, as requested in Equation 10. ∀b, c, d ∈ M : t(b) = t(c) = t(d) (10) The termcount algorithm uses the folded-in pseudo-documents for the construct constituting seed-terms and the brand name to map every tuple of them to a real number, indicating the association strength (see Equation 11). tc : c × b 7→ R (11) The association strength is defined to be the similarity of the two vectors, measured with Pearson’s correlation coefficient (Equation 12). Other similarity measures apply as well and yield similar results. tc = cor (c, b) (12) We used the ‘lsa’ package for the (statistical) programming language and computing environment R in it’s current version 0.58 (Wil06) for the basic LSA operations. The textual preprocessing was conducted with the ‘tm’ R package (Fei06) in version 0.1. The two presented algorithms have been separately implemented and were used in the following experiments. The implementations are GPL licenced and can be requested from the authors. 4 Evaluation of the Algorithms: Analysis and Discussion 4.1 Discussion of Mobile Phone Market Results Table 1 shows the correlations between the scores of our automated latent semantic analysis and the scores by human marketing experts (Mar06) for the Austrian mobile phone market dataset. In detail the table presents several parameter combinations: Brand: denotes a mobile phone company, D: a fraction of the sum of the selected singular values to the sum of all singular values. I.e., D is a shorthand for the dimension share, T : gives the threshold which correlations between two terms in the LSA space must succeed to be considered relevant, ρ: the value of Spearman’s rho statistic, and p-value: its corresponding p-value. Brand A1 One T-Mobile Drei Tele Ring A1 One T-Mobile Drei Tele Ring A1 One T-Mobile Drei Tele Ring D 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.3 0.3 0.3 0.3 0.3 T 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 ρ 0.52 0.51 0.40 0.37 0.36 0.61 0.59 0.35 0.46 0.45 0.55 0.52 0.31 0.44 0.37 p-value 0.000 0.000 0.004 0.008 0.012 0.000 0.000 0.014 0.001 0.001 0.000 0.000 0.030 0.001 0.009 Table 1: Correlations between LSA and human scores in the Austrian mobile phone market. The results of Table 1 indicate that LSA with ‘headcount’ (cp. section 3.1) is a viable method for analysing open-ended questions automatically. In fact virtually all parameter ranges and combinations (of dimension share D and threshold T ) unveil correlations between the LSA headcount method and the human reference group above 0.3 on a highly significant level. It is especially notable that the LSA headcount method shows the best results for default parameter settings, i.e., both dimension share and threshold with 0.5. With this settings we obtain a maximal correlation of the LSA scores with human scores of 0.62 at a significance level of 4.087 ∗ 10−6 for the brand A1. We receive similar results for One with correlation 0.6 at significance 9.188 ∗ 10 −6 . Both results’ importance is underlined by the fact that typically the correlation between two human groups’ scores (i.e., the intercoder reliability) also floats around at 0.7: “The criterion of .70 is often used for exploratory research.” (LSDB02, page 593). For the two brands Drei and Tele Ring we calculate correlations at almost 0.5, where T-Mobile’s result stays back at 0.35. This lower value can be explained with the small set of open-ended questions for this brand (n = 125). 4.2 Discussion of Mercedes Results Figure 4 shows free associations for the brand Mercedes with human scores and in comparison thereto the machine assigned scores created with the proposed termcount algorithm. The values correlate with Spearman’s rho statistic with ρ = .51 Machine Human innovativ spezif. Modelle Herkunftsland konservativ gutes Design Sportlichkeit gute Marke Luxus Sicherheit hoher Preis Komfort soziales Ansehen Hohe Qualität −20 0 20 40 60 80 Association Strength Figure 4: Free Associations for Mercedes scored with termcount in comparison to the humans coding. with a significance of p-value= .07. Differently than the human scores which represent the share of people who used terms aggregated in the corresponding coding construct in their responses, termcount is the percentage of the Pearson correlation. Therefore, it can also take negative values. The outlier at ‘Sicherheit’ (engl. ‘security’) can be explained as follows: obviously the seed-terms used were not sufficient enough, compared to what the human coders had in mind when they were coding the interviews. 5 Conclusion We have presented an approach based on latent semantic analysis for investigating and coding qualitative texts which tackles the main problems of qualitative research methods: high costs and inherent subjectivity. Since our method works automatically it can contribute to cost reduction as it speeds up the coding process significantly. Subjectivity is fundamentally reduced as the algorithms are well de- fined and provide perfect reproducibility of results. In other words the LSA method is highly reliable. Besides reliability validity is of great interest because we expect the method to deliver near human results. The main consideration is that a high matching between human and machine judgements is sufficient but not neccessary (since humans are error-prone). Instead we can guarantee reliability on a very high level with reasonable validity. Concerning validity it is clear that the coding strategy has to be adapted to the considered methods: in the human coding, labels should be the most concise, in the automated coding the most significant. In this respect we conclude from the results that the two algorithms suggest that LSA per se can be deployed for conducting stages of the qualitative research process and that they are viable and powerful analysis methods for increasing the quality: LSA offers highest reliability combined with appropriate validity. We propose that the headcount method is applied to larger corpora containing smaller, statements-sized interviews as in the case for the corpus provided by Marketmind. The termcount method should be applied with smaller corpora containing lengthy, in-depth interviews, as it provides a solution to work with the latentsemantic structure mined from these documents without directly referencing to the underlying documents needed to construct this latent-semantic space. In our future work, we intend to investigate more closely, how correlations in the modified vector space can be better interpreted to directly reflect human semantic association strengths. We see at least two possible starting points here: at first relate association pairs to each other to evaluate associations against other brands, against its corresponding antonym, or in comparison to other highly associated terms. The second option we see, is to separate domain-specific frequency distributions from the generic vocabulary use by identifying significant differences and to work with these significant differences. 6 Acknowledgements We would like to thank Sonja Ehrenberger and Wolfgang Rejzlik (both Marketmind Austria) for providing us with up-to-date industry market data and for giving us feedback on the methodological background of this paper. Furthermore, we would like to thank Kurt Hornik and David Mayer from the Vienna University of Economics and Business Administration and Andreas Strebinger from the York University for their invaluable feedback in various discussions. References [Arb06] Arbeitskreis Deutscher Markt- und Sozialforschungsinstitute e.V. Proportion of qualitative interviews by ADM members. December 2006. [Online; accessed 21-December-2006]. [Dum03] Susan Dumais. Latent Semantic Analysis. Annual Review of Information Science and Technology, 38:189–230, 2003. [Fei06] Ingo Feinerer. tm: Text Mining Package, 2006. R package version 0.1. [LSDB02] Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability. Human Communication Research, 28(4):587–604, October 2002. [Mar06] Marketmind. Qualitative interviews in the Austrian mobile phone market. Technical report, Marketmind Austria, 2006. [RLR+ 05] Cam Rungie, Gilles Laurent, Francesca Dall’Olmo Riley, Donald Morrison, and Tirthankar Roy. Measuring and modeling the (limited) reliability of free choice attitude questions. International Journal of Research in Marketing, 22:309–318, 2005. [SSOF98] Günter Schweiger, Andreas Strebinger, Thomas Otter, and Gereon Friederes. Qualitative Untersuchung der Marke Mercedes-Benz in vier Märkten unter Berücksichtigung von PKW und Nutzfahrzeugen. Technical report, Vienna University of Economics and Business Administration, Advertising and Market Research Institute, February 1998. [VBRGK06] Jan Van Bruggen, Ellen Rusman, Bas Giesbers, and Rob Koper. Latent Semantic Analysis of Small-Scale Corpora for Positioning in Learning Networks. Technical report, Open University Netherlands, November 2006. [Wil06] Fridolin Wild. lsa: Latent Semantic Analysis, 2006. R package version 0.58. [WS06] F. Wild and C. Stahl. Investigating Unstructured Texts with Latent Semantic Analysis. In Lenz and Decker, editors, Advances in Data Analysis, Berlin, 2006. Springer. [WSSN05] F. Wild, C. Stahl, G. Stermsek, and G. Neumann. Parameters Driving Effectiveness of Automated Essay Scoring with LSA. In Myles Danson, editor, Proceedings of the 9th International Computer Assisted Assessment Conference (CAA), pages 485–494, Loughborough, UK, July 2005. Professional Development.