Multimodal Russian Corpus (MURCO): Studying Emotions Elena Grishina Institute of Russian Language RAS 18/2 Volkhonka st., Moscow, Russia rudi2007@yandex.ru Abstract The paper introduces the Multimodal Russian Corpus (MURCO), which has been created in the framework of the Russian National Corpus (RNC). The MURCO provides the users with the great amount of phonetic, orthoepic, intonational information related to Russian. Moreover, the deeply annotated part of the MURCO contains the data concerning Russian gesticulation, speech act system, types of vocal gestures and interjections in Russian, and so on. The access to The Corpus is free. The paper describes the main types of annotation and the interface structure of the MURCO. What is MURCO? The Multimodal Russian Corpus (MURCO) is the new project in the framework of the Russian National Corpus (RNC, www.ruscorpora.ru). The pilot version of the MURCO has been open for general access since April, 2010. Since the project was described in my papers (Grishina, 2009a; Grishina, 2009b; Grishina, 2010a), I don’t intend to describe the MURCO at great length. I am planning to outline the Corpus and to present its perspectives in relation to the study of emotions. The MURCO is the result of the further development of the Spoken Sobcorpus of the RNC. The Spoken Subcorpus includes circa 8.5 million tokens and contains the texts of 3 types: public spoken Russian, private spoken Russian, and movie speech (the volume of the last is circa 4 million tokens). The Spoken Subcorpus does not include the oral speech proper; it includes only transcripts of the spoken texts (Grishina 2006). To improve it and to supplement its searching capacity, we have decided to supply the transcripts with the sound and video tracks. To avoid the problem related to the copyright offence and the privacy invasion we have used the cinematographic material in the MURCO. Naturally, in the future we are also going to include in the MURCO the patterns of the public and private spoken Russian, but the cinematographic Russian is the most appropriate material to begin the project with. It should be mentioned inter alia that the usage of the cinematographic material to elaborate and test the annotation system of the pioneering corpus is far more promising than the usage of the “natural” (public or private) spoken Russian. The main reason for it is the fact that movies include exceptionally manifold set of situations, and this situational variety results in the linguistic variety. Therefore, to annotate the movie Russian we need greater number of definitions and more elaborated system of concepts than to annotate the “real-life” Russian. In other words, the exercised annotation of the movie Russian will be useful for the mark-up of the “natural” Russian, but the opposite is not correct. The MURCO is the collection of the clixts. A clixt is the pair of a clip and the corresponding text (i.e. the corresponding part of a movie transcript). It is supposed that a user will have the opportunity to download not only the text component of a clixt (=marked up transcript), but also its sound/video component, so after downloading a user may employ any program to analyze it. The duration of a clip is within the interval of 5-20 sec. As we have mentioned above, the total volume of the movie subcorpus is about 4 million tokens. This token volume corresponds to circa 300 hours of sound- and video track. Therefore, being fulfilled the MURCO presents one of the largest open multimodal sources. Data Validity The first question which occurs when we discuss the usage of the cinematographic data in linguistic researches is the question of their validity. The standard attitude to the movie speech considers it to be “written to be spoken” and “artificial” (Sinclair, 2004). The closer look at the problem, however, shows that the discrepancies between the spontaneous “real-life” spoken language and the movie speech are not crucial and deal mainly with the higher/lower degree of text coherence. The other spoken features, first of all the set and the structure of discourse markers (lexical, morphological, and syntactical), the intonation patterns and the accompanying gesticulation are similar in the cinematographic and real face-to-face communications (Grishina, 2007a; Grishina, 2007b; Forchini, 2009). As for the emotion studies, it is well known that one of the generally accepted ways of data acquisition here is to get the actors who are supposed to perform the desired mise-en-scènes involved in the investigation process. In a very instructive paper (Busso & Narayanan, 2008) the authors have shown distinctly that the reason of the main drawbacks concerning this way of the data accessing roots is the wrong methodology. As for the authors’ opinion, two main shortcomings are as follows: the lack of acting technique and the lack of contextualization of the uttered phases. It is absolutely obvious that the cinematographic data are free from these drawbacks. Types of Annotation in MURCO We should emphasize the fact that the MURCO is not intended for studying emotions and affections: it has been designed as the specialized resource oriented basically at standard linguistic researches. However, this resource may be used efficiently when we study emotions and affections in the light of their linguistic manifestation. In my paper I will try to number the ways to obtain emotional information from the MURCO. 3.1 Via Word to Emotion The MURCO is marked up from different points of view (Grishina, 2010a). Some types of annotation are standard RNC types of annotation (metatextual, morphological, semantic annotation), some types are special for the spoken component of the RNC and, naturally, are preserved in the MURCO (sociological and accentological annotation), and, finally, some of the mark-up dimensions are specific only for the MURCO (orthoepic, speech act and gesture annotation). We do not plan to describe all these types of annotation in detail (it has been done in our early papers); we only want to illustrate the annotation zones, which can be useful when we study emotions and affections. It is obvious that the simplest way to obtain the emotion data from the MURCO is the word entry to the Corpus. We can find the data using the special morphological form of a word. For example, we may form the following query: “verb nenavidet’ (to hate), 1st person, Present, Singular”, i.e. ‘I hate smb or smth’. This query gives us the possibility to obtain clixts, in which different characters express their hatred of smd/smth in actual mode. To widen the scope of the data we can also use the set of synonyms, for example, “otvratitel’nyj ‘disgusting’, gadkij ‘nasty’, gnusnyj ‘abominable’, merzkij ‘loathsome’, omerzitel’nyj ‘sickening’, pakostnyj ‘mean, foul’”, and some others. This synonymic row lets us receive all possible types of pronouncing of these adjectives, so we can analyze the phonetic, intonation and gesture characteristics of the adjectives of disgust in Russian. Moreover, the morphological annotation in the MURCO makes it possible to distinguish the usage of these adjectives as attributes and as nominal predicates; this distinction is very important in Russian. It seems that the adjectives-attributes are used to describe the speaker’s emotions, while the adjectives-predicates express them directly. Finally, to collect clixts with the preselected emotions we may form the request which consists of emotionally tinged syntactic constructions. The example of such constructions in Russian is the word-combination chto za Xnom! ‘What X!’ This construction may express both high appraisal and condemnation (so, Chto za pogoda! ‘What weather!’ may mean both ‘good weather’ and ‘bad one’). Obviously, it may be very useful for any investigator of emotions to study the discrepancies between the opposite emotional colors of the same syntactic construction. It should be mentioned in conclusion that the MURCO annotation makes it possible to form the subcorpora of 1) the masculine and feminine cues; 2) the cues which belong to the actors of certain age; 3) the cues which were pronounced by certain actor. Therefore, we may try to arrive to the conclusions concerning sociology of emotions. 3.2 Via Speech Act to Emotion The clixts in the MURCO receive the annotation of 2 types: 1) obligatory annotation (metatextual, morphological, semantic, sociological, accentological, orthoepic), and 2) optional annotation (speech act and gesture). The last one is optional by reason of its labour-intensiveness and impossibility to automate the mark-up process. Therefore, the clixts, which are annotated from the both points of view, form the subset of the MURCO. This subcollection is named ‘the deeply annotated subcorpus’ (DA-MURCO). The volume of the DA-MURCO is supposed to be circa 0.5 million tokens, or 40 hours of phonation. The speech act annotation in the DA-MURCO covers 5 thematic areas: the mark-up of 1) speech acts proper, 2) the types of repetitions, 3) the speech manner, and 4) the types of vocal gestures, non-verbal words and interjections. 3.2.1. Speech Act Annotation The list of the Russian speech acts includes about 150 items, grouped into 13 types: Address or call, Agreement, Assertion, Citation, Complimentary, Critical utterance, Etiquette formula, Imperative, Joke, Modal utterance or performative, Negation, Question, Trade utterance. Most of these types are ambivalent from the point of view of emotional characteristics, that is they include both emotionally colored speech acts (e.g. to monkeyemotion and to quoteneutral in the type Citation, to ask smb to do smthneutral, to adviseneutral, to orderneutral and to demandemotion or to supplicateemotion in the type Imperative, and so on). Some types, however, are characterized with the inherent emotionality, that is all speech acts, which are affiliated to these types, are emotionally colored (Critical Utterances, Complimentary, Joke). Naturally, to study emotions both emotionally colored types of speech acts and ambivalent ones are useful. As for the emotionally colored types, it is quite obvious. In respect to the ambivalent types of speech acts, it is very interesting, for example, to define whether the neutral and the emotional speech acts preserve their neutrality and emotionality in different types of situations or not. Therefore, the speech act annotation makes it possible to request clixts, which contain this or that type of speech acts, no matter what their lexical, morphological and syntactic structures are. 3.2.2. Repetitions It is well known that the repetitions are of great importance in the spoken speech and are directly connected with the expression of emotions. Therefore, various types of repetitions, which are marked up in the DA-MURCO, may become very useful for the researchers of emotions. Three types of repetitions are marked up in the DA-MURCO. Firstly, we may obtain the clixts which contain the repetitions of different structure and try to analyze their possible connection with the emotionality. The structural types of repetitions are as follows: a) one-word vs. many-word repetitions b) envelope repetitions: Beregi ruku Sen’a beregi Be careful with your hand Sen’a be careful c) baton repetitions Ja ne trus no ja bojus’. Bojus’ smogu li ja I am not coward but I’m afraid. I’m afraid I can’t do it d) permutation repetitions Tak chto delat’ budem? Chto budem delat’? Well what to do we ought? What we ought to do? Well, what we ought to do? Secondly, we may request the clixts which include the repetitions of different communicative structure, i.e. the repetitions which are distributed between the participants of a communicative act: a) echo repetitions 1st: On hochet chtoby ty sygral Lenina v jego spektakle. 1st: He wants you to play Lenin in his performance. 2nd: Gad! Lenina! 2nd: Son of a bitch! Lenin! b) change of addressee (to 1st) Vot usy vam vylityj Volod’ka! (to 2nd) Vylityj! (to 1st) If you have the moustache you’ll be the spitting image of Volod’ka! (to 2nd) The spitting image! c) forwarding repetition 1st (to audience) Pobeditel’nicej stala… 2nd (to 1st) Minutochku. 1st (to audience) Minutochku! 1st (to audience) The winner is… 2nd (to 1st) Just a minute. 1st (to audience) Just a minute! d) overinterrogations 1st: Eto kajuta shestnadcataja? 2nd: Shestnadcataja. 1st: Is it the cabin number sixteen? 2nd: Sixteen. 1st: Gde etot bol’noj? 2nd: Etot? Tam. 1st: Where is this patient? 2nd: This one? There. And, finally, the repetitions may be of different intensity and emotional color. a) single vs. multiple repetitions b) repetitions with intensifiers: Blagodarim vas za vashu interesnuju ochen’ interesnuju lekciju Thank you for the interesting very interesting lecture c) monkey repetitions: Nu vot voshla kak rodnaja. a ty: “ne vojd’ot, ne vojd’ot!” Here we are it’s gone in OK. You was wrong saying: “it will hardly go in! it will hardly go in!” Apparently, the consistent description and manifold set of the repetitions appear to be very useful not only while studying the specificity of the spoken Russian, but also in emotion studies. 3.2.3. Speech Manner The speech manner annotation in the DA-MURCO describes the speech acts from the point of view of 1) speech tempo (quick speech, chanting/scanning, declamation, speech with extra pauses), 2) speech volume (whisper, loud shout, muffled shout), 3) emotional color (crying, laughing). Obviously, the last two positions are promising from the point of view of emotion studies. 3.2.4. Non-Verbal Words and Interjections It is quite evident that non-verbal words and interjections are the most striking and direct illustrations of the emotional state of a speaker. Therefore, it is quite possible to obtain the information concerning emotions from the MURCO just making the query which contains this or that interjection or non-verbal word. The problem, however, lies in the fact that non-verbal words and interjections are polysemic, so the direct query Ah or Oh will give us the great number of varied contexts and we will be forced to do a big chunk of work to distinguish 1) emotionally colored and neutral interjections and 2) interjections expressing different emotions. The recent investigation of the Russian vocal gestures Ah and Oh (Grishina, 2009c; Grishina, 2010b) shows that these units have three types of usage. Ah/Oh as exclamations. The exclamation Oh means ‘suppressed pain’, ‘smth unpleasant’, ‘exercise stress’, ‘intensity of feelings’; Oh is characterized with descending tone and throaty phonation. The exclamation Ah means ‘uninhibited pain’, ‘unexpectedness’, ‘fright’, ‘horror’; Ah is also characterized with descending tone and throaty phonation, but sometimes the tone of the Ah in the sense ‘horror’ is high and static. Ah/Oh as interjections. The interjection Oh means basically ‘surprise’ and has also some derived meanings, e.g. ‘high appraisal’, ‘sneer’, and others. The interjection Ah means basically ‘realization’ or ‘comprehension’ and has also some derived meanings, e.g. ‘intellectual satisfaction’, ‘recognition.’ Both Ah and Oh are characterized with the ascending-descending, or undulatory tone. Ah/Oh as particles. The particle Oh is the deictic one and means ‘indication’, ‘object fixing’, and some others. The particle Oh is characterized with even ascending or even descending tone, and also with the glottal stop at the beginning of its phonation. The particle Ah has 2 main meanings: a) Ah as a interrogative particle basically means ‘question’, ‘echo-question’, ‘answer to address’ and is characterized with the ascending tone; b) Ah as a negative particle means ‘disregard’ ‘annoyance’ and is characterized with the descending tone. So, we can see that the straightforward lexical query of non-verbal words and interjections gives us too heterogeneous data to deal with. Therefore, in cases when we plan to investigate relatively frequent phenomena of the kind it would be more convenient to use the DA-MURCO, where all non-verbal words and interjections are described from the point of view of their contextual meaning. 3.2 Via Gesture to Emotion It is the wide-spread opinion that the gesticulation (including the facial expressions) along with the intonation and the phonetic indicators are the main and principal media to convey the emotional information. The MURCO seems to be the resource which is generally accessible and quite considerable in terms of its volume; moreover, the MURCO includes a lot of video tracks. So, the existing gesture annotation in the DA-MURCO (see in detail (Grishina, 2010)) ought to supply a user with the shortest ways to the required emotion information obtained by means of gesticulation. It should be specially mentioned that the gesture annotation makes it possible to investigate the emotions of the silent participants of a communication act, in contrast to the other linguistic units (phonetic, intonation, lexical, grammatical). Any gesture in the DA-MURCO is supplied with 2 types of characteristics: 1) objective, which describes a gesture from the point of view of active/passive organs, their orientation relative to the speaker’s body and their movement directions; 2) subjective. The last ones are the triads of gesture type, gesture meaning, and gesture name. Till the moment we have marked out about 250 gesture meanings, which are grouped into 14 gesture types. To designate these gesture meanings more than 400 gesture names are used. These gesture names are the natural Russian words and word combinations, which describe Russian gesticulation and facial expressions. The gesture types are as follows: Adopted, Conventional, Corporate, Critical, Decorative, Deictic, Etiquette gestures, Gestures – speech acts, Gestures of inner state, Iconic, Physiological, Regulating, Rhetorical, Searching gestures. Among these 14 types of gestures the gestures of inner state and critical gestures are directly connected with the emotion studies, so we would like to sketch them in. The group of the gestures of inner state includes the following gesture meaning Every gesture meaning is supplied in the Tables 1-2 with one or two gesture names. Naturally, we have no possibilities to publish the complete list of gestures.: Gesture meaning Gesture name admiration move back from smth, sidelong glance affectation stick one's little finger out affection stroke smb’s head; embrace smb alienation shrug one’s shoulders; lift one's hands anticipation lick one's lips anxiety one’s hand to one’s lips apprehension glance back archness lift one’s eyebrow; sidelong glance boredom beat a tattoo; lean one’s head on one's arm caution shield oneself; glance back chagrin shake one’s head; lean one’s head on one's arm concentration close one’s eyes; lick one's lips confidence nod; cross one’s legs coquetry crinkle one's nose; blow kisses to smb defiance brush aside; screw up one's face despair grasp one’s head; strike one's hand on smth disappointment lift one’s eyebrows; press one's lips together disgust screw up one's face displeasure turn away; throw smth off distrust shake one’s head; blink embarrassment close one’s eyes; shake one’s head; turn away enervation toss smth up; lean against a wall friendliness stroke smb’s shoulder; embrace smb fright close one's eyes tight; recoil frolic throw back one's head; give smb a flick on the nose grief grasp one’s head; bite one’s lips guess cover one’s mouth; grasp one’s head high appraisal lift one’s eyebrows; shake one’s head; throw up one's hands impatience stamp one's foot indignation throw up one's hands; open one's eyes wide interest lift one’s eyebrow; bend forward irritability roll up one's eyes; close one’s eyes joy embrace smb; clap one's hands lust eye smb from head to foot; close one’s eyes meditation beat a tattoo; bite one’s lips nervousness with one’s hands to one’s heart; beat a tattoo perplexity look round the interlocutors; touch smth pride throw back one's head; stalk along relief close one’s eyes resolution pull one's hat over one's eyes; roll up one's sleeves sadness bend one's head on one's breast satisfaction nod; hang of the head on one side seducing touch smb; stroke one’s throat shame close one’s eyes; close one’s face with one’s hands shock look round the interlocutors; open one's eyes wide solidarity take smb by the hand; wink towards smb sudden recollection cover one’s mouth; knock one’s head; dive a jump surprise clasp one’s hands; open one’s mouth veneration kneel waiting lean against smth; to fold one' s arms willingness straighten one's tie; rub one’s hands Table 1. The group of the critical gestures includes the following gesture meaning: Gesture meaning Gesture name you are a fool! give the screw-loose sign; shake one’s hand criticism shake one's finger; shake one’s head mocking shake one’s head from side to side who cares! spread out one's arms Table 2. We can see that the gesture annotation in the DA-MURCO gives a user the data for the multidirectional emotion researches. Firstly, we can study a certain gesture as a medium of different emotions (e.g. the gesture to beat a tattoo may express boredom, meditation, nervousness, alienation, waiting, and the search for necessary word – one of the searching gestures) and try to define the significant discrepancies between different ways of realization of the same gesture in different emotional situations. On the other hand, we can analyze the cluster of different gestures which correlate with the same emotion. For example, an embarrassment may be expressed, among others, by means of the following gestures: to hunch, to close one’s eyes, to turn away from smb, to look away, to drop one’s eyes, to shade one's face with one's hand, to cover one’s mouth, to bend one's head. All these gestures have the same dominant: a confused person tries to attract a little attention, to become unobtrusive, and to achieve this goal he leaves the communicative zone and breaks the visual and speech contact with his interlocutor. In conclusion, I’d like to mention that we can combine the speech act queries with the gestures ones, and this combination lets us obtain the clixts which may contain not only required gesture and required emotion, but also this or that speech act, interjection, non-verbal word, type of repetitions and speech manner. This potential extends our ability to study emotions to a considerable degree. And if we keep in mind the relative universality of emotions and affects, it will be possible to take advantage of the MURCO as a whole and of the DA-MURCO particularly not only by Russian speakers, but also by the users who do not speak Russian. Acknowledgements The work of the MURCO group is supported by the program “Genesis and Interaction of Social, Cultural and Language Communities” of the Russian Academy of Sciences. The author’s investigation is supported by the 1) RFBR The Russian Fund of Basic Researches. (RFFI) under the grant 08-06-00371а and the grant “Elaboration of Multimodal Russian Corpus (MURCO) within the framework of Russian National Corpus (www.ruscorpora.ru)”, and 2) RFH The Russian Fund of Humanity. (RGNF) under the grants “Russian Gesticulation according to the Cinematographic Data” and “Deeply Annotated Multimodal Russian Corpus: Elaboration and Creation”. References Busso, C. , Narayanan, S. (2008). Recording Audio-Visual Emotional Databases from Actors : a Closer Look. In 6th International Conference on Language Resources and Evaluation. Marrakesh: ELRA. Forchini, P. (2009). Spontaneity reloaded: American face-to-face and movie conversation compared. In Corpus Linguistics 2009. Abstracts The 5th Corpus Linguistics Conference, 20-23 July 2009, Liverpool, p. 118. Grishina, E. (2006). Spoken Russian in the Russian National Corpus (RNC). In LREC’2006: 5th International Conference on Language Resources and Evaluation. ELRA, pp. 121-124. Grishina, E. (2007a). O markerah razgovornoj rechi (predvaritel’noje issledovanije podkorpusa kino v Nacional’nom korpuse russkogo jazyka). In Kompjuternaja lingvistika i intellektual’nyje tehnologii. Trudy mezhdunarodnoj konferencii “Dialog 2007”. Moscow, RSGU, pp. 147-156. Grishina, E. (2007b). Text Navigators in Spoken Russian. In Proceedings of the workshop “Representation of Semantic Structure of Spoken Speech” (CAEPIA’2007, Spain, 2007, 12-16.11.07, Salamanca). Salamanca, pp. 39-50. Grishina, E. (2009a). Multimodal Russian Corpus (MURCO): Types of annotation and annotator’s workbenches. In Corpus Linguistics’2009. Liverpool (forthcoming). Grishina, E. (2009b). Multimodal Russian Corpus (MURCO): general structure and user interface. In NLP, Corpus Linguistics, Corpus Based Grammar Research. Fifth International Conference. Smolenice, Slovakia, 25-27 November 2009. Proceedings. Brno, Tribun, pp. 119-131. Grishina, E. (2009c). K voprosu o sootnoshenii slova I zhesta: vokal’nyj zhest Oh v ustnoj rechi. In Komp’juternaja lingvistika i intellektual’nyje tekhnologii (Mezhdunarodnaja konferencija “Dialog 2009”, 8(15)). Computational Linguistics and Intellectual Technologies (Annual International Conference “Dialogue 2009”, 8(15)). Moscow: RSGU, pp. 80-90. Grishina, E. (2010a). Multimodal Russian Corpus (MURCO): First Steps. In LREC’2010 (forthcoming). Grishina, E. (2010b). Vokal’nyj zhest Oh v ustnoj rechi. In Komp’juternaja lingvistika i intellektual’nyje technologii (Mezhdunarodnaja konferencija “Dialog 2010”, 9(16)). Computational Linguistics and Intellectual Technologies (Annual International Conference “Dialogue 2010”, 9(16)). Moscow: RSGU, (forthcoming). Sinclair, J.M. (2004). Corpus Creation. In Corpus Linguistics: Readings in a Widening Discipline. London, Continuum, pp. 78-84.  cinematographic??affect – только глагол