This paper presents the ParlaMint corpora containing transcriptions of the sessions of the 17 Eur... more This paper presents the ParlaMint corpora containing transcriptions of the sessions of the 17 European national parliaments with half a billion words. The corpora are uniformly encoded, contain rich meta-data about 11 thousand speakers, and are linguistically annotated following the Universal Dependencies formalism and with named entities. Samples of the corpora and conversion scripts are available from the project’s GitHub repository, and the complete corpora are openly available via the CLARIN.SI repository for download, as well as through the NoSketch Engine and KonText concordancers and the Parlameter interface for on-line exploration and analysis.
Recent studies have demonstrated the interlocutor utilizes and coordinates symbolic, iconic, and ... more Recent studies have demonstrated the interlocutor utilizes and coordinates symbolic, iconic, and indexical resources in meaning communication, through descriptions, depictions, and indications, respectively (e.g. Ferrara & Hodge, 2018). In particular, Clark (2016) identifies depicting as a basic method of communication where the interlocutor expresses meaning by creating a physical analog of the depicted scene. In this talk, we explore the notion of depicting, drawing on evidence from Flemish Sign Language (VGT) narratives and gestures in face-to-face interaction in American English, offering corpus-based insights into the comparability thereof.
This paper describes a pilot analysis of gender differences in the revised transcripts of speeche... more This paper describes a pilot analysis of gender differences in the revised transcripts of speeches from the sittings in the Danish Parliament in the period from 2009 to 2017. Information about the number and duration of the speeches, the gender, age, party, and role in the party was automatically extracted from the transcripts and from other data on the Danish Parliament web site. The analysis shows statistically significant differences in the number and duration of the speeches by male and female MPs, and we also found differences in speech frequency with respect to the age of the MPs. Our analysis confirms previous studies on parliamentary data in other countries showing that the role of the MPs in their party influences their participation in the debates. Furthermore, we found that female ministers were speaking more in the period with a female prime minister than they did under a male prime minister. In the future, we will determine the statistical significance of the various pa...
Topics in Intelligent Engineering and Informatics, 2018
Mirroring and synchronization of non-verbal behavior is an important characteristics of human con... more Mirroring and synchronization of non-verbal behavior is an important characteristics of human conduct also in communication. The aims of this paper are to analyze the occurrences of mirroring gestures, which comprise head movements, facial expressions, body postures and hand gestures, in first encounters and to determine whether information about the gestures of an individual can be used to predict the presence and the class of the gestures of the interlocutor. The contribution of related speech token is also investigated. The analysis of the encounters shows that 20-30% of the head movements, facial expressions and body postures are mirrored in the corpus, while there are only few occurrences of mirrored hand gestures. The latter are therefore not included in the prediction experiments. The results of the experiments, in which various machine learning algorithms have been applied, show that information about the shape and duration of the gestures of one participant contributes to the prediction of the presence and class of the gestures of the other participant, and that adding information about the related speech tokens in some cases improves the prediction performance. These results indicate that it is useful to take mirroring into account when designing and implementing cognitive aware info-communicative devices.
ParlaMint 2.1 is a multilingual set of 17 comparable corpora containing parliamentary debates mos... more ParlaMint 2.1 is a multilingual set of 17 comparable corpora containing parliamentary debates mostly starting in 2015 and extending to mid-2020, with each corpus being about 20 million words in size. The sessions in the corpora are marked as belonging to the COVID-19 period (from November 1st 2019), or being "reference" (before that date). The corpora have extensive metadata, including aspects of the parliament; the speakers (name, gender, MP status, party affiliation, party coalition/opposition); are structured into time-stamped terms, sessions and meetings; with speeches being marked by the speaker and their role (e.g. chair, regular speaker). The speeches also contain marked-up transcriber comments, such as gaps in the transcription, interruptions, applause, etc. Note that some corpora have further information, e.g. the year of birth of the speakers, links to their Wikipedia articles, their membership in various committees, etc. The corpora are encoded according to the Parla-CLARIN TEI recommendation (https://clarin-eric.github.io/parla-clarin/), but have been validated against the compatible, but much stricter ParlaMint schemas. This entry contains the linguistically marked-up version of the corpus, while the text version is available at http://hdl.handle.net/11356/1432. The ParlaMint.ana linguistic annotation includes tokenization, sentence segmentation, lemmatisation, Universal Dependencies part-of-speech, morphological features, and syntactic dependencies, and the 4-class CoNLL-2003 named entities. Some corpora also have further linguistic annotations, such as PoS tagging or named entities according to language-specific schemes, with their corpus TEI headers giving further details on the annotation vocabularies and tools. The compressed files include the ParlaMint.ana XML TEI-encoded linguistically annotated corpus; the derived corpus in CoNLL-U with TSV speech metadata; and the vertical files (with registry file), suitable for use with CQP-based concordancers, such as CWB, noSketch Engine or KonText. Also included is the 2.1 release of the data and scripts available at the GitHub repository of the ParlaMint project. As opposed to the previous version 2.0, this version corrects some errors in various corpora and adds the information on upper / lower house for bicameral parliaments. The vertical files have also been changed to make them easier to use in the concordancers
The MUMIN multimodal coding scheme was created to experiment with annotation of multimodal commun... more The MUMIN multimodal coding scheme was created to experiment with annotation of multimodal communication in video clips of interviews taken from Swedish, Finnish and Danish television broadcasting. The coding scheme is also intended to be a general instrument for the study of gestures and facial displays in interpersonal communication, in particular the role played by multimodal expressions for feedback, turn management and sequencing.
This paper deals with the temporal coordination between facial expressions and co-occurring head ... more This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication-an aspect that is relevant to the generation of natural multimodal expressions in ECAs.
This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflec... more This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflecting quite different communicative situations. The first consists of eight video recordings of map task dialogues, in which one of the participants explains to the other an itinerary on a map. The two participants sit in different rooms and communicate via headphones without being able to see each other. The other corpus is a collection of twelve video recordings of free face-to-face conversations in which the participants, who met for the first time, were recorded in a studio while standing in front of each other talking. The language used in both corpora is Danish. We describe how spoken and nonverbal feedback have been annotated and give an account of the distribution of different types of feedback expressions in the two corpora. The analysis uncovers interesting differences in both amount and type (e.g. unimodal vs. multimodal) of feedback differences. These differences can be explained in terms of the different physical settings as well as the nature of the interaction, which is functional to the map task in the one corpus whereas in only serves the purpose of talking for the sake of talking in the other.
This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-record... more This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-recorded first acquaintance conversations between Danish speakers. The annotation includes speech transcription including word boundaries, and formal as well as functional coding of gestural behaviours, specifically head movements, facial expressions, and body posture. The corpus has served as the empirical basis for a number of studies of communication phenomena related to turn management, feedback exchange, information packaging and the expression of emotional attitudes. We describe the annotation scheme, procedure, and annotation results. We then summarise a number of studies conducted on the corpus. The corpus is available for research and teaching purposes through the authors of this article.
2020 IEEE International Conference on Human-Machine Systems (ICHMS), 2020
This study concerns the use of speech pauses, and especially breath pauses in a Danish corpus of ... more This study concerns the use of speech pauses, and especially breath pauses in a Danish corpus of spontaneous dyadic conversations. Speech pauses which have specific communicative functions are investigated in relation to their occurrences before and after other communicative units, all annotated and classified in the form of dialogue acts. Breath pauses have been addressed in only few studies even though they are important in communication and therefore should be accounted for when implementing human-machine dialogue systems. Dialogue acts, on the contrary, have been one of the backbones in dialogue systems since they generalize over different expressions of common communicative functions. In the current work, we describe the annotation of dialogue acts in the corpus and present an analysis of pauses using these annotations. To our best knowledge, dialogue acts have not been previously used for analyzing the functions of breath pauses. Our analysis shows that the most common type of pause having a communicative function in the Danish conversations are breath pauses. Breath pauses in the corpus have different uses, one of these being that of delimiting speech segments which are left unfinished and are then abandoned by the speaker (retractions in dialogue acts terminology) and therefore perceivable breathing can be a useful feature for determining spoken segments which must not be included in the dialogue history in human-machine dialogue systems.
The paper describes the main characteristics of the scientific programme of the fourth conference... more The paper describes the main characteristics of the scientific programme of the fourth conference of Digital Humanities in the Nordic Countries (DHN2019) that took place in Copenhagen in March 2019. DHN2019, as the preceding DHN conferences, aimed to connect researchers and practitioners addressing all topics that generally belong under the Digital Humanities field. The DHN conferences address in particular researchers from the Nordic countries, comprising the Baltic region, but are also open to researchers from all over the world. Thus, DHN2019 attracted participants from 27 countries. The call for papers of DHN2019 followed the strategy proposed by the organizers of the DHN2018, who attempted to encompass two conference traditions, one from the humanities accepting abstracts as submissions and one from computer science accepting full papers of varying length. The latter type of submission was the most popular in 2019 and the present proceedings collect these papers. With respect t...
2019 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 2019
This paper investigates the relation between the form and function of hand gestures in audio and ... more This paper investigates the relation between the form and function of hand gestures in audio and video recordings of American and English political iscourse of different type. Gestures have an important function in face-to-face communication contributing to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The relation between form and function of gestures has been described by some of the pioneers of gestural studies. However, since gestures are multifunctional and they must be interpreted in context, it is important to investigate to what extent the form of gestures can be used to interpret their function automatically. Individuating the relation between form and function of gestures is also important for generating appropriate gestures in various communicative situations and this knowledge is vital for the integration of machine-human communicative and cognitive functions. In this paper we show that ...
This paper addresses the semi-automatic subject area annotation of the Danish Parliament Corpus 2... more This paper addresses the semi-automatic subject area annotation of the Danish Parliament Corpus 2009-2017 in order to construct a gold standard corpus for automatic classification. The corpus consists of the transcriptions of the speeches in the Danish parliamentary meetings. In our annotation work, we mainly use subject categories proposed by Danish scholars in political sciences. The relevant subjects areas of the speeches have been manually annotated using the titles of the agendas items for the parliamentary meetings and then the subjects areas have been assigned to the corresponding speeches. Some subjects cooccur in the agendas, since they are often debated at the same time. The fact that the same speech can belong to more subject areas is further analysed. Currently, more than 29,000 speeches have been classified using the titles of the agenda items. Different evaluation strategies have been applied. We also describe automatic classification experiments on a subset of the cor...
This paper addresses differences in the word use of two left-winged and two right-winged Danish p... more This paper addresses differences in the word use of two left-winged and two right-winged Danish parties, and how these differences reflecting some of the basic stances of the parties can be used to automatically identify the party of politicians from their speeches. In the first study, the most frequent and characteristic lemmas in the manifestos of the political parties are analysed. The analysis shows that the most frequently occurring lemmas in the manifestos reflect either the ideology or the position of the parties towards specific subjects, confirming for Danish preceding studies of English and German manifestos. Successively, we scaled our analysis applying machine learning on different language models built on the transcribed speeches by members of the same parties in the Parliament (Hansards) in order to determine to what extent it is possible to predict the party of the politicians from the speeches. The speeches used are a subset of the Danish Parliament corpus 2009–2017....
This paper presents the ParlaMint corpora containing transcriptions of the sessions of the 17 Eur... more This paper presents the ParlaMint corpora containing transcriptions of the sessions of the 17 European national parliaments with half a billion words. The corpora are uniformly encoded, contain rich meta-data about 11 thousand speakers, and are linguistically annotated following the Universal Dependencies formalism and with named entities. Samples of the corpora and conversion scripts are available from the project’s GitHub repository, and the complete corpora are openly available via the CLARIN.SI repository for download, as well as through the NoSketch Engine and KonText concordancers and the Parlameter interface for on-line exploration and analysis.
Recent studies have demonstrated the interlocutor utilizes and coordinates symbolic, iconic, and ... more Recent studies have demonstrated the interlocutor utilizes and coordinates symbolic, iconic, and indexical resources in meaning communication, through descriptions, depictions, and indications, respectively (e.g. Ferrara & Hodge, 2018). In particular, Clark (2016) identifies depicting as a basic method of communication where the interlocutor expresses meaning by creating a physical analog of the depicted scene. In this talk, we explore the notion of depicting, drawing on evidence from Flemish Sign Language (VGT) narratives and gestures in face-to-face interaction in American English, offering corpus-based insights into the comparability thereof.
This paper describes a pilot analysis of gender differences in the revised transcripts of speeche... more This paper describes a pilot analysis of gender differences in the revised transcripts of speeches from the sittings in the Danish Parliament in the period from 2009 to 2017. Information about the number and duration of the speeches, the gender, age, party, and role in the party was automatically extracted from the transcripts and from other data on the Danish Parliament web site. The analysis shows statistically significant differences in the number and duration of the speeches by male and female MPs, and we also found differences in speech frequency with respect to the age of the MPs. Our analysis confirms previous studies on parliamentary data in other countries showing that the role of the MPs in their party influences their participation in the debates. Furthermore, we found that female ministers were speaking more in the period with a female prime minister than they did under a male prime minister. In the future, we will determine the statistical significance of the various pa...
Topics in Intelligent Engineering and Informatics, 2018
Mirroring and synchronization of non-verbal behavior is an important characteristics of human con... more Mirroring and synchronization of non-verbal behavior is an important characteristics of human conduct also in communication. The aims of this paper are to analyze the occurrences of mirroring gestures, which comprise head movements, facial expressions, body postures and hand gestures, in first encounters and to determine whether information about the gestures of an individual can be used to predict the presence and the class of the gestures of the interlocutor. The contribution of related speech token is also investigated. The analysis of the encounters shows that 20-30% of the head movements, facial expressions and body postures are mirrored in the corpus, while there are only few occurrences of mirrored hand gestures. The latter are therefore not included in the prediction experiments. The results of the experiments, in which various machine learning algorithms have been applied, show that information about the shape and duration of the gestures of one participant contributes to the prediction of the presence and class of the gestures of the other participant, and that adding information about the related speech tokens in some cases improves the prediction performance. These results indicate that it is useful to take mirroring into account when designing and implementing cognitive aware info-communicative devices.
ParlaMint 2.1 is a multilingual set of 17 comparable corpora containing parliamentary debates mos... more ParlaMint 2.1 is a multilingual set of 17 comparable corpora containing parliamentary debates mostly starting in 2015 and extending to mid-2020, with each corpus being about 20 million words in size. The sessions in the corpora are marked as belonging to the COVID-19 period (from November 1st 2019), or being "reference" (before that date). The corpora have extensive metadata, including aspects of the parliament; the speakers (name, gender, MP status, party affiliation, party coalition/opposition); are structured into time-stamped terms, sessions and meetings; with speeches being marked by the speaker and their role (e.g. chair, regular speaker). The speeches also contain marked-up transcriber comments, such as gaps in the transcription, interruptions, applause, etc. Note that some corpora have further information, e.g. the year of birth of the speakers, links to their Wikipedia articles, their membership in various committees, etc. The corpora are encoded according to the Parla-CLARIN TEI recommendation (https://clarin-eric.github.io/parla-clarin/), but have been validated against the compatible, but much stricter ParlaMint schemas. This entry contains the linguistically marked-up version of the corpus, while the text version is available at http://hdl.handle.net/11356/1432. The ParlaMint.ana linguistic annotation includes tokenization, sentence segmentation, lemmatisation, Universal Dependencies part-of-speech, morphological features, and syntactic dependencies, and the 4-class CoNLL-2003 named entities. Some corpora also have further linguistic annotations, such as PoS tagging or named entities according to language-specific schemes, with their corpus TEI headers giving further details on the annotation vocabularies and tools. The compressed files include the ParlaMint.ana XML TEI-encoded linguistically annotated corpus; the derived corpus in CoNLL-U with TSV speech metadata; and the vertical files (with registry file), suitable for use with CQP-based concordancers, such as CWB, noSketch Engine or KonText. Also included is the 2.1 release of the data and scripts available at the GitHub repository of the ParlaMint project. As opposed to the previous version 2.0, this version corrects some errors in various corpora and adds the information on upper / lower house for bicameral parliaments. The vertical files have also been changed to make them easier to use in the concordancers
The MUMIN multimodal coding scheme was created to experiment with annotation of multimodal commun... more The MUMIN multimodal coding scheme was created to experiment with annotation of multimodal communication in video clips of interviews taken from Swedish, Finnish and Danish television broadcasting. The coding scheme is also intended to be a general instrument for the study of gestures and facial displays in interpersonal communication, in particular the role played by multimodal expressions for feedback, turn management and sequencing.
This paper deals with the temporal coordination between facial expressions and co-occurring head ... more This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication-an aspect that is relevant to the generation of natural multimodal expressions in ECAs.
This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflec... more This paper presents an analysis of spoken and nonverbal feedback in two multimodal corpora reflecting quite different communicative situations. The first consists of eight video recordings of map task dialogues, in which one of the participants explains to the other an itinerary on a map. The two participants sit in different rooms and communicate via headphones without being able to see each other. The other corpus is a collection of twelve video recordings of free face-to-face conversations in which the participants, who met for the first time, were recorded in a studio while standing in front of each other talking. The language used in both corpora is Danish. We describe how spoken and nonverbal feedback have been annotated and give an account of the distribution of different types of feedback expressions in the two corpora. The analysis uncovers interesting differences in both amount and type (e.g. unimodal vs. multimodal) of feedback differences. These differences can be explained in terms of the different physical settings as well as the nature of the interaction, which is functional to the map task in the one corpus whereas in only serves the purpose of talking for the sake of talking in the other.
This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-record... more This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-recorded first acquaintance conversations between Danish speakers. The annotation includes speech transcription including word boundaries, and formal as well as functional coding of gestural behaviours, specifically head movements, facial expressions, and body posture. The corpus has served as the empirical basis for a number of studies of communication phenomena related to turn management, feedback exchange, information packaging and the expression of emotional attitudes. We describe the annotation scheme, procedure, and annotation results. We then summarise a number of studies conducted on the corpus. The corpus is available for research and teaching purposes through the authors of this article.
2020 IEEE International Conference on Human-Machine Systems (ICHMS), 2020
This study concerns the use of speech pauses, and especially breath pauses in a Danish corpus of ... more This study concerns the use of speech pauses, and especially breath pauses in a Danish corpus of spontaneous dyadic conversations. Speech pauses which have specific communicative functions are investigated in relation to their occurrences before and after other communicative units, all annotated and classified in the form of dialogue acts. Breath pauses have been addressed in only few studies even though they are important in communication and therefore should be accounted for when implementing human-machine dialogue systems. Dialogue acts, on the contrary, have been one of the backbones in dialogue systems since they generalize over different expressions of common communicative functions. In the current work, we describe the annotation of dialogue acts in the corpus and present an analysis of pauses using these annotations. To our best knowledge, dialogue acts have not been previously used for analyzing the functions of breath pauses. Our analysis shows that the most common type of pause having a communicative function in the Danish conversations are breath pauses. Breath pauses in the corpus have different uses, one of these being that of delimiting speech segments which are left unfinished and are then abandoned by the speaker (retractions in dialogue acts terminology) and therefore perceivable breathing can be a useful feature for determining spoken segments which must not be included in the dialogue history in human-machine dialogue systems.
The paper describes the main characteristics of the scientific programme of the fourth conference... more The paper describes the main characteristics of the scientific programme of the fourth conference of Digital Humanities in the Nordic Countries (DHN2019) that took place in Copenhagen in March 2019. DHN2019, as the preceding DHN conferences, aimed to connect researchers and practitioners addressing all topics that generally belong under the Digital Humanities field. The DHN conferences address in particular researchers from the Nordic countries, comprising the Baltic region, but are also open to researchers from all over the world. Thus, DHN2019 attracted participants from 27 countries. The call for papers of DHN2019 followed the strategy proposed by the organizers of the DHN2018, who attempted to encompass two conference traditions, one from the humanities accepting abstracts as submissions and one from computer science accepting full papers of varying length. The latter type of submission was the most popular in 2019 and the present proceedings collect these papers. With respect t...
2019 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 2019
This paper investigates the relation between the form and function of hand gestures in audio and ... more This paper investigates the relation between the form and function of hand gestures in audio and video recordings of American and English political iscourse of different type. Gestures have an important function in face-to-face communication contributing to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The relation between form and function of gestures has been described by some of the pioneers of gestural studies. However, since gestures are multifunctional and they must be interpreted in context, it is important to investigate to what extent the form of gestures can be used to interpret their function automatically. Individuating the relation between form and function of gestures is also important for generating appropriate gestures in various communicative situations and this knowledge is vital for the integration of machine-human communicative and cognitive functions. In this paper we show that ...
This paper addresses the semi-automatic subject area annotation of the Danish Parliament Corpus 2... more This paper addresses the semi-automatic subject area annotation of the Danish Parliament Corpus 2009-2017 in order to construct a gold standard corpus for automatic classification. The corpus consists of the transcriptions of the speeches in the Danish parliamentary meetings. In our annotation work, we mainly use subject categories proposed by Danish scholars in political sciences. The relevant subjects areas of the speeches have been manually annotated using the titles of the agendas items for the parliamentary meetings and then the subjects areas have been assigned to the corresponding speeches. Some subjects cooccur in the agendas, since they are often debated at the same time. The fact that the same speech can belong to more subject areas is further analysed. Currently, more than 29,000 speeches have been classified using the titles of the agenda items. Different evaluation strategies have been applied. We also describe automatic classification experiments on a subset of the cor...
This paper addresses differences in the word use of two left-winged and two right-winged Danish p... more This paper addresses differences in the word use of two left-winged and two right-winged Danish parties, and how these differences reflecting some of the basic stances of the parties can be used to automatically identify the party of politicians from their speeches. In the first study, the most frequent and characteristic lemmas in the manifestos of the political parties are analysed. The analysis shows that the most frequently occurring lemmas in the manifestos reflect either the ideology or the position of the parties towards specific subjects, confirming for Danish preceding studies of English and German manifestos. Successively, we scaled our analysis applying machine learning on different language models built on the transcribed speeches by members of the same parties in the Parliament (Hansards) in order to determine to what extent it is possible to predict the party of the politicians from the speeches. The speeches used are a subset of the Danish Parliament corpus 2009–2017....
Uploads
Papers by Costanza Navarretta