Abstract
Automatic dialogue management including dialogue act (DA) recognition is usually focused on dialogues in the audio signal. However, some dialogues are also available in a written form and their automatic analysis is also very important.
The main goal of this paper thus consists in the dialogue act recognition from printed documents. For visual DA recognition, we propose a novel deep model that combines two recurrent neural networks.
The approach is evaluated on a newly created dataset containing printed dialogues from the English VERBMOBIL corpus. We have shown that visual information does not have any positive impact on DA recognition using good quality images where the OCR result is excellent. We have also demonstrated that visual information can significantly improve the DA recognition score on low-quality images with erroneous OCR.
To the best of our knowledge, this is the first attempt focused on DA recognition from visual data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The noise is not artificial (i.e. we didn’t perform any image transformation), but we have created the noise by real usage of the scanner. We put a blank piece of paper in the scanner and we changed the scanning quality by different scanning options and the amount of light.
- 2.
References
Bunt, H.: Context and dialogue control. Think Quarterly 3(1), 19–31 (1994)
Stolcke, A., et al.: Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput. Linguist. 26(3), 339–373 (2000)
Frankel, J., King, S.: ASR-articulatory speech recognition. In: Seventh European Conference on Speech Communication and Technology (2001)
Cerisara, C., Král, P., Lenc, L.: On the effects of using word2vec representations in neural networks for dialogue act recognition. Comput. Speech Lang. 47, 175–193 (2018)
Jekat, S., Klein, A., Maier, E., Maleck, I., Mast, M., Quantz, J.J.: Dialogue acts in verbmobil (1995)
Godfrey, J.J., Holliman, E.C., McDaniel, J.: Switchboard: telephone speech corpus for research and development. In: Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ser. ICASSP 1992, pp. 517–520. IEEE Computer Society, USA (1992)
Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., Carvey, H.: The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, International Computer Science Institute, Berkely (2004)
Benedı, J.-M., et al.: Design and acquisition of a telephone spontaneous speech dialogue corpus in Spanish: Dihana. In: Fifth International Conference on Language Resources and Evaluation (LREC), pp. 1636–1639 (2006)
Colombo, P., Chapuis, E., Manica, M., Vignon, E., Varni, G., Clavel, C.: Guiding attention in sequence-to-sequence models for dialogue act prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 7594–7601 (2020)
Shang, G., Tixier, A.J.-P., Vazirgiannis, M., Lorré, J.-P.: Speaker-change aware CRF for dialogue act classification, arXiv preprint arXiv:2004.02913 (2020)
Alexandersson, J., et al.: Dialogue acts in Verbmobil 2. DFKI Saarbrücken (1998)
Reithinger, N., Klesen, M.: Dialogue act classification using language models. In: Fifth European Conference on Speech Communication and Technology (1997)
Samuel, K., Carberry, S., Vijay-Shanker, K.: Dialogue act tagging with transformation-based learning, arXiv preprint cmp-lg/9806006 (1998)
Martínek, J., Král, P., Lenc, L., Cerisara, C.: Multi-lingual dialogue act recognition with deep learning methods, arXiv preprint arXiv:1904.05606 (2019)
Cerisara, C., Jafaritazehjani, S., Oluokun, A., Le, H.: Multi-task dialog act and sentiment recognition on mastodon, arXiv preprint arXiv:1807.05013 (2018)
Li, J., Fei, H., Ji, D.: Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 616–626 (2020)
Zhang, Q., Fu, J., Liu, X., Huang, X.: Adaptive co-attention network for named entity recognition in tweets. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)
Audebert, N., Herold, C., Slimani, K., Vidal, C.: Multimodal deep networks for text and image-based document classification. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1167, pp. 427–443. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43823-4_35
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext.zip: compressing text classification models, arXiv preprint arXiv:1612.03651 (2016)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
Jain, R., Wigington, C.: Multimodal document image classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 71–77. IEEE (2019)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017)
Martínek, J., Lenc, L., Král, P., Nicolaou, A., Christlein, V.: Hybrid training data for historical text OCR. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 565–570. IEEE (2019)
Han, L., Kamdar, M.R.: MRI to MGMT: predicting methylation status in glioblastoma patients using convolutional recurrent neural networks (2017)
Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013)
Sauvola, J., Pietikäinen, M.: Adaptive document image binarization. Pattern Recogn. 33(2), 225–236 (2000)
Acknowledgements
This work has been partly supported from ERDF “Research and Development of Intelligent Components of Advanced Technologies for the Pilsen Metropolitan Area (InteCom)” (no.: CZ.02.1.01/0.0/0.0/17_048/0007267) and by Grant No. SGS-2019-018 Processing of heterogeneous data and its specialized applications. We would like to thank also Mr. Matěj Zeman for some implementation work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Martínek, J., Král, P., Lenc, L. (2021). Dialogue Act Recognition Using Visual Information. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12822. Springer, Cham. https://doi.org/10.1007/978-3-030-86331-9_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-86331-9_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86330-2
Online ISBN: 978-3-030-86331-9
eBook Packages: Computer ScienceComputer Science (R0)
