Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Context-Aware Multimodal Human–Computer Interaction

  • Chapter
Interactive Collaborative Information Systems

Part of the book series: Studies in Computational Intelligence ((SCI,volume 281))

  • 1580 Accesses

Abstract

Crisis response and management involve the collaboration of many people. To perform and coordinate their activities, they must rely on detailed and accurate information about the crisis, the environment, and many more factors. To ensure collaboration of emergency services and high-quality care for victims, the ability to supply dynamic and contextually correlated information is necessary. However, current approaches to construct globally consistent views of crises suffer from problems identified in [60]: (a) the setting of events is constantly changing, (b) the information is distributed across geographically distant locations, and (c) the complexity of the crisis management organization makes it difficult and time consuming to collaborate and verify obtained information.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Albacete, P.L., Chang, S., Polese, G., Baker, B.: Iconic language design for people with significant speech and multiple impairments. In: Proc. of the ACM Assets 1994, CA, USA, pp. 23–30. ACM, New York (1994)

    Google Scholar 

  2. Alexandersson, J., Becker, T.: The Formal Foundations Underlying Overlay. In: IWCS-5, The Netherlands, pp. 22–36 (2003)

    Google Scholar 

  3. Anson, D.K., Moist, P., Przywara, M., Wells, H., Saylor, H., Maxime, H.: The Effects of Word Completion and Word Prediction on Typing Rates using On-Screen Keyboards. In: Proc. of RESNA. RESNA Press, Arlington (2005)

    Google Scholar 

  4. Basu, A., Sankar, S., Chakraborty, K., Bhattarcharya, S., Choudhury, M., Patel, R.: Vernacula Educational Communication Tool for the People with Multiple Disabilities. In: Development by Design Conference, Bangalore (2002)

    Google Scholar 

  5. Beardon, C.: CD-Icon, an iconic language-based on conceptual dependency. Intelligent Tutoring Media 3(4) (1992)

    Google Scholar 

  6. Benjamins, T.: MACSIM: Multi-Agent Crisis Simulator, Interpreter and Monitor. Master thesis, TU Delft (2006)

    Google Scholar 

  7. Bevelier, D., Corina, D.P., Neville, H.J.: Brain and Language: a Perspective from Sign Language. Neuron 21, 275–278 (1998)

    Article  Google Scholar 

  8. Bliss, C.K.: The Blissymbols Picture Book. Semantography Press, Sidney (1984)

    Google Scholar 

  9. Bosma, W., André, E.: Exploiting emotions to disambiguate dialogue acts. In: Proc. of IUI, Portugal, pp. 85–92 (2004)

    Google Scholar 

  10. Bui, T.H., Poel, M., Nijholt, A., Zwiers, J.: A Tractable DDN-POMDP Approach to Affective Dialogue Modeling for General Probabilistic Frame-based Dialogue Systems. In: IJCAI 2007, India (2007)

    Google Scholar 

  11. Carpenter, B.: The Logic of Typed Feature Structures. Cambridge University Press, England (1992)

    Book  MATH  Google Scholar 

  12. Cassell, J.: Embodied conversational interface agents. Communication ACM 43(4), 70–78 (2000)

    Article  Google Scholar 

  13. Cassell, J., Stocky, T., Bickmore, T., Gao, Y., Nakano, Y., Ryokai, K., Tversky, D., Vaucelle, C., Vilhjálmsson, H.: MACK: Media lab Autonomous Conversational Kiosk. In: Proc. of Imagina 2002, Monte Carlo (2002)

    Google Scholar 

  14. Catizone, R., Setzer, A., Wilks, Y.: Multimodal dialogue management in the Comic project. In: Proc. of EACL, Hungary (2003)

    Google Scholar 

  15. Champoux, B., Fujisawa, K., Inoue, T., Iwadate, Y.: Transmitting Visual Information: Icons become Words. IEEE - Information Visualization, 244–249 (2000)

    Google Scholar 

  16. Chin Jr., G., Stephan, E.G., Gracio, D.K., Kuchar, O.A., Whitney, P.D., Schuchardt, K.L.: Developing Concept-Based User Interfaces for Scientific Computing. Computer 39(9), 26–34 (2006)

    Article  Google Scholar 

  17. Chiţu, A.G., Rothkrantz, L.J.M., Wiggers, P.: Comparison between different feature extraction techniques for audio-visual. Journal on Multimodal User Interfaces 1(1), 7–20 (2007)

    Article  Google Scholar 

  18. Cohn, N.: Visual Syntactic Structures, Towards a Generative Grammar of Visual Language (2003), http://www.emaki.net/essays/

  19. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active Appearance Models. In: European Conf. Computer Vision, pp. 484–498 (1998)

    Google Scholar 

  20. Datcu, D., Rothkrantz, L.J.M.: Multimodal workbench for human emotion recognition. In: Software Demo at IEEE CVPR 2007, USA (2007)

    Google Scholar 

  21. Datcu, D., Rothkrantz, L.J.M.: The use of Active Appearance Model for facial expression recognition in crisis environments. In: Proc. of ISCRAM 2007, USA, pp. 515–524 (2007)

    Google Scholar 

  22. Davis, S., Mermelstein, P.: Comparison of Parametric Representations for Monosyllabic Word Recognition. IEEE ASSP 28, 357–366 (1980)

    Article  Google Scholar 

  23. Dor, R.: The Ear’s Mind, a Computer Model of the Fundamental Mechanisms of the Perception of Sound, Master thesis, TU Delft (2005)

    Google Scholar 

  24. Dymon, U.J.: An Analysis of Emergency Map Symbology. Int. Journal of Emergency Management 1(3), 227–237 (2003)

    Article  Google Scholar 

  25. Ekman, P., Friesen, W.F.: Unmasking the Face. Prentice-Hall, Inc., Englewood Cliffs (1975)

    Google Scholar 

  26. Farberow, N.L., Frederick, C.J.: Training Manual for Human Service Workers in Major Disasters. Rockville, Maryland - National Institute of Mental Health (1978)

    Google Scholar 

  27. Fitrianie, S.: An Icon-based Communication Tool on a PDA, Postgraduate Thesis. TU Eindhoven (2004)

    Google Scholar 

  28. Fitrianie, S., Rothkrantz, L.J.M.: Communication in Crisis Situations using Icon Language. In: Proc. of IEEE ICME 2005, The Netherlands, pp. 1370–1373 (2005)

    Google Scholar 

  29. Fitrianie, S., Datcu, D., Rothkrantz, L.J.M.: Constructing Knowledge of the World in Crisis Situations using Visual Language. In: Proc of. IEEE SMC 2006, Taiwan, pp. 121–126 (2006)

    Google Scholar 

  30. Fitrianie, S., Rothkrantz, L.J.M.: An Adaptive Keyboard with Personalized Language-based Features. In: Matoušek, V., Mautner, P. (eds.) TSD 2007. LNCS (LNAI), vol. 4629, pp. 131–138. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  31. Fitrianie, S., Yang, Z., Rothkrantz, L.J.M.: Developing concept-based user interface using icons for reporting observations. In: ISCRAM 2008, USA (2008)

    Google Scholar 

  32. Foster, M.E., White, M., Setzer, A., Catizone, R.: Multimodal generation in the COMIC dialogue. In: ACL 2005, USA, pp. 45–48 (2004)

    Google Scholar 

  33. Gustafson, J., Bell, L., Beskow, J., Boye, J., Carlson, R., Edlund, J., Granström, B., House, D., Wirén, M.: Adapt - a Multimodal Conversational Dialogue System in an Apartment Domain. In: ICSLP 2000 (2), pp. 134–137 (2000)

    Google Scholar 

  34. Housz, T.I.: The elephant’s memory (1994–1996), http://www.khm.de/timot

  35. Homeland Security Working Group: Symbology Reference (2003), http://www.fgdc.gov/HSWG/index.html

  36. IBM e-business Case Studies: Bullhead City Police Department, http://www.ibm.com/e-business/doc/content/casestudy/47405.html

  37. Johanson, B., Fox, A., Winograd, T.: The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms. IEEE Pervasive Computing 1(2), 67–74 (2002)

    Article  Google Scholar 

  38. Joshi, A.K., Vijay-Shanker, K.: Compositional Semantics for Lexicalized Tree-Adjoining Grammars. In: Proc. of Computational Semantics, The Netherlands (1999)

    Google Scholar 

  39. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proc. of IEEE Automatic Face and Gesture Recognition, France (2000)

    Google Scholar 

  40. Karlson, A., Bederson, B., Contreras-Vidal, J.: Understanding Single Handed Use of Handheld Devices. In: Jo, L. (ed.) Handbook of Research on User Interface Design and Evaluation for Mobile Technology (2006) (in press)

    Google Scholar 

  41. Keizer, S.: Reasoning under uncertainty in Natural Language Dialogue using Bayesian Networks, Doctoral Dissertation. Twente University (2003)

    Google Scholar 

  42. Kjeldskov, J., Kolbe, N.: Interaction Design for Handheld Computers. In: Proc. of APCHI 2002. Science Press, China (2002)

    Google Scholar 

  43. Leemans, N.E.M.P.: VIL: A Visual Inter Lingua. Doctoral Dissertation, Worcester Polytechnic Institute, USA (2001)

    Google Scholar 

  44. Lucas, B.D., Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision. In: IJCAI 1981, pp. 674–679 (1981)

    Google Scholar 

  45. MacKenzie, I.S., Zhang, S.X., Soukoreff, R.W.: Text Entry using Soft Keyboards. Behaviour and Information Technology 18, 235–244 (1999)

    Article  Google Scholar 

  46. Mankoff, J., Abowd, G.D.: Cirrin - A Word-Level Unistroke Keyboard for Pen Input. In: ACM UIST 1998, pp. 213–214 (1998)

    Google Scholar 

  47. Maybury, M.: Intelligent User Interfaces: An Introduction. In: Proc. of IUI 1999, pp. 3–4. ACM, New York (1999)

    Chapter  Google Scholar 

  48. McTear, M.F.: Spoken dialogue technology: enabling the conversational user interface. ACM Computing Surveys 34(1), 90–169 (2002)

    Article  Google Scholar 

  49. Mehrotra, S., Butts, C., Kalashnikov, D., Venkatasubramanian, N., Altintas, K., Hariharan, P., Lee, H., Ma, Y., Myers, A., Wickramasuriya, J., Eguchi, R., Huyck, C.: Camas - a Citizen Awareness System for Crisis Mitigation. In: Proc. of ACM SIGMOD 2004, New York, USA, pp. 955–956 (2004)

    Google Scholar 

  50. MESA (Mobile Broadband for Emergency and Safety Applications) project, http://www.projectmesa.org

  51. Moore, L.K.: CRS Report for Congress: Public Safety Communication Policy, Confessional Research Service, the Library of Congress (2006)

    Google Scholar 

  52. Norman, D.: Things That Make Us Smart. Addison-Wesley Publishing Co., Reading (1993)

    Google Scholar 

  53. den Os, E., Boves, L.: Towards Ambient Intelligence: Multimodal Computers that Understand Our Intentions. In: Proc. of eChallenges e-2003 (2003)

    Google Scholar 

  54. Otten, J., van Heijningen, B., Lafortune, J.F.: The Virtual Crisis Management center - An ICT implementation to canalize information! In: ISCRAM 2004, Brussels (2004)

    Google Scholar 

  55. Oviatt, S., Coulston, R., Lunsford, R.: When Do We Interact Multimodally?: Cognitive Load and Multimodal Communication Patterns. In: Proc. of ICMI 2004, pp. 129–136. ACM, New York (2004)

    Chapter  Google Scholar 

  56. Perlovsky, L.I.: Emotions, Learning and Control. In: Proc. of International Symposium: Intelligent Control, Intelligent Systems and Semiotics, pp. 131–137 (1999)

    Google Scholar 

  57. Pfleger, N., Alexandersson, J., Becker, T.: Scoring Functions for Overlay and their Application in Discourse Processing. In: KONVENS 2002, Germany, pp. 139–146 (2002)

    Google Scholar 

  58. Potamiano, A., Ammicht, E., Kuo, H.-K.J.: Dialogue Management the Bell Labs Communicator System. In: Proc. of ICSLP, Beijing, China, vol. 2, pp. 603–606 (2000)

    Google Scholar 

  59. Proxem, Proxem Antilope (2008), http://www.proxem.com

  60. Ramaswamy, S., Rogers, M., Crockett, A.D., Feaker, D., Carter, M.: WHISPER – Service Integrated Incident Management System. Int. Journal of Intelligent Control and Systems 11(2), 114–123 (2006)

    Google Scholar 

  61. Rothkrantz, L.J.M., van Vark, R.J., Peters, A., Andeweg, N.A.: Dialogue control the Alparon system. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2000. LNCS (LNAI), vol. 1902, pp. 333–338. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  62. Schank, R., Abelson, R.: Scripts, Plans, Goals and Understanding. Erlbaum, Hillsdale (1977)

    MATH  Google Scholar 

  63. Museum, S.F.: The Virtual Museum of the City of San Francisco: San Francisco 9-1-1 Dispatch Tapes, October 17 (1989), http://www.sfmuseum.org/1989/sf911.html , http://www.sfmuseum.net/1989/sc911.html

  64. Sharma, R., Yeasin, M., Krahnstoever, N., Rauschert, I., Cai, G., Brewer, I., MacEachren, A.M., Sengupta, K.: IEEE, Speech-Gesture Driven Multimodal Interfaces for Crisis Management.  91(9), 1327–1354 (2003)

    Google Scholar 

  65. Singhal, A., Rattine-Flaherty, E.: Pencils and Photos as Tools of Communicative Research and Praxis. International Communication Gazette 68(4), 313–330 (2006)

    Article  Google Scholar 

  66. Smith, P.: Toward a Practical Measure of Hypertext Usability. Interacting with Computers 8(4), 365–381 (1996)

    Article  Google Scholar 

  67. Steedman, M., Baldridge, J.: Combinatory Categorial Grammar. In: Borsley, R., Borjars, K. (eds.) Non-Transformational Syntax. Blackwell, Malden (2005)

    Google Scholar 

  68. Tatomir, B., Rothkrantz, L.J.M.: Intelligent System for Exploring Dynamic Crisis Environments. In: Van de Walle, B., Turoff, M. (eds.) ISCRAM 2006, Newark, NJ, SA (2006)

    Google Scholar 

  69. Vark, R.J.V., de Vreught, J.P.M., Rothkrantz, L.J.M.: Classification of public transport information dialogues using an information-based coding scheme. In: Proc. of ECAI, pp. 55–69 (1996)

    Google Scholar 

  70. Viola, P., Jones, M.: Robust Real-time Object Detection. In: 2nd Int. Workshop on Statistical and Computational Theories of Vision-Modeling, Learning, Computing, and Sampling (2001)

    Google Scholar 

  71. Wahlster, W.: Dialogue Systems Go Multimodal: The Smartkom experience. In: SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  72. Wallace, R.S.: The Elements of AIML Style (2003), http://alicebot.org

  73. Ward, D.A., Blackwell, A., MacKay, D.: Dasher - a Data Entry Interface Using Continuous Gesture and Language Models. In: ACM UIST, pp. 129–136. ACM, NY (2000)

    Chapter  Google Scholar 

  74. WNBC: Exclusive - 911 tapes tell horror of 9/11 (part 1 and 2): Tapes released for first time (2002), http://www.wnbc.com/news/1315646/detail.html

  75. Yang, Z., Rothkrantz, L.J.M.: Dynamic Scripting in Crisis Environments. In: HCI International 2007, China, pp. 554–563 (2007)

    Google Scholar 

  76. Zhai, S., Kristensson, P.-O.: Shorthand Writing on Stylus Keyboard. In: ACM CHI, pp. 97–104 (2003)

    Google Scholar 

  77. Zue, V., Seneff, S., Glass, J., Polifroni, J., Pao, C., Hazen, T., Hetherington, L.: Jupiter - A telephone-based conversational interface for weather information. IEEE Trans. on Speech and Audio Processing 8(1) (2000)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Man Machine Interaction, Delft University of Technology, Mekelweg 4, 2628CD, The Netherlands

    Siska Fitrianie, Zhenke Yang, Dragoş Datcu, Alin G. Chiţu & Léon J. M. Rothkrantz

  2. Faculty of Technical Sciences, Netherlands Defence Academy, Den Helder, The Netherlands

    Léon J. M. Rothkrantz

Authors
  1. Siska Fitrianie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenke Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dragoş Datcu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alin G. Chiţu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Léon J. M. Rothkrantz
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Delft Center for Systems and Control, Delft University of Technology, Mekelweg 2, 2628, Delft, CD, The Netherlands

    Robert Babuška

  2. Faculty of Science, Informatics Institute, Science Park 107, 1098, Amsterdam, XG, The Netherlands

    Frans C. A. Groen

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Fitrianie, S., Yang, Z., Datcu, D., Chiţu, A.G., Rothkrantz, L.J.M. (2010). Context-Aware Multimodal Human–Computer Interaction. In: Babuška, R., Groen, F.C.A. (eds) Interactive Collaborative Information Systems. Studies in Computational Intelligence, vol 281. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11688-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-11688-9_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-11687-2

  • Online ISBN: 978-3-642-11688-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation