research-article

Free access

Sketch-Thru-Plan: a multimodal interface for command and control

Authors:

R. Matthews WessonAuthors Info & Claims

Communications of the ACM, Volume 58, Issue 4

Pages 56 - 65

https://doi.org/10.1145/2735589

Published: 23 March 2015 Publication History

All formats PDF

Abstract

Speaking military jargon, users can create labels and draw symbols to position objects on digitized maps.

References

[1]

Alberts, D.S. and Hayes, R.E. Understanding Command and Control. DoD Command and Control Research Program Publication Series, Washington, D.C., 2006.

Google Scholar

[2]

Bolt, R.A. Voice and gesture at the graphics interface. ACM Computer Graphics 14, 3 (1980), 262--270.

Digital Library

Google Scholar

[3]

Cheyer, A. and Julia, L. Multimodal maps: An agent-based approach. In Proceedings of the International Conference on Cooperative Multimodal Communication (Eindhoven, the Netherlands, May). Springer, 1995, 103--113.

Google Scholar

[4]

Clarkson, J.D. and Yi, J. LeatherNet: A synthetic forces tactical training system for the USMC commander. In Proceedings of the Sixth Conference on Computer Generated Forces and Behavioral Representation Technical Report IST-TR-96-18, University of Central Florida, Institute for Simulation and Training, Orlando, FL, 1996, 275--281.

Google Scholar

[5]

Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. QuickSet: Multimodal interaction for distributed applications. In Proceedings of the Fifth ACM International Conference on Multimedia (Seattle, WA, Nov. 9--13). ACM Press, New York, 1997, 31--40.

Digital Library

Google Scholar

[6]

Cohen, P.R. and McGee, D.R. Tangible multimodal interfaces for safety-critical applications. Commun. ACM 47, 1 (Jan. 2004), 41--46.

Digital Library

Google Scholar

[7]

Cohen, P.R., McGee, D., Oviatt, S., Wu, L., Clow, J., King, R., Julier, S., and Rosenblum, L. Multimodal interaction for 2D and 3D environments. IEEE Computer Graphics and Applications 19, 4 (Apr. 1999), 10--13.

Digital Library

Google Scholar

[8]

Courtemanche, A.J. and Ceranowicz, A. ModSAF development status. In Proceedings of the Fifth Conference on Computer Generated Forces and Behavioral Representation, University of Central Florida, Institute for Simulation and Training, Orlando, FL, 1995, 3--13.

Google Scholar

[9]

Dowding, J., Gawron, J.M., Appelt, D., Bear, J., Cherny, L., Moore, R., and Moran, D. Gemini: A natural language system for spoken-language understanding. In Proceedings of the 31^st Annual Meeting of the Association for Computational Linguistics (Ohio State University, Columbus, OH, June 22--26). Association for Computational Linguistics, Stroudsburg, PA, 1993, 54--61.

Digital Library

Google Scholar

[10]

Dowding, J., Frank, J., Hockey, B.A., Jonsson, A., Aist, G., and Hieronymus, J. A spoken-dialogue interface to the EUROPA planner. In Proceedings of the Third International NASA Workshop on Planning and Scheduling for Space (Washington, D.C.). NASA, 2002.

Google Scholar

[11]

Greene, H., Stotts, L., Patterson, R., and Greenburg, J. Command Post of the Future: Successful Transition of a Science and Technology Initiative to a Program of Record. Defense Acquisition University, Fort Belvoir, VA, Jan. 2010; http://www.dau.mil

Google Scholar

[12]

Hammond, T., Logsdon, D., Peschel, J., Johnston, J., Taele, P., Wolin, A., and Paulson, B. A sketch-recognition interface that recognizes hundreds of shapes in course-of-action diagrams. In Proceedings of ACM CHI Conference on Human Factors in Computing Systems (Atlanta, Apr. 10--15). ACM Press, New York, 2010, 4213--4218.

Digital Library

Google Scholar

[13]

Hyman, P. Speech-to-speech translations stutter, but researchers see mellifluous future. Commun. ACM 57, 4 (Apr. 2014), 16--19.

Digital Library

Google Scholar

[14]

Johnston, M., Cohen, P.R., McGee, D., Oviatt, S.L., Pittman, J.A., and Smith, I. Unification-based multimodal integration. In Proceedings of the 35^th Annual Meeting of the Association for Computational Linguistics and Eighth Annual Meeting of the European ACL (Madrid, Spain, July 7--12). Association for Computational Linguistics, Stroudsburg, PA, 1997, 281--288.

Digital Library

Google Scholar

[15]

Johnston, M., Bangalore, S., Varireddy, G., Stent, A., Ehlen, P., Walker, M., Whittaker, S., and Maloor, P. MATCH: An architecture for multimodal dialogue systems. In Proceedings of the 40^th Annual Meeting of the Association for Computational Linguistics (Philadelphia, PA, July). Association for Computational Linguistics, Stroudsburg, PA, 2002, 376--383.

Digital Library

Google Scholar

[16]

Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P.R., and Feiner, S. Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In Proceedings of the Seventh International Conference on Multimodal Interfaces (Trento, Italy, Oct. 4--6). ACM Press, New York, 2005, 12--19.

Digital Library

Google Scholar

[17]

Kara, L.B. and Stahovich, T.F. An image-based, trainable symbol recognizer for hand-drawn sketches. Computers and Graphics 29, 4 (2005), 501--517.

Digital Library

Google Scholar

[18]

Kumar, S., Cohen, P.R., and Coulston, R. Multimodal interaction under exerted conditions in a natural field setting. In Proceedings of the Sixth International Conference on Multimodal Interfaces (State College, PA, Oct. 13--15). ACM Press, New York, 2004, 227--234.

Digital Library

Google Scholar

[19]

MacEachren, A.M., Cai, G., Brewer, I., and Chen, J. Supporting map-based geo-collaboration through natural interfaces to large-screen display. Cartographic Perspectives 54 (Spring 2006), 16--34.

Crossref

Google Scholar

[20]

Moran, D.B., Cheyer, A.J., Julia, L.E., Martin, D.L., and Park, S. Multimodal user interfaces in the Open Agent Architecture. In Proceedings of the Second International Conference on Intelligent User Interfaces (Orlando, FL, Jan. 6--9). ACM Press, New York, 1997, 61--68.

Digital Library

Google Scholar

[21]

Myers, K., Kolojejchick, J., Angiolillo, C., Cummings, T., Garvey, T., Gervasio, M., Haines, W., Jones, C., Knittel, J., Morley, D., Ommert, W., and Potter, S. Learning by demonstration for military planning and decision making: A deployment story. In Proceedings of the 23^rd Innovative Applications of Artificial Intelligence Conference (San Francisco, CA, Aug. 6--10). AAAI Press, Menlo Park, CA, 2011, 1597--1604.

Google Scholar

[22]

O'Hara, K., Gonzalez, G., Sellen, A., Penney, G., Varnavas, A., Mentis, H., Criminisi, A., Corish, R., Rouncefield, M., Dastur, N., and Carrell, T. Touchless interaction in surgery. Commun. ACM 57, 1 (Jan. 2014), 70--77.

Digital Library

Google Scholar

[23]

Oviatt, S.L. Multimodal interfaces. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, Revised Third Edition, J. Jacko, Ed. Lawrence Erlbaum Associates, Mahwah, NJ, 2012, 405--430.

Google Scholar

[24]

Oviatt, S.L. Taming recognition errors with a multimodal architecture. Commun. ACM 43, 9 (Sept. 2000), 45--51.

Digital Library

Google Scholar

[25]

Oviatt, S.L. and Cohen, P.R. Perceptual user interfaces: Multimodal interfaces that process what comes naturally. Commun. ACM 43, 3 (Mar. 2000), 45--53.

Digital Library

Google Scholar

[26]

Oviatt, S.L. and Cohen, P.R. The Paradigm Shift to Multimodality in Contemporary Computer Interfaces. Morgan & Claypool Publishers, San Francisco, CA, 2015.

Crossref

Google Scholar

[27]

Stilman, B., Yakhnis, V., and Umanskiy, O. Strategies in large-scale problems. In Adversarial Reasoning: Computational Approaches to Reading the Opponent's Mind, A. Kott and W. McEneaney, Eds. Chapman & Hall/CRC, London, U.K., 2007, 251--285.

Google Scholar

[28]

U.S. Army. U.S. Army Field Manual 101--5-1, Chapter 5, 1997; http://armypubs.army.mil/doctrine/dr_pubs/dr_a/pdf/fm1_02c1.pdf

Google Scholar

Cited By

View all

Bråthen K(2022)Krigsspill i operasjonsplanlegging: Hva kan datasimuleringer bidra med?Scandinavian Journal of Military Studies10.31374/sjms.1295:1(309-322)Online publication date: 19-Sep-2022
https://doi.org/10.31374/sjms.129
Brathen KSeehuus RMevassvik O(2022)Simulation Support to Wargaming for Tactical Operations PlanningSimulation and Wargaming10.1002/9781119604815.ch9(223-247)Online publication date: 7-Jan-2022
https://doi.org/10.1002/9781119604815.ch9
Vukovic MStolar MLech M(2021)Cognitive Load Estimation From Speech Commands to Simulated AircraftIEEE/ACM Transactions on Audio, Speech, and Language Processing10.1109/TASLP.2021.305749229(1011-1022)Online publication date: 2021
https://doi.org/10.1109/TASLP.2021.3057492
Show More Cited By

Index Terms

Sketch-Thru-Plan: a multimodal interface for command and control

Recommendations

Demonstration of sketch-thru-plan: a multimodal interface for command and control
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

This paper demonstrates a multimodal system called Sketch-Thru-Plan (STP) that enables users to speak and draw doctrinal language and symbols in order to create courses of action. We argue that STP can meet many of the challenges inherent in building ...
Using a geometric-based sketch recognition approach to sketch Chinese radicals
AAAI'08: Proceedings of the 23rd national conference on Artificial intelligence - Volume 3

Unlike English, where unfamiliar words can be queried for its meaning by typing out its letters, the analogous operation in Chinese is far from trivial due to the nature of its written language. One approach for querying Chinese characters involve ...
Sketch express: facial expressions made easy
SBIM '11: Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling

Finding an effective control interface to manipulate complex geometric objects has traditionally relied on experienced users to place the animation controls. This process, whether for key framed or for motion captured animation, takes a lot of time and ...

Comments

Information & Contributors

Information

Published In

Communications of the ACM Volume 58, Issue 4

April 2015

86 pages

ISSN:0001-0782

EISSN:1557-7317

DOI:10.1145/2749359

Editor:
Moshe Y. Vardi
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 March 2015

Published in CACM Volume 58, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Popular
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
8,744
Total Downloads

Downloads (Last 12 months)233
Downloads (Last 6 weeks)28

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Bråthen K(2022)Krigsspill i operasjonsplanlegging: Hva kan datasimuleringer bidra med?Scandinavian Journal of Military Studies10.31374/sjms.1295:1(309-322)Online publication date: 19-Sep-2022
https://doi.org/10.31374/sjms.129
Brathen KSeehuus RMevassvik O(2022)Simulation Support to Wargaming for Tactical Operations PlanningSimulation and Wargaming10.1002/9781119604815.ch9(223-247)Online publication date: 7-Jan-2022
https://doi.org/10.1002/9781119604815.ch9
Vukovic MStolar MLech M(2021)Cognitive Load Estimation From Speech Commands to Simulated AircraftIEEE/ACM Transactions on Audio, Speech, and Language Processing10.1109/TASLP.2021.305749229(1011-1022)Online publication date: 2021
https://doi.org/10.1109/TASLP.2021.3057492
Vukovic MCavedon LThangarajah JRodriguez S(2021)Performance degrades less under increased workload with the addition of speech control in a dynamic environmentApplied Ergonomics10.1016/j.apergo.2021.10348696(103486)Online publication date: Oct-2021
https://doi.org/10.1016/j.apergo.2021.103486
Zhuo HZha YKambhampati STian X(2020)Discovering Underlying Plans Based on Shallow ModelsACM Transactions on Intelligent Systems and Technology10.1145/336827011:2(1-30)Online publication date: 24-Jan-2020
https://dl.acm.org/doi/10.1145/3368270
Wang WLiu HLin WChen YYang J(2020)Investigation on Works and Military Applications of Artificial IntelligenceIEEE Access10.1109/ACCESS.2020.30098408(131614-131625)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3009840
Johnston M(2019)Multimodal integration for interactive conversational systemsThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233798(21-76)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233798
(2019)IntroductionThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233797(1-20)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233797
Baig MKavakli M(2018)Qualitative analysis of a multimodal interface system using speech/gesture2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA)10.1109/ICIEA.2018.8398188(2811-2816)Online publication date: May-2018
https://doi.org/10.1109/ICIEA.2018.8398188
Zhuo HSingh SMarkovitch S(2017)Human-aware plan recognitionProceedings of the Thirty-First AAAI Conference on Artificial Intelligence10.5555/3298023.3298103(3686-3692)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.5555/3298023.3298103
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF Chinese translation

eReader

View online with eReader.

eReader

Digital Edition

View this article in digital edition.

Digital Edition

Magazine Site

View this article on the magazine site (external)

Magazine Site

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Demonstration of sketch-thru-plan: a multimodal interface for command and control

Using a geometric-based sketch recognition approach to sketch Chinese radicals

Sketch express: facial expressions made easy

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Digital Edition

Magazine Site

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations