Article

Free access

The role of natural language in a multimodal interface

Author:

Philip R. CohenAuthors Info & Claims

UIST '92: Proceedings of the 5th annual ACM symposium on User interface software and technology

Pages 143 - 149

https://doi.org/10.1145/142621.142641

Published: 01 December 1992 Publication History

Abstract

Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language problems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

References

[1]

A. Chapanis, R. N. Parrish, R. B. Ochsman, and (3. D. Weeks. Studies in interactive communication: II. The effects of four communication modes on the linguistic performance of teams during cooperative problem solving. Human Factors, 19(2):101-125, April 1977.

[2]

K. Church and R. Patil. Coping with syntactic ambiguity or how to put the block in the box on the table. Amerzcan Journal of Computational Linguisizcs, 8(3-4):139-149, 1982.

Digital Library

[3]

P. R. Cohen. The pragmatics of referring and the modality of communication. Computational Linguistzcs, 10(2):97-146, April-June 1984.

Digital Library

[4]

P. R. Cohen, M. Dalrymple, D. B. Moran, F. C. N. Pereira, J. W. Sullivan, R. A. Gargan, J. L. Schlossberg, and S. W. Tyler. Synergistic use of direct manipulation and natural language. In Human Factors in Computing Systems: CHI'89 Conference Proceedings, New York, New York, April 1989. ACM, Addison Wesley Publishing Co.

Digital Library

[5]

M. Dalrymple. The interpretation of tense and aspect in English. In Proceedings of the ~6th Annual Meeting of the Association for Computational L:~nguistics, Buffalo, New York, ~lune 1988.

Digital Library

[6]

B. J. Grosz. Focusing and description in natm-M language dialogues. In A. K. 3oshi, B. Webber, and I. Sag, editors, Elements of Discourse Understanding. Cambridge University Press, 1981.

[7]

B. J. Grosz, A. K. Joshi, and S. Weinstein. Providing a unified account of definite noun phrases in discourse. In Proceedings of the 21st Ann~al Meettng of the Association for Computational Linguistics, pages 44-50, Cambridge, Massachusetts, 1983.

Digital Library

[8]

J. R. Hobbs. Resolving pronoun reference. Lingua, 44, 1978. Reprinted in Readings in Natural Language Processing, Grosz, B. J., Sparck Jones, K., and Webber, B. L. eds., Morgan Kaufman Publishers, inc., Los Altos, California, 1986.

Digital Library

[9]

E. L. Hutchins, J. D. Hollan, an'd D. A. Norman. Direct manipulation interfaces. In D. A. Norman and S. W. Draper, editors, User Centered Sysiem Design, pages 87-124. Lawrence Erlbaum Publisher, Hillsdale, New Jersey, 1986.

[10]

J. D. Mackinlay, G. G. Robertson, and S. K. Card. The perspective wall: Detail and context smoothly integrated. In S. P. Robertson, G. M. Olson, and J. S. Olson, editors, Human Factors in Computing Systems: CHI'91 Conference Proceedings, pages 173-t79, New Orleans, Lousiana, May 1991. SIGCItI, ACM Press.

Digital Library

[11]

J. D. Moore and W. R. Swartout. Pointing: A way toward explanation dialogue. In Proceedings: of the Eighth Naiional Conference on Artificial lntek ligence, pages 457-464, Cambridge, Massachusetts, July 1990. American Association for Artificial Intelligence, AAAI Press/MIT Press.

[12]

J. G. Neal and S. C. Shapiro. Intelligent multimedia interface technology. In J. W. Sullivan a,nd S. W. Tyler, editors, Intelligent User inierfaces, chapter 3, pages 45-68. ACM Press Frontier Series, Addison Wesley Publishing Co., New York, New York, 1991.

[13]

S. L. Oviatt. Pen/voice: Complementary multimodal communication, in Proceedings of Speech Tech'92, pages 238-241, New York, February 1992.

[14]

S. L. Oviatt and P. R. Cohen. Discourse structure and performance efficiency in interactive and noninteractive spoken modalities. Computer Speech and Language, 5(4):297-326, 1991a.

[15]

Fernando C. N. Pereira. Can drawing be liberated from the yon neumann style. In Proc. of 1983 A CM Database Week. ACM, 1983.

Digital Library

[16]

R. Reichman. Plain.speaking: A theory and grammar of spontaneous discourse. PhD thesis, Department of Computer Science, Harvard University, Cambridge, Massachusetts, 1981.

[17]

J. Rothenberg. Knowledge-based simulation at the RAm) Corporation. in P. A. Fishwick and R. B. Modjeski, editors, Knowledge-based Simulation, Advances in Simulation 4, pages 133--161. Springer-Verlag, New York, 1991.

Digital Library

[18]

A. D. Rubin. A theoretical taxonomy of the differences between oral and written language. In Theoretical Issues in Reading Comprehension. Lawrence Erlbaum Assocs., Hillsdale, New Jersey, 1980.

[19]

B. Shneiderman. Natural vs. precise concise languages for human operation of computers: Research issues and experimental approaches. In Proceedings of the 18th Annual Meeting of the Association for Computational Linguistics, pages 139--141, Philadelphia, Pennsylvania, June 1980.

Digital Library

[20]

Ben Shneiderman. Direct manipulation' A step beyond programming languages. IEEE Computer, 16(8)'57-69, 1983.

[21]

S. W. Tyler, J. L. Schlossberg, Jr. R. A. Gargan, L. K. Cook, and J. W. Sullivan. An intelligent interface architecture for adaptive interaction. In J. W. Sullivan and S. W. Tyler, editors, Intelligent User Interfaces, chapter 3, pages 45-68. ACM Press Frontier Series, Addison Wesley Publishing Co., New York, New York, 1991.

Digital Library

[22]

W. Wahlster. User and discourse models for multimodal communication, in J. W. Sullivan and $. W. Tyler, editors, Intelligent User Interfaces, chapter 3, pages 45-68. ACM Press Frontier Series, Addison Wesley Publishing Co., New York, New York, 1991.

Digital Library

[23]

D. Warren and F. Pereira. An efficient easily adaptable system for interpreting natural language queries. American Journal of Computational Linguistics, 8(3)'110-123, 1982.

Digital Library

[24]

B. L. Webber. So what can we talk about now? In M. Brady and R. Berwick, editors, Computational Models of Discourse. MIT Press, Cambridge, Massachusetts, 1983. Reprinted in Readings in Natural Language Processing, Grosz, B. J., Sparck Jones, K., and Webber, B. L. eds., Morgan Kaufman Publishers, inc., Los Altos, California, 1986.

Digital Library

Cited By

Zhang LPan JGettig JOney SGuo A(2024)VRCopilot: Authoring 3D Layouts with Generative AI Models in VRProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676451(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676451
Kumar SAlam MKhursheed ZBashar SKalam N(2024)Enhancing Relational Database Interaction through Open AI and Stanford Core NLP-Based on Natural Language Interface2024 5th International Conference on Recent Trends in Computer Science and Technology (ICRTCST)10.1109/ICRTCST61793.2024.10578418(589-602)Online publication date: 9-Apr-2024
https://doi.org/10.1109/ICRTCST61793.2024.10578418
Rautiainen SPantano MTraganos KAhmadi SSaenz JMohammed WMartinez Lastra J(2022)Multimodal Interface for Human–Robot CollaborationMachines10.3390/machines1010095710:10(957)Online publication date: 20-Oct-2022
https://doi.org/10.3390/machines10100957
Show More Cited By

Index Terms

The role of natural language in a multimodal interface

Recommendations

Interactive natural language interface

To override the complexity of SQL, and to facilitate the manipulation of data in databases for common people (not SQL professionals), many researches have turned out to use natural language instead of SQL. The idea of using natural language instead of ...
Transportability and generality in a natural-language interface system
IJCAI'83: Proceedings of the Eighth international joint conference on Artificial intelligence - Volume 1

This paper describes the design of a transportable natural language (NL) interface to databases and the constraints that transportability places on each component of such a system. By a transportable NL system, we mean an NL processing system that is ...
Automatic generation of graphical user interface prototypes from unrestricted natural language requirements
ASE '19: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering

High-fidelity GUI prototyping provides a meaningful manner for illustrating the developers' understanding of the requirements formulated by the customer and can be used for productive discussions and clarification of requirements and expectations. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '92: Proceedings of the 5th annual ACM symposium on User interface software and technology

December 1992

216 pages

ISBN:0897915496

DOI:10.1145/142621

Chairmen:
Jock Mackinlay
Xerox Palo Alto Research Center, Palo Alto, CA
,
Mark Green
Univ. of Alberta, Edmonton, Alta, Canada

Copyright © 1992 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 1992

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

UIST '92

Sponsor:

UIST '92: User Interface Software Technology 92

November 15 - 18, 1992

California, Monteray, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

109
Total Citations
View Citations
1,642
Total Downloads

Downloads (Last 12 months)215
Downloads (Last 6 weeks)42

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang LPan JGettig JOney SGuo A(2024)VRCopilot: Authoring 3D Layouts with Generative AI Models in VRProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676451(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676451
Kumar SAlam MKhursheed ZBashar SKalam N(2024)Enhancing Relational Database Interaction through Open AI and Stanford Core NLP-Based on Natural Language Interface2024 5th International Conference on Recent Trends in Computer Science and Technology (ICRTCST)10.1109/ICRTCST61793.2024.10578418(589-602)Online publication date: 9-Apr-2024
https://doi.org/10.1109/ICRTCST61793.2024.10578418
Rautiainen SPantano MTraganos KAhmadi SSaenz JMohammed WMartinez Lastra J(2022)Multimodal Interface for Human–Robot CollaborationMachines10.3390/machines1010095710:10(957)Online publication date: 20-Oct-2022
https://doi.org/10.3390/machines10100957
Liu JLi YGoel M(2022)A semantic-based approach to digital content placement for immersive environmentsThe Visual Computer10.1007/s00371-022-02707-839:12(5989-6003)Online publication date: 9-Nov-2022
https://doi.org/10.1007/s00371-022-02707-8
Shah APareek J(2022)Human Computer Interaction Proclivity Formed from the Analysis and Interpretation of SurveyAdvances in Information Communication Technology and Computing10.1007/978-981-19-0619-0_19(211-226)Online publication date: 10-May-2022
https://doi.org/10.1007/978-981-19-0619-0_19
Yu BSilva C(2020)FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow SystemIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2019.293466826:1(1-11)Online publication date: Jan-2020
https://doi.org/10.1109/TVCG.2019.2934668
Tumuluri RDahl DPaternò FZancanaro M(2019)Standardized representations and markup languages for multimodal interactionThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233806(347-392)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233806
Johnston M(2019)Multimodal integration for interactive conversational systemsThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233798(21-76)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233798
Hamilton WLupfer NBotello NTesch TStacy AMerrill JWilliford BBentley FKerne AMandryk RHancock MPerry MCox A(2018)Collaborative Live Media CurationProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174129(1-14)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3174129
Zamora JWrede BNagai YKomatsu THanheide MNatale L(2017)I'm Sorry, Dave, I'm Afraid I Can't Do ThatProceedings of the 5th International Conference on Human Agent Interaction10.1145/3125739.3125766(253-260)Online publication date: 17-Oct-2017
https://dl.acm.org/doi/10.1145/3125739.3125766
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents