research-article

CommandSpace: modeling the relationships between tasks, descriptions and features

Authors:

Mira Dontcheva,

Gierad LaputAuthors Info & Claims

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

Pages 167 - 176

https://doi.org/10.1145/2642918.2647395

Published: 05 October 2014 Publication History

Abstract

Users often describe what they want to accomplish with an application in a language that is very different from the application's domain language. To address this gap between system and human language, we propose modeling an application's domain language by mining a large corpus of Web documents about the application using deep learning techniques. A high dimensional vector space representation can model the relationships between user tasks, system commands, and natural language descriptions and supports mapping operations, such as identifying likely system commands given natural language queries and identifying user tasks given a trace of user operations. We demonstrate the feasibility of this approach with a system, CommandSpace, for the popular photo editing application Adobe Photoshop. We build and evaluate several applications enabled by our model showing the power and flexibility of this approach.

References

[1]

Amershi, S., Fogarty, J., and Weld, D. Regroup: interactive machine learning for on-demand group creation in social networks. In CHI'12 (2012), 21--30.

Digital Library

[2]

Androutsopoulos, I., Ritchie, G. D., and Thanisch, P. Natural language interfaces to databases-an introduction. arXiv preprint cmp-lg/9503016 (1995).

[3]

Apache. Nutch. https://nutch.apache.org/. Accessed: 2014-04--15.

[4]

Branavan, S., Zettlemoyer, L. S., and Barzilay, R. Reading between the lines: Learning to map high-level instructions to commands. In ACL'10 (2010), 1268--1277.

Digital Library

[5]

Brandt, J., Dontcheva, M., Weskamp, M., and Klemmer, S. R. Example-centric programming: Integrating web search into the development environment. In CHI'10 (2010), 513--522.

Digital Library

[6]

Davison, B. D., and Hirsh, H. Predicting sequences of user actions. In AAAI/ICML Workshop on Predicting the Future (1998), 5--12.

[7]

Fourney, A., Mann, R., and Terry, M. Query-feature graphs: bridging user vocabulary and system functionality. In UIST'11, ACM (2011), 207--216.

Digital Library

[8]

Han, J., Cheng, H., Xin, D., and Yan, X. Frequent pattern mining: current status and future directions. Data Mining and Knowledge Discovery 15, 1 (2007), 55--86.

Digital Library

[9]

Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S. R. What would other programmers do: suggesting solutions to error messages. In CHI'10, ACM (2010), 1019--1028.

Digital Library

[10]

Horvitz, E., Breese, J., Heckerman, D., Hovel, D., and Rommelse, K. The lumiere project: Bayesian user modeling for inferring the goals and needs of software users. In Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc. (1998), 256--265.

Digital Library

[11]

Kohlschütter, C., Fankhauser, P., and Nejdl, W. Boilerplate detection using shallow text features. In WSDM'10, ACM (2010), 441--450.

Digital Library

[12]

Kononenko, O., Dietrich, D., Sharma, R., and Holmes, R. Automatically locating relevant programming help online. In VL/HCC'12, IEEE (2012), 127--134.

[13]

Laput, G., Adar, E., Dontcheva, M., and Li, W. Tutorial-based interfaces for cloud-enabled applications. In UIST'12, ACM (2012), 113--122.

Digital Library

[14]

Laput, G. P., Dontcheva, M., Wilensky, G., Chang, W., Agarwala, A., Linder, J., and Adar, E. Pixeltone: a multimodal interface for image editing. In CHI'13, ACM (2013), 2185--2194.

Digital Library

[15]

Lau, T., Bergman, L., Castelli, V., and Oblinger, D. Sheepdog: learning procedures for technical support. In IUI'04, ACM (2004), 109--116.

Digital Library

[16]

Le, Q. V., and Mikolov, T. Distributed representations of sentences and documents. CoRR abs/1405.4053 (2014).

[17]

Li, W., Matejka, J., Grossman, T., Konstan, J. A., and fitzmaurice, G. Design and evaluation of a command recommendation system for software applications. TOCHI 18, 2 (2011), 6.

Digital Library

[18]

Mikolov, T. word2vec, accessed 2014-04--15. https://code.google.com/p/word2vec/.

[19]

Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[20]

Mikolov, T., Yih, W.-t., and Zweig, G. Linguistic regularities in continuous space word representations. In NAACL-HLT'13 (2013), 746--751.

[21]

NLTK. Natural language toolkit. http://www.nltk.org/. Accessed: 2014-04--15.

[22]

Norman, D. A. Cognitive engineering. User centered system design (1986), 31--61.

[23]

Ratinov, L., Roth, D., Downey, D., and Anderson, M. Local and global algorithms for disambiguation to wikipedia. In NAACL-HLT'11, ACL (2011), 1375--1384.

Digital Library

[24]

Rocchio, J. J. Relevance feedback in information retrieval. In The SMART Retrieval System: Experiments in Automatic Document Processing, G. Salton, Ed. Prentice-Hall, Englewood Cliffs NJ, 1971.

[25]

Varelas, G., Voutsakis, E., Raftopoulou, P., Petrakis, E. G., and Milios, E. E. Semantic similarity methods in wordnet and their application to information retrieval on the web. In International Workshop on Web Information and Data Management, ACM (2005), 10--16.

Digital Library

[26]

Yahoo! Search BOSS. https://boss.yahoo.com/. Accessed: 2014-04--15.

[27]

Zhang, Z., and Nasraoui, O. Mining search engine query logs for query recommendation. In WWW'06, ACM (2006), 1039--1040.

Digital Library

Cited By

Masson DMalacria SCasiez GVogel D(2024)DirectGPT: A Direct Manipulation Interface to Interact with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642462(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642462
Park JO'Brien JCai CMorris MLiang PBernstein M(2023)Generative Agents: Interactive Simulacra of Human BehaviorProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606763(1-22)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606763
Kim TChoi DChoi YKim J(2022)Stylette: Styling the Web with Natural LanguageProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501931(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501931
Show More Cited By

Index Terms

CommandSpace: modeling the relationships between tasks, descriptions and features
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Evaluating the usability of natural language query languages and interfaces to Semantic Web knowledge bases

The need to make the contents of the Semantic Web accessible to end-users becomes increasingly pressing as the amount of information stored in ontology-based knowledge bases steadily increases. Natural language interfaces (NLIs) provide a familiar and ...
Towards portable natural language interfaces to knowledge bases - The case of the ORAKEL system

The customization of a natural language interface to a certain application, domain or knowledge base still represents a major effort for end users given the current state-of-the-art. In this article, we present our natural language interface ORAKEL, ...
The economics of natural language interfaces: natural language processing technology as a scarce resource

This paper discusses appropriate application areas for natural language interfaces (NLIs) to databases. This requires comparing NLIs with competing approaches, including other user-friendly interfaces, and training of users with less user-friendly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

October 2014

722 pages

ISBN:9781450330695

DOI:10.1145/2642918

General Chair:
Hrvoje Benko
Microsoft Research, USA
,
Program Chairs:
Mira Dontcheva
Adobe, USA
,
Daniel Wigdor
University of Toronto, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '14

Sponsor:

UIST '14: The 27th Annual ACM Symposium on User Interface Software and Technology

October 5 - 8, 2014

Hawaii, Honolulu, USA

Acceptance Rates

UIST '14 Paper Acceptance Rate 74 of 333 submissions, 22%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
607
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Masson DMalacria SCasiez GVogel D(2024)DirectGPT: A Direct Manipulation Interface to Interact with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642462(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642462
Park JO'Brien JCai CMorris MLiang PBernstein M(2023)Generative Agents: Interactive Simulacra of Human BehaviorProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606763(1-22)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606763
Kim TChoi DChoi YKim J(2022)Stylette: Styling the Web with Natural LanguageProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501931(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501931
Arsan DZaidi ASagar AKumar R(2021)App-Based Task Shortcuts for Virtual AssistantsThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474808(1089-1099)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474808
Li TPopowski LMitchell TMyers BKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Screen2Vec: Semantic Embedding of GUI Screens and GUI ComponentsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445049(1-15)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445049
Li TMitchell TMyers B(2021)Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning AgentsArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_15(495-537)Online publication date: 5-Nov-2021
https://doi.org/10.1007/978-3-030-82681-9_15
Liu ZLiu ZMunzner TBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Data-driven Multi-level Segmentation of Image Editing LogsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376152(1-12)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376152
Srinivasan ADontcheva MAdar EWalker SFu WPan SBrdiczka OChau PCalvary G(2019)Discovering natural language commands in multimodal interfacesProceedings of the 24th International Conference on Intelligent User Interfaces10.1145/3301275.3302292(661-672)Online publication date: 17-Mar-2019
https://dl.acm.org/doi/10.1145/3301275.3302292
Tao ZLi SWang ZFang CYang LZhao HFu YTeredesai AKumar VLi YRosales RTerzi EKarypis G(2019)Log2IntentProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3330889(1055-1063)Online publication date: 25-Jul-2019
https://dl.acm.org/doi/10.1145/3292500.3330889
Lam MYoung GXu CKrishna RBernstein MBrewster SFitzpatrick GCox AKostakos V(2019)EeveeExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312929(1-6)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290607.3312929
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents