Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3314221.3314594acmconferencesArticle/Chapter ViewAbstractPublication PagespldiConference Proceedingsconference-collections
research-article
Open access

Genie: a generator of natural language semantic parsers for virtual assistant commands

Published: 08 June 2019 Publication History
  • Get Citation Alerts
  • Abstract

    To understand diverse natural language commands, virtual assistants today are trained with numerous labor-intensive, manually annotated sentences. This paper presents a methodology and the Genie toolkit that can handle new compound commands with significantly less manual effort. We advocate formalizing the capability of virtual assistants with a Virtual Assistant Programming Language (VAPL) and using a neural semantic parser to translate natural language into VAPL code. Genie needs only a small realistic set of input sentences for validating the neural model. Developers write templates to synthesize data; Genie uses crowdsourced paraphrases and data augmentation, along with the synthesized data, to train a semantic parser. We also propose design principles that make VAPL languages amenable to natural language translation. We apply these principles to revise ThingTalk, the language used by the Almond virtual assistant. We use Genie to build the first semantic parser that can support compound virtual assistants commands with unquoted free-form parameters. Genie achieves a 62% accuracy on realistic user inputs. We demonstrate Genie’s generality by showing a 19% and 31% improvement over the previous state of the art on a music skill, aggregate functions, and access control.

    Supplementary Material

    WEBM File (p394-campagna.webm)

    References

    [1]
    Tiago A. Almeida, José María G. Hidalgo, and Akebo Yamakami. 2011. Contributions to the study of SMS spam filtering. In Proceedings of the 11th ACM symposium on Document engineering - DocEng '11. ACM Press.
    [2]
    David Alvarez-Melis and Tommi S Jaakkola. 2017. Tree-structured decoding with doubly-recurrent neural networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR-2017).
    [3]
    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
    [4]
    Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse. 178–186.
    [5]
    I. Beltagy and Chris Quirk. 2016. Improved Semantic Parsers For IfThen Statements. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
    [6]
    Jonathan Berant and Percy Liang. 2014. Semantic Parsing via Paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
    [7]
    Giovanni Campagna, Michael Fischer, Mehrad Moradshahi, Silei Xu, Jackie Yang, Richard Yang, and Monica S. Lam. 2019. Almond: The Stanford Open Virtual Assistant Project. https://almond.stanford.edu .
    [8]
    Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant. In Proceedings of the 26th International Conference on World Wide Web -WWW ’17. ACM Press, New York, New York, USA, 341–350.
    [9]
    Giovanni Campagna, Silei Xu, Rakesh Ramesh, Michael Fischer, and Monica S. Lam. 2018. Controlling Fine-Grain Sharing in Natural Language with a Virtual Assistant. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (sep 2018), 1–28.
    [10]
    Rich Caruana. 1998. Multitask Learning. (1998), 95–133.
    [11]
    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. CoRR abs/1312.3005 (2013). arXiv: 1312.3005 http://arxiv.org/abs/1312.3005
    [12]
    David L Chen and Raymond J Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011). 859–865.
    [13]
    Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017. Cosette: An Automated Prover for SQL. In CIDR.
    [14]
    Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml
    [15]
    Li Dong and Mirella Lapata. 2016. Language to Logical Form with Neural Attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
    [16]
    Floydhub. 2019. Word Language Model . https://github.com/floydhub/word-language-model .
    [17]
    Winthrop Nelson Francis. 1965. A Standard Corpus of Edited Present-Day American English. Vol. 26. JSTOR. 267 pages.
    [18]
    Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT. Association for Computational Linguistics, Atlanta, Georgia, 758–764. http://cs.jhu.edu/~ccb/publications/ppdb.pdf
    [19]
    Fabio Gasparetti. 2016. Modeling user interests from web browsing activities. Data Mining and Knowledge Discovery 31, 2 (nov 2016), 502–547.
    [20]
    Anuj Kumar Goyal, Angeliki Metallinou, and Spyros Matsoukas. 2018. Fast and Scalable Expansion of Natural Language Understanding Functionality for Intelligent Agents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). Association for Computational Linguistics.
    [21]
    Kazuma Hashimoto, caiming xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
    [22]
    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. 1693–1701.
    [23]
    If This Then That 2011. If This Then That. http://ifttt.com .
    [24]
    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a Neural Semantic Parser from User Feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
    [25]
    Robin Jia and Percy Liang. 2016. Data Recombination for Neural Semantic Parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
    [26]
    Rohit J. Kate and Raymond J. Mooney. 2006. Using string-kernels for learning semantic parsers. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06. Association for Computational Linguistics.
    [27]
    Rohit J Kate, Yuk Wah Wong, and Raymond J Mooney. 2005. Learning to transform natural to formal languages. In Proceedings of the 20th national conference on Artificial intelligence-Volume 3. AAAI Press, 1062–1068.
    [28]
    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
    [29]
    Hemanth Kini and Gabby Wright. 2018. Thingpedia Spotify Skill. https: //almond.stanford.edu/thingpedia/devices/by-id/com.spotify .
    [30]
    Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, and Spyros Matsoukas. 2018. The Alexa Meaning Representation Language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). Association for Computational Linguistics.
    [31]
    Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. 2010. What is Twitter, a social network or a news media?. In WWW ’10: Proceedings of the 19th international conference on World wide web. ACM, New York, NY, USA, 591–600.
    [32]
    Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Memetracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '09. ACM Press.
    [33]
    Percy Liang. 2013. Lambda Dependency-Based Compositional Semantics. CoRR abs/1309.4408 (2013). arXiv: 1309.4408 http: //arxiv.org/abs/1309.4408
    [34]
    Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. 2016. Latent Predictor Networks for Code Generation. (2016).
    [35]
    Chang Liu, Xinyun Chen, Eui Chul Shin, Mingcheng Chen, and Dawn Song. 2016. Latent Attention For If-Then Program Synthesis. In Advances in Neural Information Processing Systems. 4574–4582.
    [36]
    Douglas MacMillan. 2018. Amazon Says It Has Over 10,000 Employees Working on Alexa, Echo. https://www.wsj.com/articles/amazonsays-it-has-over-10-000-employees-working-on-alexa-echo1542138284 . The Wall Street Journal (2018).
    [37]
    Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics.
    [38]
    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems. 6294–6305.
    [39]
    Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The Natural Language Decathlon: Multitask Learning as Question Answering. arXiv preprint arXiv:1806.08730 (2018).
    [40]
    Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černock ` y, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association.
    [41]
    Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. 2017. Attend to You: Personalized Image Captioning with Context Sequence Memory Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.
    [42]
    Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
    [43]
    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 1532–1543.
    [44]
    Vittorio Perera, Tagyoung Chung, Thomas Kollar, and Emma Strubell. 2018. Multi-task learning for parsing the alexa meaning representation language. In American Association for Artificial Intelligence (AAAI). 181–224.
    [45]
    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics.
    [46]
    Chris Quirk, Raymond Mooney, and Michel Galley. 2015. Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
    [47]
    Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract Syntax Networks for Code Generation and Semantic Parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1139–1149.
    [48]
    Prajit Ramachandran, Peter Liu, and Quoc Le. 2017. Unsupervised Pretraining for Sequence to Sequence Learning. (2017).
    [49]
    Jitesh Shetty and Jafar Adibi. 2004. The Enron email dataset database schema and brief statistical report. Information sciences institute technical report, University of Southern California 4, 1 (2004), 120–128.
    [50]
    Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929–1958.
    [51]
    Mark Steedman and Jason Baldridge. 2011. Combinatory Categorial Grammar. In Non-Transformational Syntax. Wiley-Blackwell, 181–224.
    [52]
    Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017. Building Natural Language Interfaces to Web APIs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management - CIKM '17. ACM Press.
    [53]
    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104–3112.
    [54]
    Lappoon R. Tang and Raymond J. Mooney. 2001. Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing. In Machine Learning: ECML 2001. Springer Berlin Heidelberg, 466–477.
    [55]
    Boris A. Trakhtenbrot. 1950. Impossibility of an algorithm for the decision problem in finite classes. Doklady Akademii Nauk SSSR 70 (1950), 569–572.
    [56]
    Blase Ur, Melwyn Pak Yong Ho, Stephen Brawner, Jiyun Lee, Sarah Mennicken, Noah Picard, Diane Schulze, and Michael L. Littman. 2016. Trigger-Action Programming in the Wild: An Analysis of 200,000 IFT T T Recipes. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, 3227–3231.
    [57]
    Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a Semantic Parser Overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 1332–1342.
    [58]
    Yuk Wah Wong and Raymond Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 960–967.
    [59]
    Yuk Wah Wong and Raymond J. Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics, 439–446.
    [60]
    Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequencebased Structured Prediction for Semantic Parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1341–1350.
    [61]
    Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436 (2017).
    [62]
    Jaewon Yang and Jure Leskovec. 2011. Patterns of Temporal Variation in Online Media. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM ’11). ACM, New York, NY, USA, 177–186.
    [63]
    Ziyu Yao, Xiujun Li, Jianfeng Gao, Brian Sadler, and Huan Sun. 2018. Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning. arXiv e-prints, Article arXiv:1808.06740 (Aug 2018), arXiv:1808.06740 pages. arXiv: cs.CL/1808.06740
    [64]
    Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General-Purpose Code Generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 440–450.
    [65]
    Kang Min Yoo, Youhyun Shin, and Sang-goo Lee. 2018. Data Augmentation for Spoken Language Understanding via Joint Variational Generation. CoRR abs/1809.02305 (2018). arXiv: 1809.02305 http: //arxiv.org/abs/1809.02305
    [66]
    John M Zelle and Raymond J Mooney. 1994. Inducing deterministic Prolog parsers from treebanks: A machine learning approach. In AAAI. 748–753.
    [67]
    John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the thirteenth national conference on Artificial intelligence-Volume 2. AAAI Press, 1050–1055.
    [68]
    Luke S Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence. AUAI Press, 658–666.
    [69]
    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv preprint arXiv:1709.00103 (2017).

    Cited By

    View all
    • (2024)Toward Faceted Skill Recommendation in Intelligent Personal AssistantsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645201(640-649)Online publication date: 18-Mar-2024
    • (2024)ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language ModelsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642517(1-23)Online publication date: 11-May-2024
    • (2024)More than a framework: Sketching out technical enablers for natural language-based source code generationComputer Science Review10.1016/j.cosrev.2024.10063753(100637)Online publication date: Aug-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    PLDI 2019: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation
    June 2019
    1162 pages
    ISBN:9781450367127
    DOI:10.1145/3314221
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 June 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    Author Tags

    1. data augmentation
    2. data engineering
    3. semantic parsing
    4. training data generation
    5. virtual assistants

    Qualifiers

    • Research-article

    Conference

    PLDI '19
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 406 of 2,067 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)255
    • Downloads (Last 6 weeks)19
    Reflects downloads up to 11 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Toward Faceted Skill Recommendation in Intelligent Personal AssistantsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645201(640-649)Online publication date: 18-Mar-2024
    • (2024)ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language ModelsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642517(1-23)Online publication date: 11-May-2024
    • (2024)More than a framework: Sketching out technical enablers for natural language-based source code generationComputer Science Review10.1016/j.cosrev.2024.10063753(100637)Online publication date: Aug-2024
    • (2024)Case Study in Developing Extensible Virtual Assistant Using Genie FrameworkFrontier Computing on Industrial Applications Volume 410.1007/978-981-99-9342-0_1(1-10)Online publication date: 21-Jan-2024
    • (2023)Protein Fitness Prediction Is Impacted by the Interplay of Language Models, Ensemble Learning, and Sampling MethodsPharmaceutics10.3390/pharmaceutics1505133715:5(1337)Online publication date: 25-Apr-2023
    • (2022)A Deep Learning-Based Chinese Semantic Parser for the Almond Virtual AssistantSensors10.3390/s2205189122:5(1891)Online publication date: 28-Feb-2022
    • (2022)How to manage a task-oriented virtual assistant software project: an experience report管理面向任务的虚拟助手软件系统的经验性研究Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.210046723:5(749-762)Online publication date: 25-May-2022
    • (2022)Unified Meaning Representation Format (UMRF) - A Task Description and Execution Formalism for HRIACM Transactions on Human-Robot Interaction10.1145/352258011:4(1-25)Online publication date: 8-Sep-2022
    • (2022)IDE augmented with human-learning inspired natural language programmingProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings10.1145/3510454.3516832(110-114)Online publication date: 21-May-2022
    • (2022)Fraship: A Framework to Support End-User Personalization of Smart Home Services with Runtime Knowledge GraphCompanion Proceedings of the Web Conference 202210.1145/3487553.3524710(987-995)Online publication date: 25-Apr-2022
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media