research-article

DIY: Assessing the Correctness of Natural Language to SQL Systems

Authors:

Arpit Narechania,

Gonzalo RamosAuthors Info & Claims

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Pages 597 - 607

https://doi.org/10.1145/3397481.3450667

Published: 14 April 2021 Publication History

Abstract

Designing natural language interfaces for querying databases remains an important goal pursued by researchers in natural language processing, databases, and HCI. These systems receive natural language as input, translate it into a formal database query, and execute the query to compute a result. Because the responses from these systems are not always correct, it is important to provide people with mechanisms to assess the correctness of the generated query and computed result. However, this assessment can be challenging for people who lack expertise in query languages. We present Debug-It-Yourself (DIY), an interactive technique that enables users to assess the responses from a state-of-the-art natural language to SQL (NL2SQL) system for correctness and, if possible, fix errors. DIY provides users with a sandbox where they can interact with (1) the mappings between the question and the generated query, (2) a small-but-relevant subset of the underlying database, and (3) a multi-modal explanation of the generated query. End-users can then employ a back-of-the-envelope calculation debugging strategy to evaluate the system’s response. Through an exploratory study with 12 users, we investigate how DIY helps users assess the correctness of the system’s answers and detect & fix errors. Our observations reveal the benefits of DIY while providing insights about end-user debugging strategies and underscore opportunities for further improving the user experience.

References

[1]

J. Berant, D. Deutch, A. Globerson, T. Milo, and T. Wolfson. 2019. Explaining Queries Over Web Tables to Non-experts. In 2019 IEEE 35th International Conference on Data Engineering (ICDE). 1570–1573. https://doi.org/10.1109/ICDE.2019.00144

[2]

Sonia Bergamaschi, Francesco Guerra, Matteo Interlandi, Raquel Trillo Lado, Yannis Velegrakis, 2013. QUEST: a keyword search system for relational data based on semantic and machine learning techniques. (2013).

[3]

Lukas Blunschi, Claudio Jossen, Donald Kossmann, Magdalini Mori, and Kurt Stockinger. 2012. Soda: Generating sql for business users. Proceedings of the VLDB Endowment 5, 10 (2012), 932–943.

Digital Library

[4]

Ben Bogin, Matt Gardner, and Jonathan Berant. 2019. Global Reasoning over Database Structures for Text-to-SQL Parsing. arxiv:1908.11214 [cs.CL]

[5]

Ben Bogin, Matt Gardner, and Jonathan Berant. 2019. Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing. arxiv:1905.06241 [cs.CL]

[6]

John Brooke. 2013. SUS: a retrospective. Journal of usability studies 8, 2 (2013), 29–40.

Digital Library

[7]

DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases. arxiv:2004.03125 [cs.CL]

[8]

Jonathan Danaparamita and Wolfgang Gatterbauer. 2011. QueryViz: helping users understand SQL queries and their patterns. In Proceedings of EDBT. 558–561.

Digital Library

[9]

Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. 2020. Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback. arXiv:2005.02539 (2020).

[10]

Jake Feasel. 2021. SQL Fiddle. http://sqlfiddle.com/, accessed 2021-01-01.

[11]

Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. In Proceedings of ACM UIST. 489–500.

Digital Library

[12]

Barbara J Grosz, Douglas E Appelt, Paul A Martin, and Fernando CN Pereira. 1987. TEAM: an experiment in the design of transportable natural-language interfaces. Artificial Intelligence 32, 2 (1987), 173–243.

Digital Library

[13]

Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. arxiv:1905.08205 [cs.CL]

[14]

Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-SQL: reinforce schema representation with context. arXiv:1908.08113 (2019).

[15]

Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. TAPAS: Weakly Supervised Table Parsing via Pre-training. arXiv:2004.02349 (2020).

[16]

Enamul Hoque, Vidya Setlur, Melanie Tory, and Isaac Dykeman. 2018. Applying pragmatics principles for interaction with visual analytics. IEEE TVCG 24, 1 (2018), 309–318.

[17]

Mohd Ibrahim and Rodina Ahmad. 2010. Class diagram extraction from textual requirements using natural language processing (NLP) techniques. In 2010 Second International Conference on Computer Research and Development. IEEE, 200–204.

Digital Library

[18]

Jan-Frederik Kassel and Michael Rohs. 2018. Valletto: A multi-modal Interface for Ubiquitous Visual Analytics. In ACM CHI ’18 Extended Abstracts.

Digital Library

[19]

Esther Kaufmann, Abraham Bernstein, and Lorenz Fischer. 2007. NLP-Reduce: A naive but domainindependent natural language interface for querying ontologies. In 4th European Semantic Web Conference ESWC. 1–2.

[20]

Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. 2020. Bertrand-DR: Improving Text-to-SQL using a Discriminative Re-ranker. arxiv:2002.00557 [cs.CL]

[21]

Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020. Answering Questions about Charts and Generating Visual Explanations. In Proceedings of ACM CHI. :1–:13.

Digital Library

[22]

Andreas Kokkalis, Panagiotis Vagenas, Alexandros Zervakis, Alkis Simitsis, Georgia Koutrika, and Yannis Ioannidis. 2012. Logos: a system for translating queries into narratives. In Proceedings of ACM SIGMOD. 673–676.

Digital Library

[23]

G. Koutrika, A. Simitsis, and Y. E. Ioannidis. 2010. Explaining structured queries in natural language. In 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010). 333–344.

[24]

Dongjun Lee. 2019. Clause-Wise and Recursive Decoding for Complex and Cross-Domain Text-to-SQL Generation. arxiv:1904.08835 [cs.CL]

[25]

Aristotelis Leventidis, Jiahui Zhang, Cody Dunne, Wolfgang Gatterbauer, HV Jagadish, and Mirek Riedewald. 2020. QueryVis: Logic-based diagrams help users understand complicated SQL queries faster. In Proceedings of ACM SIGMOD. 2303–2318.

Digital Library

[26]

Fei Li and Hosagrahar V Jagadish. 2014. NaLIR: an interactive natural language interface for querying relational databases. In Proceedings of ACM SIGMOD. 709–712.

Digital Library

[27]

Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based Neural Text-to-SQL Generation. arxiv:1905.13326 [cs.CL]

[28]

A. Narechania, A. Srinivasan, and J. Stasko. 2021. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries. IEEE TVCG 27, 2 (2021), 369–379. https://doi.org/10.1109/TVCG.2020.3030378

[29]

Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of ACL IJCNLP. 1470–1480.

[30]

Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of ACM IUI. ACM, 149–157.

[31]

Vidya Setlur, Sarah E Battersby, Melanie Tory, Rich Gossweiler, and Angel X Chang. 2016. Eviza: A natural language interface for visual analysis. In Proceedings of ACM UIST. 365–377.

Digital Library

[32]

Vidya Setlur, Melanie Tory, and Alex Djalali. 2019. Inferencing underspecified natural language utterances in visual analysis. In Proceedings of ACM IUI. 40–51.

Digital Library

[33]

Alkis Simitsis and Yannis Ioannidis. 2009. DBMSs should talk back too. arXiv:0909.1786 (2009).

[34]

Arjun Srinivasan and John Stasko. 2018. Orko: Facilitating multi-modal interaction for visual exploration and analysis of networks. IEEE TVCG 24, 1 (2018), 511–521.

[35]

Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017. Building natural language interfaces to web apis. In Proceedings of ACM CIKM. 177–186.

Digital Library

[36]

Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, and Ryen W White. 2018. Natural language interfaces with fine-grained user interaction: A case study on web apis. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 855–864.

Digital Library

[37]

Yiwen Sun, Jason Leigh, Andrew Johnson, and Sangyoon Lee. 2010. Articulate: A semi-automated model for translating natural language queries into meaningful visualizations. In Proceedings of the International Symposium on Smart Graphics. 184–195.

[38]

W3Schools. [n.d.]. SQL Tryit Editor v1.6. https://www.w3schools.com/sql/trysql.asp?filename=trysql_select_all, accessed 2020-12-29.

[39]

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of ACL. Online, 7567–7578.

[40]

Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust Text-to-SQL generation with execution-guided decoding. arXiv:1807.03100 (2018).

[41]

Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. arxiv:1711.04436 [cs.CL]

[42]

Bowen Yu and Cláudio T Silva. 2019. FlowSense: A natural language interface for visual data exploration within a dataflow system. IEEE TVCG 26, 1 (2019), 1–11.

[43]

Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018. SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task. arxiv:1810.05237 [cs.CL]

[44]

Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S Lasecki, and Dragomir Radev. 2019. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. arxiv:1909.05378 [cs.CL]

[45]

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv:1809.08887 (2018).

[46]

Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. SParC: Cross-Domain Semantic Parsing in Context. arxiv:1906.02285 [cs.CL]

[47]

Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions. arxiv:1909.00786 [cs.CL]

[48]

Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR abs/1709.00103(2017).

Cited By

Xie LZheng CXia HQu HZhu-Tian C(2024)WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code VisualizationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676374(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676374
Tian YKummerfeld JLi TZhang T(2024)SQLucid: Grounding Natural Language Database Queries with Interactive ExplanationsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676368(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676368
Ning ZTian YZhang ZZhang TLi T(2024)Insights into Natural Language Database Query Errors: from Attention Misalignment to User Handling StrategiesACM Transactions on Interactive Intelligent Systems10.1145/365011414:4(1-32)Online publication date: 2-Mar-2024
https://dl.acm.org/doi/10.1145/3650114
Show More Cited By

Index Terms

DIY: Assessing the Correctness of Natural Language to SQL Systems
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
2. Information systems
  1. Data management systems
    1. Query languages

Index terms have been assigned to the content through auto-classification.

Recommendations

Is natural language querying practical?

The study reported here involved the use of the natural language query system INTELLECT. It evaluated the level of correct interpretation to investigate whether the use of such a system is practical. Two sets of queries generated by two groups of senior-...
Natural language querying in SAP-ERP platform
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

With the omnipresence of mobile devices coupled with recent advances in automatic speech recognition capabilities, there has been a growing demand for natural language query (NLQ) interface to retrieve information from the knowledge bases. Business ...
NALSD: A Natural Language Interface for Spatial Databases
SSTD '23: Proceedings of the 18th International Symposium on Spatial and Temporal Data

Spatial databases have a wide range of applications such as urban planning, engineering management and data visualization for epidemic investigation. The number of users in spatial databases becomes significantly large due to the increasing demand of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

April 2021

618 pages

ISBN:9781450380171

DOI:10.1145/3397481

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IUI '21

Sponsor:

IUI '21: 26th International Conference on Intelligent User Interfaces

April 14 - 17, 2021

TX, College Station, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
465
Total Downloads

Downloads (Last 12 months)149
Downloads (Last 6 weeks)12

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie LZheng CXia HQu HZhu-Tian C(2024)WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code VisualizationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676374(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676374
Tian YKummerfeld JLi TZhang T(2024)SQLucid: Grounding Natural Language Database Queries with Interactive ExplanationsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676368(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676368
Ning ZTian YZhang ZZhang TLi T(2024)Insights into Natural Language Database Query Errors: from Attention Misalignment to User Handling StrategiesACM Transactions on Interactive Intelligent Systems10.1145/365011414:4(1-32)Online publication date: 2-Mar-2024
https://dl.acm.org/doi/10.1145/3650114
Paden JNarechania AEndert A(2024)BiasBuzz: Combining Visual Guidance with Haptic Feedback to Increase Awareness of Analytic Behavior during Visual Data AnalysisExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651064(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651064
Gao JGebreegziabher SChoo KLi TPerrault SMalone T(2024)A Taxonomy for Human-LLM Interaction Modes: An Initial ExplorationExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650786(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650786
Ko HJeon HPark GKim DKim NKim JSeo J(2024)Natural Language Dataset Generation Framework for Visualizations Powered by Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642943(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642943
Huang YZhou YChen RPan CShu XWeng DWu Y(2024)Interactive Table Synthesis With Natural LanguageIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332912030:9(6130-6145)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3329120
Feng YWang XPan BWong KRen YLiu SYan ZMa YQu HChen W(2024)XNLI: Explaining and Diagnosing NLI-Based Visual Data AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324000330:7(3813-3827)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3240003
Ning ZZhang ZSun TTian YZhang TLi T(2023)An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database QueriesProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584067(633-649)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584067
Ruoff MMyers BMaedche A(2023)ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task LearningProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580964(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580964
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents