short-paper

Open access

Sevi: Speech-to-Visualization through Neural Machine Translation

Authors:

Mourad Ouzzani,

Hongyang ChenAuthors Info & Claims

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

Pages 2353 - 2356

https://doi.org/10.1145/3514221.3520150

Published: 11 June 2022 Publication History

Abstract

Data visualization is a powerful tool for understating information through visual cues. However, allowing novices to create visualization artifacts for what they want to see is not easy, just as not everyone can write SQL queries. Arguably, the most natural way to specify what to visualize is through natural language or speech, similar to our daily search on Google or Apple Siri, leaving to the system the task of reasoning about what to visualize and how.

In this demo, we present Sevi an end-to-end data visualization system that acts as a virtual assistant to allow novices to create visualizations through either natural language or speech. Sevi is powered by two main components: Speech2Text which is based on Google Cloud Speech-to-Text Rest API, and Text2VIS, which uses an end-to-end neural machine translation model called ncNet trained using a cross-domain benchmark called nvBench. Both ncNet and nvBench have been developed by us. We will walk the audience through two general domain datasets, one related to COVID-19 and the other on NBA player statistics, to highlight how Sevi enables novices to easily create data visualizations. Because nvBench contains Text2VIS training samples from 105 domains (e.g., sport, college, hospital, etc.), the audience can play with speech or text input with any of these domains.

Supplementary Material

MP4 File (SIGMOD22-modde04.mp4)

In this video, we present Sevi, an end-to-end data visualization system that acts as a virtual assistant to allow novices to create visualizations through either natural language or speech.

Download
20.56 MB

References

[1]

Amazon's QuickSight, https://aws.amazon.com/cn/blogs/aws/amazon-quicksight-q-to-answer-ad-hoc-business-questions/.

[2]

Microsoft Power BI Q&A. https://docs.microsoft.com/en-us/power-bi/create-reports/power-bi-tutorial-q-and-a.

[3]

SpotIQ AI-Driven Insignts (2nd Edition). https://www.thoughtspot.com/resources#white_paper.

[4]

W. Cui, X. Zhang, and et al. Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. IEEE Transactions on Visualization and Computer Graphics, 26(1):906--916, 2020.

[5]

E. Dong, H. Du, and L. Gardner. An interactive web-based dashboard to track covid-19 in real time. In The Lancet infectious diseases, volume 20, 2020.

[6]

J. R. Finkel, T. Grenager, and C. D. Manning. Incorporating non-local information into information extraction systems by gibbs sampling. In ACL, 2005.

Digital Library

[7]

Y. Luo, W. Li, T. Zhao, X. Yu, L. Zhang, G. Li, and N. Tang. Deeptrack: Monitoring and exploring spatio-temporal data - A case of tracking COVID-19 -. Proc. VLDB Endow., 13(12):2841--2844, 2020.

Digital Library

[8]

Y. Luo, X. Qin, C. Chai, N. Tang, G. Li, and W. Li. Steerable self-driving data visualization. IEEE Transactions on Knowledge and Data Engineering, 2020.

Digital Library

[9]

Y. Luo, X. Qin, N. Tang, and G. Li. Deepeye: Towards automatic data visualization. In ICDE 2018, Paris, France, April 16--19, 2018, pages 101--112, 2018.

[10]

Y. Luo, X. Qin, N. Tang, G. Li, and X. Wang. Deepeye: Creating good data visualizations by keyword search. In SIGMOD, 2018.

Digital Library

[11]

Y. Luo, N. Tang, G. Li, C. Chai, W. Li, and X. Qin. Synthesizing natural language to visualization (NL2VIS) benchmarks from NL2SQL benchmarks. In SIGMOD' 21, China, June 20--25, 2021, pages 1235--1247. ACM, 2021.

[12]

Y. Luo, N. Tang, G. Li, and et al. Deepeye: A data science system for monitoring and exploring COVID-19 data. IEEE Data Eng. Bull., 2020.

[13]

Y. Luo, N. Tang, G. Li, J. Tang, C. Chai, and X. Qin. Natural language to visualization by neural machine translation. IEEE Transactions on Visualization and Computer Graphics, 28(1):217--226, 2022.

Digital Library

[14]

C. D. Manning and et al. The stanford corenlp natural language processing toolkit. In ACL, pages 55--60, 2014.

[15]

A. Narechania and et al. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. In VIS, 2020.

[16]

X. Qin, Y. Luo, N. Tang, and G. Li. Making data visualization more efficient and effective: a survey. VLDB J., 29(1):93--117, 2020.

Digital Library

[17]

A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer. Vega-lite: A grammar of interactive graphics. IEEE TVCG, 23(1):341--350, 2017.

[18]

V. Setlur, M. Tory, and A. Djalali. Inferencing underspecified natural language utterances in visual analysis. In IUI, pages 40--51, 2019.

Digital Library

[19]

A. Vaswani and et al. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, 2017.

Digital Library

[20]

B. Yu and C. T. Silva. Flowsense: A natural language interface for visual data exploration within a dataflow system. IEEE TVCG, pages 1--11, 2020.

[21]

H. Yuan and G. Li. A survey of traffic prediction: from spatio-temporal data to intelligent transportation. Data Sci. Eng., 6(1):63--85, 2021.

Cited By

Li GLi RFeng YZhang YLuo YLiu C(2024)CoInsight: Visual Storytelling for Hierarchical Tables With Connected InsightsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338855330:6(3049-3061)Online publication date: Jun-2024
https://doi.org/10.1109/TVCG.2024.3388553
Jadoon AYu CShi Y(2024)ContextMate: a context-aware smart agent for efficient data analysisCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-023-00144-7Online publication date: 16-Apr-2024
https://doi.org/10.1007/s42486-023-00144-7
Kavaz EPuig ARodríguez I(2023)Chatbot-Based Natural Language Interfaces for Data Visualisation: A Scoping ReviewApplied Sciences10.3390/app1312702513:12(7025)Online publication date: 11-Jun-2023
https://doi.org/10.3390/app13127025
Show More Cited By

Index Terms

Sevi: Speech-to-Visualization through Neural Machine Translation
1. Human-centered computing
  1. Visualization
    1. Visualization systems and tools
2. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Data analytics

Recommendations

Visualization-based improvement of neural machine translation
Abstract
We introduce a novel visual-interactive approach for analyzing, understanding, and correcting neural machine translation. Our system supports users in automatically translating documents using neural machine translation and identifying and ...
Graphical abstract

Display Omitted
Highlights
- A visual-interactive approach for analyzing, understanding, and correcting NMT.
- Visualizations show model-specific information and metrics for translation quality.
- Our approach supports both LSTM-based NMT models and the ...
Impacts of machine translation and speech synthesis on speech-to-speech translation

This paper analyzes the impacts of machine translation and speech synthesis on speech-to-speech translation systems. A typical speech-to-speech translation system consists of three components: speech recognition, machine translation and speech ...
Neural machine translation advised by statistical machine translation
AAAI'17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence

Neural Machine Translation (NMT) is a new approach to machine translation that has made great progress in recent years. However, recent studies show that NMT generally produces fluent but inadequate translations (Tu et al. 2016b; 2016a; He et al. 2016; ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

June 2022

2597 pages

ISBN:9781450392495

DOI:10.1145/3514221

General Chair:
Zachary Ives
University of Pennsylvania (USA)
,
Program Chairs:
Angela Bonifati
Lyon 1 University (France)
,
Amr El Abbadi
University of California, Santa Barbara (USA)

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

TAL Education
NSF of China
Huawei Technologies
BNRist
Zhejiang Lab?s International Talent Fund for Young Professionals

Conference

SIGMOD/PODS '22

Sponsor:

SIGMOD

SIGMOD/PODS '22: International Conference on Management of Data

June 12 - 17, 2022

PA, Philadelphia, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
899
Total Downloads

Downloads (Last 12 months)361
Downloads (Last 6 weeks)37

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li GLi RFeng YZhang YLuo YLiu C(2024)CoInsight: Visual Storytelling for Hierarchical Tables With Connected InsightsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338855330:6(3049-3061)Online publication date: Jun-2024
https://doi.org/10.1109/TVCG.2024.3388553
Jadoon AYu CShi Y(2024)ContextMate: a context-aware smart agent for efficient data analysisCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-023-00144-7Online publication date: 16-Apr-2024
https://doi.org/10.1007/s42486-023-00144-7
Kavaz EPuig ARodríguez I(2023)Chatbot-Based Natural Language Interfaces for Data Visualisation: A Scoping ReviewApplied Sciences10.3390/app1312702513:12(7025)Online publication date: 11-Jun-2023
https://doi.org/10.3390/app13127025
Luo YZhou YTang NLi GChai CShen L(2023)Learned Data-aware Image Representations of Line Charts for Similarity SearchProceedings of the ACM on Management of Data10.1145/35889421:1(1-29)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588942
Maddigan PSusnjak T(2023)Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language ModelsIEEE Access10.1109/ACCESS.2023.327419911(45181-45193)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3274199
Babaian T(2023)Exploring Design Principles for Speech-to-Visualization Data Entry InterfacesHCI International 2023 Posters10.1007/978-3-031-35998-9_3(18-23)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35998-9_3

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents