Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3491102.3517485acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

CrossData: Leveraging Text-Data Connections for Authoring Data Documents

Published: 28 April 2022 Publication History

Abstract

Data documents play a central role in recording, presenting, and disseminating data. Despite the proliferation of applications and systems designed to support the analysis, visualization, and communication of data, writing data documents remains a laborious process, requiring a constant back-and-forth between data processing and writing tools. Interviews with eight professionals revealed that their workflows contained numerous tedious, repetitive, and error-prone operations. The key issue that we identified is the lack of persistent connection between text and data. Thus, we developed CrossData, a prototype that treats text-data connections as persistent, interactive, first-class objects. By automatically identifying, establishing, and leveraging text-data connections, CrossData enables rich interactions to assist in the authoring of data documents. An expert evaluation with eight users demonstrated the usefulness of CrossData, showing that it not only reduced the manual effort in writing data documents but also opened new possibilities to bridge the gap between data exploration and writing.

Supplementary Material

MP4 File (3491102.3517485-video-figure.mp4)
Video Figure
MP4 File (3491102.3517485-video-preview.mp4)
Video Preview

References

[1]
Katrin Affolter, Kurt Stockinger, and Abraham Bernstein. 2019. A Comparative Survey of Recent Natural Language Interfaces for Databases. VLDB Journal 28, 5 (2019), 793–819. https://doi.org/10.1007/s00778-019-00567-8 arxiv:1906.08990
[2]
Robert Amar, James Eagan, and John Stasko. 2005. Low-level Components of Analytic Activity in Information Visualization. In Proc. of InfoVis. IEEE, 111–117. https://doi.org/10.1109/INFVIS.2005.1532136
[3]
Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Andres Monroy-Hernandez, and Pourang Irani. 2017. Authoring Data-Driven Videos with DataClips. IEEE TVCG 23, 1 (2017), 501–510. https://doi.org/10.1109/TVCG.2016.2598647
[4]
Sriram Karthik Badam, Zhicheng Liu, and Niklas Elmqvist. 2019. Elastic Documents: Coupling Text and Tables through Contextual Visualizations for Enhanced Document Reading. IEEE TVCG 25, 1 (2019), 661–671. https://doi.org/10.1109/TVCG.2018.2865119
[5]
Michel Beaudouin-Lafon. 2000. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Proc. of CHI, Thea Turner and Gerd Szwillus (Eds.). ACM, 446–453. https://doi.org/10.1145/332040.332473
[6]
Sonia Bergamaschi, Francesco Guerra, Matteo Interlandi, Raquel Trillo-Lado, and Yannis Velegrakis. 2013. QUEST: A Keyword Search System for Relational Data Based on Semantic and Machine Learning Techniques. Proc. VLDB Endowment 6, 12 (2013), 1222–1225. https://doi.org/10.14778/2536274.2536281
[7]
Virginia Braun and Victoria Clarke. 2019. Reflecting on Reflexive Thematic Analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597. https://doi.org/10.1080/2159676X.2019.1628806
[8]
Tom B Brown, Jared Kaplan, Nick Ryder, Tom Henighan, Mark Chen, Ariel Herbert-voss, Daniel M Ziegler, Gretchen Krueger, Amanda Askell, Christopher Hesse, and Sam Mccandlish. 2020. Language Models are Few-Shot Learners. In Proc. of NeurIPS. Curran Associates, Inc.
[9]
Pew Research Center. 2021. What the 2020 Electorate Looks Like by Party, Race and Ethnicity, Age, Education and Religion. https://www.pewresearch.org/fact-tank/2020/10/26/what-the-2020-electorate-looks-like-by-party-race-and-ethnicity-age-education-and-religion/
[10]
Angel X. Chang, Manolis Savva, and Christopher D. Manning. 2014. Learning Spatial Knowledge for Text to 3D Scene Generation. In Proc. of EMNLP. ACL, 2028–2038. https://doi.org/10.3115/v1/d14-1217
[11]
Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas Funkhouser, and Silvio Savarese. 2018. Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings. In Proc. of ACCV, Vol. 11363. Springer, 100–116. https://doi.org/10.1007/978-3-030-20893-6_7 arXiv:1803.08495
[12]
Zhutian Chen, Wai Tong, Qianwen Wang, Benjamin Bach, and Huamin Qu. 2020. Augmenting Static Visualizations with PapARVis Designer. In Proc. of CHI. ACM, 1–12. https://doi.org/10.1145/3313831.3376436
[13]
Zhutian Chen, Yun Wang, Qianwen Wang, Yong Wang, and Huamin Qu. 2020. Towards Automated Infographic Design : Deep Learning-based Auto-Extraction of Extensible Timeline. TVCG 26(2020), 917–926. https://doi.org/10.1109/TVCG.2019.2934810
[14]
Matthew Conlen and Jeffrey Heer. 2018. Idyll: A Markup Language for Authoring and Publishing Interactive Articles on the Web. In Proc. of UIST. ACM, 29–67. https://doi.org/10.1017/9781108657846.003
[15]
Bob Coyne and Richard Sproat. 2001. WordsEye: An Automatic Text-to-Scene Conversion System. In Proc. of SIGGRAPH. ACM, 487–496. https://doi.org/10.1145/383259.383316
[16]
Weiwei Cui, Xiaoyu Zhang, Yun Wang, He Huang, Bei Chen, Lei Fang, Haidong Zhang, Jian Guan Lou, and Dongmei Zhang. 2020. Text-to-Viz: Automatic Generation of Infographics from Proportion-Related Natural Language Statements. IEEE TVCG 26, 1 (2020), 906–916. https://doi.org/10.1109/TVCG.2019.2934785
[17]
Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In Proc. of CHI. ACM. https://doi.org/10.1145/3290605.3300295
[18]
Explosion.Ai. 2021. SpaCy. https://spacy.io/
[19]
Marianela Ciolfi Felice, Nolwenn Maudet, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2016. Beyond Snapping: Persistent, Tweakable Alignment and Distribution with StickyLines. In Proc. of UIST. ACM, 133–144. https://doi.org/10.1145/2984511.2984577
[20]
National Center for Education Statistics. 2021. Condition of Education 2021. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2021144
[21]
Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie Karahalios. 2015. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proc. of UIST. ACM, 489–500. https://doi.org/10.1145/2807442.2807478
[22]
Han L. Han, Miguel A. Renom, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2020. Textlets: Supporting Constraints and Consistency in Text Documents. In Proc. of CHI. ACM, 1–13. https://doi.org/10.1145/3313831.3376804
[23]
Bilibili Inc. 2021. Announces First Quarter 2021 Financial Results. https://ir.bilibili.com/node/7621/pdf
[24]
Project Jupyter. 2021. Jupyter. https://jupyter.org/
[25]
Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: Understanding Data Visualizations via Question Answering. In Proc. of CVPR. IEEE, 5648–5656. https://doi.org/10.1109/CVPR.2018.00592
[26]
Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020. Answering Questions about Charts and Generating Visual Explanations. In Proc. of CHI. ACM. https://doi.org/10.1145/3313831.3376467
[27]
Dae Hyun Kim, Enamul Hoque, Juho Kim, and Maneesh Agrawala. 2018. Facilitating Document Reading by Linking Text and Tables. In Proc. of UIST. ACM, 423–434. https://doi.org/10.1145/3242587.3242617
[28]
Nam Wook Kim, Nathalie Henry Riche, Benjamin Bach, Guanpeng Xu, Matthew Brehmer, Ken Hinckley, Michel Pahud, Haijun Xia, Michael J Mcguffin, Hanspeter Pfister, and Mathew Brehmer. 2019. DataToon: Drawing Data Comics About Dynamic Networks with Pen + Touch Interaction. CHI (2019), 12. https://doi.org/10.1145/3290605.3300335
[29]
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual Constituency Parsing with Self-Attention and Pre-Training. In Proc. of ACL. ACM, 3499–3505. https://doi.org/doi.org/10.18653/v1/p19-1340
[30]
Donald E. Knuth. 1984. Literate Programming. Comput. J. 27, 2 (1984), 97–111. https://doi.org/10.1093/comjnl/27.2.97
[31]
Nicholas Kong, Marti A. Hearst, and Maneesh Agrawala. 2014. Extracting References Between Text and Charts via Crowdsourcing. In Proc. of CHI. ACM, 31–40. https://doi.org/10.1145/2556288.2557241
[32]
Chufan Lai, Zhixian Lin, Ruike Jiang, Yun Han, Can Liu, and Xiaoru Yuan. 2020. Automatic Annotation Synchronizing with Textual Description for Visualization. In Proc. of CHI. ACM, 1–13. https://doi.org/10.1145/3313831.3376443
[33]
Shahid Latif, Diao Liu, and Fabian Beck. 2018. Exploring Interactive Linking Between Text and Visualization. In Proc. of EuroVis. Eurographics Association. https://doi.org/10.2312/eurovisshort.20181084
[34]
Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, and Nam Wook Kim. 2021. Kori: Interactive Synthesis of Text and Charts in Data Documents. IEEE TVCG (2021). arXiv:2108.04203http://arxiv.org/abs/2108.04203
[35]
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end Neural Coreference Resolution. In Proc. of ACL. ACL, 188–197. https://doi.org/10.18653/v1/D17-1018
[36]
Can Liu, Liwenhan Xie, Yun Han, Datong Wei, and Xiaoru Yuan. 2020. AutoCaption: An Approach to Generate Natural Language Description from Visualization Automatically. In Proc. of PacificVis. IEEE, 191–195.
[37]
Zhicheng Liu, John Thompson, Alan Wilson, Mira Dontcheva, James Delorey, Sam Grigg, Bernard Kerr, and John Stasko. 2018. Data Illustrator: Augmenting Vector Design Tools with Lazy Data Binding for Expressive Visualization Authoring. Conference on Human Factors in Computing Systems - Proceedings 2018-April (2018), 1–13. https://doi.org/10.1145/3173574.3173697
[38]
Tanya Marwah, Gaurav Mittal, and Vineeth N. Balasubramanian. 2017. Attentive Semantic Video Generation Using Captions. In Proc. of ICCV, Vol. 2017-Octob. IEEE, 1435–1443. https://doi.org/10.1109/ICCV.2017.159 arxiv:1708.05980
[39]
Arpit Narechania, Arjun Srinivasan, and John Stasko. 2021. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries. IEEE TVCG 27, 2 (2021), 369–379. https://doi.org/10.1109/TVCG.2020.3030378 arXiv:2008.10723
[40]
National Institues of Health. 2021. Impact of NIH Research. https://www.nih.gov/about-nih/what-we-do/impact-nih-research/our-knowledge
[41]
Bureau of Labor Statistics. 2021. Consumer Price Index News Release. https://www.bls.gov/news.release/cpi.htm
[42]
California Department of Public Health. 2018. California Student Tobacco Survey 2017-18. https://www.cdph.ca.gov/Programs/CCDPHP/DCDIC/CTCB/CDPH Document Library/ResearchandEvaluation/Reports/2017-18CaliforniaStudentTobaccoSurveyBiennialReport.pdf
[43]
World Health Organization. 2021. Weekly epidemiological update on COVID-19 - 18 May 2021. https://www.who.int/publications/m/item/weekly-epidemiological-update-on-covid-19---18-may-2021
[44]
Fatma Özcan, Abdul Quamar, Jaydeep Sen, Chuan Lei, and Vasilis Efthymiou. 2020. State of the Art and Open Challenges in Natural Language Interfaces to Data. In Proc. of SIGMOD. ACM, 2629–2636. https://doi.org/10.1145/3318464.3383128
[45]
Studio R. 2021. R Markdown. https://rmarkdown.rstudio.com/index.html
[46]
Donghao Ren, Bongshin Lee, and Matthew Brehmer. 2019. Charticulator: Interactive Construction of Bespoke Chart Layouts. IEEE TVCG 25, 1 (2019), 789–799. https://doi.org/10.1109/TVCG.2018.2865158
[47]
Steve Rubin, Floraine Berthouzoz, Gautham J. Mysore, Wilmot Li, and Maneesh Agrawala. 2012. UnderScore: Musical Underlays for Audio Stories. In Proc. of UIST. ACM, 359–366.
[48]
Diptikalyan Saha, Avrilia Floratou, Karthik Sankaranarayanan, Umar Farooq Minhas, Ashish R. Mittal, and Fatma Özcan. 2016. ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores Diptikalyan. Proc. VLDB Endowment 9, 12 (2016), 1209—-1220. https://doi.org/10.4324/9780203932148
[49]
Arvind Satyanarayan and Jeffrey Heer. 2014. Lyra: An Interactive Visualization Design Environment. CGF 33, 3 (2014), 351–360. https://doi.org/10.1111/cgf.12391
[50]
Vidya Setlur, Sarah E. Battersby, Melanie Tory, Rich Gossweiler, and Angel X. Chang. 2016. Eviza: A Natural Language Interface for Visual Analysis. In Proc. of UIST. ACM, 365–377. https://doi.org/10.1145/2984511.2984588
[51]
Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles. CoRR abs/1809.0(2018), 1–11. arxiv:1809.05054http://arxiv.org/abs/1809.05054
[52]
Ben Shneiderman. 1982. The Future of Interactive Systems and the Emergence of Direct Manipulation. Behaviour and Information Technology 1, 3 (1982), 237–256. https://doi.org/10.1080/01449298208914450
[53]
Arjun Srinivasan, Mira Dontcheva, Eytan Adar, and Seth Walker. 2019. Discovering Natural Language Commands in Multimodal Interfaces. In Proc. of IUI. ACM. https://doi.org/10.1145/3301275.3302292
[54]
Arjun Srinivasan, Bongshin Lee, Nathalie Henry Riche, Steven M. Drucker, and Ken Hinckley. 2020. InChorus: Designing Consistent Multimodal Interactions for Data Visualization on Tablet Devices. In Proc. of CHI. ACM, 1–13. https://doi.org/10.1145/3313831.3376782 arXiv:2001.06423
[55]
Arjun Srinivasan, Nikhila Nyapathy, and Bongshin Lee. 2021. Collecting and Characterizing Natural Language Uterances for Specifying Data Visualizations. In Proc. of CHI. ACM. https://doi.org/10.1145/3411764.3445400
[56]
Nicole Sultanum, Zoya Bylinskii, and Zhicheng Liu. 2021. Leveraging Text-Chart Links to Support Authoring of Data-Driven Articles with Vizflow. In Proc. of CHI. ACM. https://doi.org/10.1145/3411764.3445354
[57]
Anh Truong, Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2016. QuickCut: An Interactive Tool for Editing Narrated Video. In Proc. of UIST. ACM, 497–507. https://doi.org/10.1145/2984511.2984569
[58]
Prasetya Utama, Nathaniel Weir, Fuat Basik, Carsten Binnig, Ugur Cetintemel, Benjamin Hättasch, Amir Ilkhechi, Shekar Ramaswamy, and Arif Usta. 2018. An End-to-end Neural Natural Language Interface for Databases. CoRR abs/1804.0(2018). arxiv:1804.00401http://arxiv.org/abs/1804.00401
[59]
Bret Victor. 2011. Explorable Explanations. http://worrydream.com/ExplorableExplanations/
[60]
Bret Victor. 2013. Drawing Dynamic Visualizations. http://worrydream.com/DrawingDynamicVisualizationsTalkAddendum/
[61]
Haijun Xia. 2020. Crosspower: Bridging Graphics and Linguistics. In Proc. of UIST. ACM, 722–734. https://doi.org/10.1145/3379337.3415845
[62]
Haijun Xia, Bruno Araujo, Tovi Grossman, and Daniel Wigdor. 2016. Object-Oriented Drawing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4610–4621. https://doi.org/10.1145/2858036.2858075
[63]
Haijun Xia, Nathalie Henry Riche, Fanny Chevalier, Bruno De Araujo, and Daniel Wigdor. 2018. DataInk: Direct and Creative Data-Oriented Drawing. In CHI. ACM, 223. https://doi.org/10.1145/3173574.3173797
[64]
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR (2017), 1–12. arxiv:1709.00103http://arxiv.org/abs/1709.00103

Cited By

View all
  • (2024)WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code VisualizationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676374(1-14)Online publication date: 13-Oct-2024
  • (2024)"The Data Says Otherwise" — Towards Automated Fact-checking and Communication of Data ClaimsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676359(1-20)Online publication date: 13-Oct-2024
  • (2024)DataDive: Supporting Readers' Contextualization of Statistical Statements with Data ExplorationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645155(623-639)Online publication date: 18-Mar-2024
  • Show More Cited By

Index Terms

  1. CrossData: Leveraging Text-Data Connections for Authoring Data Documents
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
        April 2022
        10459 pages
        ISBN:9781450391573
        DOI:10.1145/3491102
        This work is licensed under a Creative Commons Attribution International 4.0 License.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 28 April 2022

        Check for updates

        Author Tags

        1. Data Document
        2. Interactive Article
        3. Language-oriented Authoring
        4. Natural Language Processing
        5. Text-based Editing

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        CHI '22
        Sponsor:
        CHI '22: CHI Conference on Human Factors in Computing Systems
        April 29 - May 5, 2022
        LA, New Orleans, USA

        Acceptance Rates

        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)474
        • Downloads (Last 6 weeks)49
        Reflects downloads up to 13 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code VisualizationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676374(1-14)Online publication date: 13-Oct-2024
        • (2024)"The Data Says Otherwise" — Towards Automated Fact-checking and Communication of Data ClaimsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676359(1-20)Online publication date: 13-Oct-2024
        • (2024)DataDive: Supporting Readers' Contextualization of Statistical Statements with Data ExplorationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645155(623-639)Online publication date: 18-Mar-2024
        • (2024)Epigraphics: Message-Driven Infographics AuthoringProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642172(1-18)Online publication date: 11-May-2024
        • (2024)Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641969(1-20)Online publication date: 11-May-2024
        • (2024)More Than Data Stories: Broadening the Role of Visualization in Contemporary JournalismIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328758530:8(5240-5259)Online publication date: Aug-2024
        • (2024)A Qualitative Interview Study of Distributed Tracing Visualisation: A Characterisation of Challenges and OpportunitiesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324159630:7(3828-3840)Online publication date: Jul-2024
        • (2024)XNLI: Explaining and Diagnosing NLI-Based Visual Data AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324000330:7(3813-3827)Online publication date: Jul-2024
        • (2023)Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math TextbooksProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606827(1-16)Online publication date: 29-Oct-2023
        • (2023)Graphologue: Exploring Large Language Model Responses with Interactive DiagramsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606737(1-20)Online publication date: 29-Oct-2023
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media