research-article

Public Access

DataDiff: User-Interpretable Data Transformation Summaries for Collaborative Data Analysis

Authors:

Gunce Su Yilmaz,

Tana Wattanawaroon,

Abhishek Nigam,

Aaron J. Elmore,

Aditya ParameswaranAuthors Info & Claims

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Pages 1769 - 1772

https://doi.org/10.1145/3183713.3193564

Published: 27 May 2018 Publication History

Abstract

Interest in collaborative dataset versioning has emerged due to complex, ad-hoc, and collaborative nature of data science, and the need to record and reason about data at various stages of pre-processing, cleaning, and analysis. To support effective collaborative dataset versioning, one critical operation is differentiation : to succinctly describe what has changed from one dataset to the next. Differentiation, or diffing, allows users to understand changes between two versions, to better understand the evolution process, or to support effective merging or conflict detection across versions. We demonstrate DataDiff, a practical and concise data-diff tool that provides human-interpretable explanations of changes between datasets without reliance on the operations that led to the changes.

References

[1]

{n. d.}. dbv. ({n. d.}). https://dbv.vizuina.com/

[2]

{n. d.}. Liquibase. ({n. d.}). http://www.liquibase.org/

[3]

{n. d.}. Noms. ({n. d.}). https://github.com/attic-labs/noms

[4]

2017. Towards a Theory of Data-Diff: Optimal Synthesis of Succinct Data Modification Script. Technical Report. http://data-people.cs.illinois.edu/papers/datadiff. pdf

[5]

Azza Abouzied et al. 2013. Learning and verifying quantified boolean queries by example. In PODS. ACM, 49--60.

Digital Library

[6]

Ilsoo Ahn et al. 1986. Performance evaluation of a temporal database management system. In ACM SIGMOD Record, Vol. 15. ACM, 96--107.

Digital Library

[7]

Mohammed Al-Kateb et al. 2013. Temporal query processing in Teradata. In EDBT '13. ACM, 573--578.

Digital Library

[8]

Anant Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J Elmore, Samuel Madden, and Aditya G Parameswaran. 2015. Datahub: Collaborative data science &dataset version management at scale. CIDR (2015).

[9]

Angela Bonifati et al. 2016. Learning join queries from user examples. TODS 40, 4 (2016), 24.

Digital Library

[10]

Amit Chavan and Amol Deshpande. 2017. DEX: Query Execution in a Delta-based Storage System. In SIGMOD. ACM, 171--186.

Digital Library

[11]

Anish Das Sarma et al. 2010. Synthesizing view definitions from data. In ICDT. ACM, 89--103.

Digital Library

[12]

Joseph M Hellerstein et al. 2017. Ground: A Data Context Service. In CIDR.

[13]

Silu Huang, Liqi Xu, Jialin Liu, Aaron J. Elmore, and Aditya G. Parameswaran. 2017. OrpheusDB: Bolt-on Versioning for Relational Databases. PVLDB (2017).

Digital Library

[14]

Christian S Jensen and Richard T Snodgrass. 1999. Temporal data management. IEEE Transactions on Knowledge and Data Engineering 11, 1 (1999), 36--44.

Digital Library

[15]

Linan Jiang, Betty Salzberg, David B Lomet, and Manuel Barrena García. 2000. The BT-tree: A Branched and Temporal Access Method. In VLDB. 451--460.

Digital Library

[16]

Gad M Landau et al. 1995. Historical queries along multiple lines of time evolution. The VLDB Journal 4, 4 (1995), 703--726.

Digital Library

[17]

Michael Maddox, David Goehring, Aaron J Elmore, Samuel Madden, Aditya Parameswaran, and Amol Deshpande. 2016. Decibel: The relational dataset branching system. VLDB 9, 9 (2016), 624--635.

Digital Library

[18]

Kiril Panev and Sebastian Michel. 2016. Reverse Engineering Top-k Database Queries with PALEO. In EDBT. 113--124.

[19]

Theodoros Rekatsinas et al. 2017. HoloClean: Holistic Data Repairs with Probabilistic Inference. PVLDB 10, 11 (2017), 1190--1201.

Digital Library

[20]

Betty Joan Salzberg and David B Lomet. 1995. Branched and Temporal Index Structures. College of Computer Science, Northeastern University.

[21]

Cynthia M Saracco, Matthias Nicola, and Lenisha Gandhi. 2010. A matter of time: Temporal data management in DB2 for z/OS. IBM Corporation, New York (2010).

[22]

Q. T. Tran, C. Y. Chan, and S. Parthasarathy. 2009. Query by Output.

[23]

Meihui Zhang et al. 2013. Reverse engineering complex join queries. In SIGMOD. ACM, 809--820.

Digital Library

Cited By

Shraga RMiller R(2023)Explaining Dataset Changes for Semantic Data Versioning with Explain-Da-VProceedings of the VLDB Endowment10.14778/3583140.358316916:6(1587-1600)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.14778/3583140.3583169
Li YDeng GBai CYang JWang GZhang HBai JYuan HXu MWang S(2023)Demystifying the QoS and QoE of Edge-hosted Video Streaming Applications in the Wild with SNESetProceedings of the ACM on Management of Data10.1145/36267231:4(1-29)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626723

Index Terms

DataDiff: User-Interpretable Data Transformation Summaries for Collaborative Data Analysis
1. Information systems
  1. Data management systems
    1. Information integration
      1. Data cleaning

Recommendations

Managing versions of web documents in a transaction-time web server
WWW '04: Proceedings of the 13th international conference on World Wide Web

This paper presents a transaction-time HTTP server, called TTApache that supports document versioning. A document often consists of a main file formatted in HTML or XML and several included files such as images and stylesheets. A change to any of the ...
DyVer: Dynamic Version Handling for Array Databases
ICS '23: Proceedings of the 37th ACM International Conference on Supercomputing

Array databases are important data management systems for scientific applications. In array databases, version handling is an important problem due to the no-overwrite feature of scientific data. Existing studies for optimizing data versioning in array ...
MySQL Collaboration by Approving and Tracking Updates with Dependencies: A Versioning Approach
Computational Science and Its Applications – ICCSA 2022 Workshops
Abstract
In recent times, data science has seen a rapid increase in the need for individuals and teams to analyze and manipulate data at scale for various scientific and commercial purposes. Groups often collaboratively analyze datasets, thereby leading to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
362
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)11

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shraga RMiller R(2023)Explaining Dataset Changes for Semantic Data Versioning with Explain-Da-VProceedings of the VLDB Endowment10.14778/3583140.358316916:6(1587-1600)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.14778/3583140.3583169
Li YDeng GBai CYang JWang GZhang HBai JYuan HXu MWang S(2023)Demystifying the QoS and QoE of Edge-hosted Video Streaming Applications in the Wild with SNESetProceedings of the ACM on Management of Data10.1145/36267231:4(1-29)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626723

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents