research-article

CLAMS: Bringing Quality to Data Lakes

Authors:

Mina Farid,

Alexandra Roatis,

Ihab F. Ilyas,

Hella-Franziska Hoffmann,

Xu ChuAuthors Info & Claims

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

Pages 2089 - 2092

https://doi.org/10.1145/2882903.2899391

Published: 26 June 2016 Publication History

Get Access

Abstract

With the increasing incentive of enterprises to ingest as much data as they can in what is commonly referred to as "data lakes", and with the recent development of multiple technologies to support this "load-first" paradigm, the new environment presents serious data management challenges. Among them, the assessment of data quality and cleaning large volumes of heterogeneous data sources become essential tasks in unveiling the value of big data. The coveted use of unstructured and semi-structured data in large volumes makes current data cleaning tools (primarily designed for relational data) not directly adoptable.

We present CLAMS, a system to discover and enforce expressive integrity constraints from large amounts of lake data with very limited schema information (e.g., represented as RDF triples). This demonstration shows how CLAMS is able to discover the constraints and the schemas they are defined on simultaneously. CLAMS also introduces a scale-out solution to efficiently detect errors in the raw data. CLAMS interacts with human experts to both validate the discovered constraints and to suggest data repairs.

CLAMS has been deployed in a real large-scale enterprise data lake and was experimented with a real data set of 1.2 billion triples. It has been able to spot multiple obscure data inconsistencies and errors early in the data processing stack, providing huge value to the enterprise.

References

[1]

A. Chalamalla, I. F. Ilyas, M. Ouzzani, and P. Papotti. Descriptive and Prescriptive Data Cleaning. In SIGMOD, 2014.

Digital Library

Google Scholar

[2]

S. Chaudhuri and U. Dayal. An Overview of Data Warehousing and OLAP Technology. SIGMOD Rec., 26(1):65--74, Mar. 1997.

Digital Library

Google Scholar

[3]

X. Chu, I. F. Ilyas, and P. Papotti. Discovering Denial Constraints. Proc. VLDB Endow., 6(13):1498--1509, Aug. 2013.

Digital Library

Google Scholar

[4]

X. Chu, I. F. Ilyas, and P. Papotti. Holistic Data Cleaning: Put Violations Into Context. In ICDE, 2013.

Digital Library

Google Scholar

[5]

I. F. Ilyas and X. Chu. Trends in cleaning relational data: Consistency and deduplication. Foundations and Trends in Databases, 5(4):281--393, 2015.

Digital Library

Google Scholar

Cited By

View all

Abu HKirienko AHomonenko A(2024)Method of Transition from Data Warehouses to Geographic Information System Data Lakes Based on Lambda ArchitectureIntellectual Technologies on Transport10.20295/2413-2527-2024-137-45-55(45-55)Online publication date: 14-Apr-2024
https://doi.org/10.20295/2413-2527-2024-137-45-55
Mozaffari MDignös AGamper JStörl U(2024)Self-tuning Database Systems: A Systematic Literature Review of Automatic Database Schema Design and TuningACM Computing Surveys10.1145/3665323Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665323
Fan GShraga RMiller R(2024)Gen-T: Table Reclamation in Data Lakes2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00272(3532-3545)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00272
Show More Cited By

Index Terms

CLAMS: Bringing Quality to Data Lakes
1. Information systems
  1. Data management systems
    1. Database design and models
      1. Graph-based database models
  2. Information systems applications
    1. Data mining
      1. Data cleaning

Recommendations

A Data Quality in Use model for Big Data

Beyond the hype of Big Data, something within business intelligence projects is indeed changing. This is mainly because Big Data is not only about data, but also about a complete conceptual and technological stack including raw and processed data, ...
Leveraging the Data Lake: Current State and Challenges
Big Data Analytics and Knowledge Discovery
Abstract
The digital transformation leads to massive amounts of heterogeneous data challenging traditional data warehouse solutions in enterprises. In order to exploit these complex data for competitive advantages, the data lake recently emerged as a ...
Piglet: Interactive and Platform Transparent Analytics for RDF & Dynamic Data
WWW '16 Companion: Proceedings of the 25th International Conference Companion on World Wide Web

Data analytics has gained more and more focus during recent years and many data processing platforms have been developed. They all provide a powerful but often complex API that users have to learn. Furthermore, results can only be stored or printed, ...

Comments

Information & Contributors

Information

Published In

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

June 2016

2300 pages

ISBN:9781450335317

DOI:10.1145/2882903

General Chairs:
Fatma Özcan
IBM Research, USA
,
Georgia Koutrika
HP Labs, USA
,
Program Chair:
Sam Madden
Massachusetts Institute of Technology, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'16

Sponsor:

SIGMOD

SIGMOD/PODS'16: International Conference on Management of Data

June 26 - July 1, 2016

California, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

51
Total Citations
View Citations
1,074
Total Downloads

Downloads (Last 12 months)74
Downloads (Last 6 weeks)7

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Abu HKirienko AHomonenko A(2024)Method of Transition from Data Warehouses to Geographic Information System Data Lakes Based on Lambda ArchitectureIntellectual Technologies on Transport10.20295/2413-2527-2024-137-45-55(45-55)Online publication date: 14-Apr-2024
https://doi.org/10.20295/2413-2527-2024-137-45-55
Mozaffari MDignös AGamper JStörl U(2024)Self-tuning Database Systems: A Systematic Literature Review of Automatic Database Schema Design and TuningACM Computing Surveys10.1145/3665323Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665323
Fan GShraga RMiller R(2024)Gen-T: Table Reclamation in Data Lakes2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00272(3532-3545)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00272
Diamantini CPotena DStorti E(2024)Analytic Processing in Data Lakes: A Semantic Query-Driven Discovery ApproachInformation Systems Frontiers10.1007/s10796-024-10471-4Online publication date: 14-Feb-2024
https://doi.org/10.1007/s10796-024-10471-4
Stach C(2023)Data Is the New Oil–Sort of: A View on Why This Comparison Is Misleading and Its Implications for Modern Data AdministrationFuture Internet10.3390/fi1502007115:2(71)Online publication date: 12-Feb-2023
https://doi.org/10.3390/fi15020071
Psallidas FAgrawal ASugunan CIbrahim KKaranasos KCamacho-Rodríguez JFloratou ACurino CRamakrishnan R(2023)OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance from Database Query Event LogsProceedings of the VLDB Endowment10.14778/3611540.361155516:12(3662-3675)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611555
Fan GWang JLi YZhang DMiller R(2023)Semantics-Aware Dataset Discovery from Data Lakes with Contextualized Column-Based Representation LearningProceedings of the VLDB Endowment10.14778/3587136.358714616:7(1726-1739)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.14778/3587136.3587146
Khatiwada AFan GShraga RChen ZGatterbauer WMiller RRiedewald M(2023)SANTOS: Relationship-based Semantic Table Union SearchProceedings of the ACM on Management of Data10.1145/35886891:1(1-25)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588689
Hoseini SAli AShaker HQuix CFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)SEDAR: A Semantic Data Reservoir for Heterogeneous DatasetsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614753(5056-5060)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614753
Narechania ADu FSinha ARossi RHoffswell JGuo SKoh ENavathe SEndert A(2023)DataPilot: Utilizing Quality and Usage Information for Subset Selection during Visual Data PreparationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581509(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581509
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A Data Quality in Use model for Big Data

Leveraging the Data Lake: Current State and Challenges

Piglet: Interactive and Platform Transparent Analytics for RDF & Dynamic Data

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

A Data Quality in Use model for Big Data

Leveraging the Data Lake: Current State and Challenges

Piglet: Interactive and Platform Transparent Analytics for RDF & Dynamic Data

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations