short-paper

Towards large-scale data discovery: position paper

Authors:

Raul Castro Fernandez,

Ziawasch Abedjan,

Samuel Madden,

Michael StonebrakerAuthors Info & Claims

ExploreDB '16: Proceedings of the Third International Workshop on Exploratory Search in Databases and the Web

Pages 3 - 5

https://doi.org/10.1145/2948674.2948675

Published: 26 June 2016 Publication History

Get Access

Abstract

With thousands of data sources spread across multiple databases and data lakes, modern organizations face a data discovery challenge. Analysts spend more time finding relevant data to answer the questions at hand than analyzing it.

In this paper we introduce a data discovery system that facilitates locating relevant data among thousands of data sources. We represent data sources succinctly through signatures, and then create search paths that permit quick execution of a set of data discovery primitives used for finding relevant data. We have built a prototype that is being used to solve data discovery challenges of two big organizations.

References

[1]

R. Agrawal and R. Srikant. Searching with Numbers. In WWW, 2002.

Digital Library

Google Scholar

[2]

M. J. Cafarella, A. Halevy, et al. WebTables: Exploring the Power of Tables on the Web. VLDB, 2008.

Digital Library

Google Scholar

[3]

A. Das Sarma, L. Fang, et al. Finding Related Tables. In SIGMOD, 2012.

Digital Library

Google Scholar

[4]

M. Datar, N. Immorlica, et al. Locality-sensitive Hashing Scheme Based on P-stable Distributions. In SCG, 2004.

Digital Library

Google Scholar

[5]

A. Halevy, A. Rajaraman, et al. Data Integration: The Teenage Years. In VLDB, 2006.

Digital Library

Google Scholar

[6]

V. M. Megler and D. Maier. Are Data Sets Like Documents?: Evaluating Similarity-Based Ranked Search over Scientific Data. In TKDE, 2015.

Crossref

Google Scholar

[7]

V. M. Megler, D. Maier, et al. Data like this: Ranked search of genomic data vision paper. In ExploreDB, 2015.

Digital Library

Google Scholar

[8]

I. Terrizano, P. Schwarz, et al. Data Wrangling: The Challenging Journey from the Wild to the Lake. In CIDR, 2015.

Google Scholar

Cited By

View all

Feng YRahman SFeng AChen VKandogan E(2024)CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI SystemsProceedings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI10.1145/3665601.3669846(16-25)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3665601.3669846
Sadiq SAryani ADemartini GHua WIndulska MBurton-Jones AKhosravi HBenavides-Prado DSellis TSomeh IVaithianathan RWang SZhou X(2022)Information Resilience: the nexus of responsible and agile approaches to information useThe VLDB Journal10.1007/s00778-021-00720-231:5(1059-1084)Online publication date: 16-Jan-2022
https://doi.org/10.1007/s00778-021-00720-2
Atzeni PBellomarini LPapotti PTorlone R(2019)Meta-mappings for schema mapping reuseProceedings of the VLDB Endowment10.14778/3303753.330376112:5(557-569)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.14778/3303753.3303761
Show More Cited By

Recommendations

A document-based data warehousing approach for large scale data mining
ICPCA/SWS'12: Proceedings of the 2012 international conference on Pervasive Computing and the Networked World

Data mining techniques are widely applied and data warehousing is relatively important in this process. Both scalability and efficiency have always been the key issues in data warehousing. Due to the explosive growth of data, data warehousing today is ...
Towards Data Discovery by Example
Heterogeneous Data Management, Polystores, and Analytics for Healthcare
Abstract
Data scientists today have to query an avalanche of multi-source data (e.g., data lakes, company databases) for diverse analytical tasks. Data discovery is labor-intensive as users have to find the right tables, and the combination thereof to ...
Towards web search engine scale data mining
AusDM '09: Proceedings of the Eighth Australasian Data Mining Conference - Volume 101

Data mining is one of the most critical driving technologies behind Web search engines. Web search engine scale data mining posts many grand challenges, ranging from efficiency and scalability to diversity and adaptability. In this talk, I will review ...

Comments

Information & Contributors

Information

Published In

ExploreDB '16: Proceedings of the Third International Workshop on Exploratory Search in Databases and the Web

June 2016

38 pages

ISBN:9781450343121

DOI:10.1145/2948674

Editors:
Senjuti Basu Roy
University of Washington Tacoma
,
Kostas Stefanidis
University of Tampere
,
Georgia Koutrika
HP Labs
,
Mirek Riedewald
Northeastern University
,
Laks V. S. Lakshmanan
University of British Columbia

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Conference

SIGMOD/PODS'16

Sponsor:

LogicBlox

SIGMOD/PODS'16: International Conference on Management of Data

June 26 - July 1, 2016

California, San Francisco

Acceptance Rates

ExploreDB '16 Paper Acceptance Rate 5 of 11 submissions, 45%;

Overall Acceptance Rate 11 of 21 submissions, 52%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
387
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Feng YRahman SFeng AChen VKandogan E(2024)CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI SystemsProceedings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI10.1145/3665601.3669846(16-25)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3665601.3669846
Sadiq SAryani ADemartini GHua WIndulska MBurton-Jones AKhosravi HBenavides-Prado DSellis TSomeh IVaithianathan RWang SZhou X(2022)Information Resilience: the nexus of responsible and agile approaches to information useThe VLDB Journal10.1007/s00778-021-00720-231:5(1059-1084)Online publication date: 16-Jan-2022
https://doi.org/10.1007/s00778-021-00720-2
Atzeni PBellomarini LPapotti PTorlone R(2019)Meta-mappings for schema mapping reuseProceedings of the VLDB Endowment10.14778/3303753.330376112:5(557-569)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.14778/3303753.3303761
Abedjan ZBreß SMarkl VRabl TSoto J(2019)Data Management Systems Research at TU BerlinACM SIGMOD Record10.1145/3335409.333541547:4(23-28)Online publication date: 17-May-2019
https://dl.acm.org/doi/10.1145/3335409.3335415
Abedjan ZGolab LNaumann FPapenbrock T(2018)Data ProfilingSynthesis Lectures on Data Management10.2200/S00878ED1V01Y201810DTM05210:4(1-154)Online publication date: 7-Nov-2018
https://doi.org/10.2200/S00878ED1V01Y201810DTM052
Polyzotis NRoy SWhang SZinkevich M(2018)Data Lifecycle Challenges in Production Machine LearningACM SIGMOD Record10.1145/3299887.329989147:2(17-28)Online publication date: 11-Dec-2018
https://dl.acm.org/doi/10.1145/3299887.3299891
Shah VKumar AZhu X(2017)Are key-foreign key joins safe to avoid when learning high-capacity classifiers?Proceedings of the VLDB Endowment10.14778/3157794.315780411:3(366-379)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.14778/3157794.3157804
Kruse SHahn DWalter MNaumann FLim EWinslett MSanderson MFu ASun JCulpepper SLo EHo JDonato DAgrawal RZheng YCastillo CSun ATseng VLi C(2017)MetacrateProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3133180(2483-2486)Online publication date: 6-Nov-2017
https://dl.acm.org/doi/10.1145/3132847.3133180

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

A document-based data warehousing approach for large scale data mining

Towards Data Discovery by Example

Towards web search engine scale data mining