Correlation Sketches for Approximate Join-Correlation Queries

The growing number of available structured datasets, from Web tables and open-data portals to enterprise data, open up new opportunities to enrich analytics and improve machine learning models through data augmentation. In this paper, we introduce a new class of augmentation queries, join-correlation queries, which given a column $Q$ and a join column $K_Q$ from a query table $\mathcal{T}_Q$, retrieve tables $\mathcal{T}_X$ in a dataset collection (or data lake) $\cal{D}$ such that $\mathcal{T}_X$ is joinable with $\mathcal{T}_Q$ on $K_Q$ and there is a column $C \in \mathcal{T}_X$ such that $Q$ is correlated with $C$. A straightforward approach to evaluate these queries is to first find joinable tables, and then to explicitly compute correlations between $Q$ and all columns of the discovered tables. However, for queries over large collections or that return large tables, doing so for many candidate results is prohibitively expensive. To efficiently support correlation column discovery, we 1) propose a new sketching method that enables the construction of an index for a large number of tables and that provides accurate estimates for join-correlation queries, and 2) explore different scoring strategies that effectively rank the query results based on how well the columns are correlated with the query. We carry out a detailed experimental evaluation, using both synthetic and real data, which shows that our sketching method attains high accuracy and the scoring strategies lead to high-quality rankings.