Big data skipping in the cloud

According to today's best practices, cloud compute and storage services should be deployed and managed independently. However, this generates a problem for big data analytics in the cloud: potentially huge datasets need to be shipped from the storage service to the compute service to analyse the data. To address this, minimizing the amount of data sent across the network is critical to achieve good performance and low cost. Data skipping is a technique which achieves this for SQL style analytics on structured data.

Data skipping stores summary metadata for each object (or file) in a dataset. For each column in the object, the summary might include minimum and maximum values, a list or bloom filter of the appearing values, or other metadata which succinctly represents the data in that column. This metadata can then be indexed to support efficient retrieval, although since it can be orders of magnitude smaller than the data itself, this step may not be essential. This metadata can be used during query evaluation to skip over objects which have no relevant data. False positives for object relevance are acceptable since the query execution engine will ultimately filter the data at the row level. However false negatives must be avoided to ensure correctness of query results.

Unlike fully inverted database indexes, data skipping indexes are much smaller than the data itself. This property is critical in the cloud, since otherwise a full index scan could increase the amount of data sent across the network instead of reducing it. In the context of database systems, data skipping is used as an additional technique which complements classical indexes. It is referred to as synopsis in DB2 [6] and zone maps in Oracle [9], where in both cases it is limited to min/max metadata. Data skipping and the associated topic of data layout, has been addressed in recent research papers [7, 8] and is also used in cloud analytics platforms [3,4]. Data skipping can also be built into specific data formats [1].

We implemented data skipping support for Apache Spark SQL [2] without changing core Spark, in the form of an addon Scala library which can be added to the classpath and used in Spark applications. Our work applies to storage systems which implement the Hadoop FileSystem API, which includes various object storage systems as well as HDFS. Metadata is stored in Elasticsearch (ES) [5], and additional metadata stores can be supported in future using a pluggable API. Our approach prunes the list of candidate objects for any given Spark SQL query according to the associated data skipping metadata, stored and indexed in ES. Our technique applies to all Spark supported native formats e.g. JSON, CSV, Avro, Parquet, ORC, and can benefit from the latest optimizations built in to those formats in Spark. Unlike approaches which embed data skipping metadata inside the data format itself [1], which require reading at least part of the object, our approach avoids touching irrelevant objects altogether.