Continuous Data Ingestion and Transformation in Snowflake

Continuous data growth and the need for close-to-now analytics challenge today's data loading patterns for data warehouses. Traditionally, data is ingested into data warehouses with batch-style loading processes. This also used to be the way to for cloud-based data platforms, such as Snowflake. Data was chunked up into files which were loaded into cloud storage and registered to a table. Subsequently, derived results were fully recomputed if they were based on data sets with new data. Processing inefficiencies and cost concerns prohibited frequent ingestion and refreshes, leaving many latency-sensitive use cases unaddressed.

Over the past years, Snowflake has released and improved several features to continuously ingest and transform data. These features ease the definition of declarative data pipelines and enable our users to continuously load huge amounts of data and frequently update derived results in a cost-efficient manner. Snowflake's user interface (UI) provides detailed insights into the operational aspects of data pipelines and supports debugging and failure investigations.

With this demo, we show how to seamlessly combine Snowflake's data ingestion and transformation features to built sophisticated data pipelines that cost-efficiently process large amounts of incoming data. Our demo scenario resembles the ingestion and transformation steps of an order analytics system of a B2B retail business. We show and explain the declarative definition of the data pipeline, observe its continuous operation, and leverage Snowflake's UI to dig into details such as query plans for incremental result maintenance.