research-article

Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark

Authors:

Matei ZahariaAuthors Info & Claims

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Pages 601 - 613

https://doi.org/10.1145/3183713.3190664

Published: 27 May 2018 Publication History

Get Access

Abstract

With the ubiquity of real-time data, organizations need streaming systems that are scalable, easy to use, and easy to integrate into business applications. Structured Streaming is a new high-level streaming API in Apache Spark based on our experience with Spark Streaming. Structured Streaming differs from other recent streaming APIs, such as Google Dataflow, in two main ways. First, it is a purely declarative API based on automatically incrementalizing a static relational query (expressed using SQL or DataFrames), in contrast to APIs that ask the user to build a DAG of physical operators. Second, Structured Streaming aims to support end-to-end real-time applications that integrate streaming with batch and interactive analysis. We found that this integration was often a key challenge in practice. Structured Streaming achieves high performance via Spark SQL's code generation engine and can outperform Apache Flink by up to 2x and Apache Kafka Streams by 90x. It also offers rich operational features such as rollbacks, code updates, and mixed streaming/batch execution. We describe the system's design and use cases from several hundred production deployments on Databricks, the largest of which process over 1 PB of data per month.

References

[1]

Daniel J. Abadi, Yanif Ahmad, Magdalena Balazinska, Mitch Cherniack, Jeong hyon Hwang, Wolfgang Lindner, Anurag S. Maskey, Er Rasin, Esther Ryvkina, Nesime Tatbul, Ying Xing, and Stan Zdonik. 2005. The design of the borealis stream processing engine In CIDR. 277--289.

Google Scholar

[2]

Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, and Sam Whittle. 2015. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-scale, Unbounded, Out-of-order Data Processing. Proc. VLDB Endow., Vol. 8, 12 (Aug. 2015), 1792--1803. /10.1145/223784.223848

Digital Library

Google Scholar

Cited By

View all

Rigas STzouveli PKollias S(2024)An End-to-End Deep Learning Framework for Fault Detection in Marine MachinerySensors10.3390/s2416531024:16(5310)Online publication date: 16-Aug-2024
https://doi.org/10.3390/s24165310
Saravanan SBalasubramanian U(2024)An Adaptive Scalable Data Pipeline for Multiclass Attack Classification in Large-Scale IoT NetworksBig Data Mining and Analytics10.26599/BDMA.2023.90200277:2(500-511)Online publication date: Jun-2024
https://doi.org/10.26599/BDMA.2023.9020027
Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Show More Cited By

Index Terms

Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark
1. Information systems
  1. Data management systems
    1. Database design and models
      1. Data model extensions
        Data streams
    2. Database management system engines
      1. Stream management

Recommendations

Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, or minimizing memory usage. However, each technique operates under different assumptions ...
Experimental study on the performance and resource utilization of data streaming frameworks
CCGrid '18: Proceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

With the advent of the Internet of Things (IoT), data stream processing have gained increased attention due to the ever-increasing need to process heterogeneous and voluminous data streams. This work addresses the problem of selecting a correct stream ...
An Investigative Testing of Structured and Unstructured Data Formats in Big Data Application Using Apache Spark
Abstract
Apache Spark programming framework provides an effective and vigorous open-source solution for Big data testing and management. It offers a rich set of big data applications through a programming interface. Testing procedures deliver optimum ...

Comments

Information & Contributors

Information

Published In

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

154
Total Citations
View Citations
1,901
Total Downloads

Downloads (Last 12 months)267
Downloads (Last 6 weeks)20

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rigas STzouveli PKollias S(2024)An End-to-End Deep Learning Framework for Fault Detection in Marine MachinerySensors10.3390/s2416531024:16(5310)Online publication date: 16-Aug-2024
https://doi.org/10.3390/s24165310
Saravanan SBalasubramanian U(2024)An Adaptive Scalable Data Pipeline for Multiclass Attack Classification in Large-Scale IoT NetworksBig Data Mining and Analytics10.26599/BDMA.2023.90200277:2(500-511)Online publication date: Jun-2024
https://doi.org/10.26599/BDMA.2023.9020027
Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Jindal ABeedkar KSingh VMohammed JSingla TGupta AChoudhary K(2024)Reactive Dataflow for Inflight Error Handling in ML WorkflowsProceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning10.1145/3650203.3663333(51-61)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3650203.3663333
Santhanam KRaghavan DRahman MVenkatesh TKunjal NThaker PLevis PZaharia M(2024)ALTOProceedings of the 4th Workshop on Machine Learning and Systems10.1145/3642970.3655844(117-125)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3642970.3655844
Saxena SGhatak SKolla RMukherjee DChakraborty TBaeza-Yates RBonchi F(2024)DPHGNN: A Dual Perspective Hypergraph Neural NetworksProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672047(2548-2559)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672047
Henning SVogel ALeichtfried MErtl ORabiser RBalsamo SKnottenbelt WAbad CShang W(2024)ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing FrameworksProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645036(2-13)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645036
Vogel AHenning SPerez-Wohlfeil EErtl ORabiser R(2024)A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing FrameworksProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666040(171-182)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666040
Meldrum MCarbone P(2024)μWheel: Aggregate Management for Streams and QueriesProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666031(54-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666031
Segeljakt KHaridi SCarbone P(2024)AquaLang: A Dataflow Programming LanguageProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666030(42-53)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666030
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems

Experimental study on the performance and resource utilization of data streaming frameworks

An Investigative Testing of Structured and Unstructured Data Formats in Big Data Application Using Apache Spark

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems

Experimental study on the performance and resource utilization of data streaming frameworks

An Investigative Testing of Structured and Unstructured Data Formats in Big Data Application Using Apache Spark

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations