research-article

Open access

Arc: an IR for batch and stream programming

Authors:

Klas Segeljakt,

Christian Schulte,

Seif HaridiAuthors Info & Claims

DBPL 2019: Proceedings of the 17th ACM SIGPLAN International Symposium on Database Programming Languages

Pages 53 - 58

https://doi.org/10.1145/3315507.3330199

Published: 23 June 2019 Publication History

Abstract

In big data analytics, there is currently a large number of data programming models and their respective frontends such as relational tables, graphs, tensors, and streams. This has lead to a plethora of runtimes that typically focus on the efficient execution of just a single frontend. This fragmentation manifests itself today by highly complex pipelines that bundle multiple runtimes to support the necessary models. Hence, joint optimization and execution of such pipelines across these frontend-bound runtimes is infeasible. We propose Arc as the first unified Intermediate Representation (IR) for data analytics that incorporates stream semantics based on a modern specification of streams, windows and stream aggregation, to combine batch and stream computation models. Arc extends Weld, an IR for batch computation and adds support for partitioned, out-of-order stream and window operators which are the most fundamental building blocks in contemporary data streaming.

References

[1]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, Vol. 16. 265–283.

Digital Library

[2]

Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, et al. 2015. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment 8, 12 (2015), 1792–1803.

Digital Library

[3]

Arvind Arasu, Shivnath Babu, and Jennifer Widom. 2006. The CQL continuous query language: semantic foundations and query execution. VLDBJ (2006).

Digital Library

[4]

Arvind Arasu and Jennifer Widom. 2004. Resource sharing in continuous sliding-window aggregates. In VLDB.

Digital Library

[5]

Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015).

[6]

Paris Carbone, Jonas Traub, Asterios Katsifodimos, Seif Haridi, and Volker Markl. 2016. Cutty: Aggregate Sharing for User-Defined Windows. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM.

Digital Library

[7]

Peter MD Gray, Larry Kerschberg, Peter JH King, and Alexandra Poulovassilis. 2013. The functional approach to data management: modeling, analyzing and integrating heterogeneous data. Springer Science & Business Media.

[8]

Martin Hirzel, Robert Soulé, Scott Schneider, Buğra Gedik, and Robert Grimm. 2014. A catalog of stream processing optimizations. ACM Computing Surveys (CSUR) 46, 4 (2014), 46.

Digital Library

[9]

Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel, Karthik Ramasamy, and Siddarth Taneja. 2015. Twitter Heron: Stream Processing at Scale. In ACM SIGMOD.

Digital Library

[10]

Jin Li, Kristin Tufte, Vladislav Shkapenyuk, Vassilis Papadimos, Theodore Johnson, and David Maier. 2008. Out-of-order processing: a new architecture for high-performance stream systems. Proceedings of the VLDB Endowment 1, 1 (2008), 274–288.

Digital Library

[11]

David Maier, Jin Li, Peter Tucker, Kristin Tufte, and Vassilis Papadimos. 2005. Semantics of data streams and operators. In International Conference on Database Theory. Springer, 37–52.

Digital Library

[12]

Shoumik Palkar, James Thomas, Deepak Narayanan, Pratiksha Thaker, Rahul Palamuttam, Parimajan Negi, Anil Shanbhag, Malte Schwarzkopf, Holger Pirk, Saman Amarasinghe, et al. 2018. Evaluating end-to-end optimization for data analytics applications in weld. Proceedings of the VLDB Endowment 11, 9 (2018), 1002–1015.

Digital Library

[13]

Shoumik Palkar, James J Thomas, Anil Shanbhag, Deepak Narayanan, Holger Pirk, Malte Schwarzkopf, Saman Amarasinghe, Matei Zaharia, and Stanford InfoLab. 2017. Weld: A common runtime for high performance data analytics. In Conference on Innovative Data Systems Research (CIDR).

[14]

Kostas Patroumpas and Timos Sellis. 2006. Window specification over data streams. In Current Trends in Database Technology–EDBT 2006. Springer, 445–464.

Digital Library

[15]

Benjamin C Pierce. 2005. Advanced topics in types and programming languages. MIT press.

Digital Library

[16]

Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. ACM SIGPLAN Notices 48, 6 (2013), 519–530.

Digital Library

[17]

Robert Soulé, Martin Hirzel, Buğra Gedik, and Robert Grimm. 2016. River: an intermediate language for stream processing. Software: Practice and Experience 46, 7 (2016), 891–929.

Digital Library

[18]

Utkarsh Srivastava and Jennifer Widom. 2004. Flexible time management in data stream systems. In Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. ACM, 263–274.

Digital Library

[19]

Kanat Tangwongsan, Martin Hirzel, Scott Schneider, and Kun-Lung Wu. 2015. General incremental sliding-window aggregation. In VLDB.

Digital Library

[20]

Jonas Traub, Philipp Grulich, Alejandro Rodriguez Cuellar, Sebastian Breß, Asterios Katsifodimos, Tilmann Rabl, and Volker Markl. 2019. Efficient Window Aggregation with General Stream Slicing. In EDBT. ACM.

[21]

Pete Tucker, Kristin Tufte, Vassilis Papadimos, and David Maier. 2008. NEXMark–A Benchmark for Queries over Data Streams (DRAFT). Technical Report. Technical report, OGI School of Science & Engineering at OHSU, Septembers.

[22]

Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. HotCloud (2010).

Digital Library

Cited By

Laddad SCheung AHellerstein JMilano M(2025)Flo: A Semantic Foundation for Progressive Stream ProcessingProceedings of the ACM on Programming Languages10.1145/37048459:POPL(241-270)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704845
Cutler JWatson CNkurumeh EHilliard PGoldstein HStanford CPierce B(2024)Stream TypesProceedings of the ACM on Programming Languages10.1145/36564348:PLDI(1412-1436)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656434
Bonifati ATommasini RBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)An Overview of Continuous Querying in (Modern) Data SystemsCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3654679(605-612)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3654679
Show More Cited By

Index Terms

Arc: an IR for batch and stream programming
1. Computing methodologies
  1. Distributed computing methodologies
2. Software and its engineering
  1. Software notations and tools
    1. Compilers
    2. Context specific languages

Recommendations

Dual-Paradigm Stream Processing
ICPP '18: Proceedings of the 47th International Conference on Parallel Processing

Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable ...
Characterization of Big Data Stream Processing Pipeline: A Case Study using Flink and Kafka
BDCAT '17: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

In recent years there has been a surge in applications focusing on streaming data to generate insights in real-time. Both academia, as well as industry, have tried to address this use case by developing a variety of Stream Processing Engines (SPEs) with ...
Real-time analysis of market data leveraging Apache Flink
DEBS '22: Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems

In this paper, we present a solution to the DEBS 2022 Grand Challenge (GC). According to the GC requirements, the proposed software continuously observes notifications about financial instruments being traded, aiming to timely detect breakout patterns. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DBPL 2019: Proceedings of the 17th ACM SIGPLAN International Symposium on Database Programming Languages

June 2019

84 pages

ISBN:9781450367189

DOI:10.1145/3315507

General Chairs:
Alvin Cheung,
Kim Nguyễn

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Stiftelsen för Strategisk Forskning

Conference

PLDI '19

Sponsor:

SIGPLAN

PLDI '19: 40th ACM SIGPLAN Conference on Programming Language Design and Implementation

June 23, 2019

AZ, Phoenix, USA

Acceptance Rates

Overall Acceptance Rate 10 of 15 submissions, 67%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
593
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)17

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Laddad SCheung AHellerstein JMilano M(2025)Flo: A Semantic Foundation for Progressive Stream ProcessingProceedings of the ACM on Programming Languages10.1145/37048459:POPL(241-270)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704845
Cutler JWatson CNkurumeh EHilliard PGoldstein HStanford CPierce B(2024)Stream TypesProceedings of the ACM on Programming Languages10.1145/36564348:PLDI(1412-1436)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656434
Bonifati ATommasini RBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)An Overview of Continuous Querying in (Modern) Data SystemsCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3654679(605-612)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3654679
van Beurden PScholz S(2022)On Generating Out-Of-Core GPU Code for Multi-Dimensional Array OperationsProceedings of the 34th Symposium on Implementation and Application of Functional Languages10.1145/3587216.3587223(1-13)Online publication date: 31-Aug-2022
https://dl.acm.org/doi/10.1145/3587216.3587223
Sharkova DChernokoz ATrofimov ASokolov NGorshkova EKuralenok INovikov B(2022)Adaptive SQL Query Optimization in Distributed Stream Processing: A Preliminary StudySoftware Foundations for Data Interoperability10.1007/978-3-030-93849-9_7(96-109)Online publication date: 19-Jan-2022
https://doi.org/10.1007/978-3-030-93849-9_7
Meldrum MSegeljakt KKroll LCarbone PSchulte CHaridi S(2019)ArconProceedings of Real-Time Business Intelligence and Analytics10.1145/3350489.3350492(1-3)Online publication date: 26-Aug-2019
https://dl.acm.org/doi/10.1145/3350489.3350492

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten