research-article

Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines

Authors:

Bonaventura Del Monte,

Volker MarklAuthors Info & Claims

SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

Pages 2471 - 2486

https://doi.org/10.1145/3318464.3389723

Published: 31 May 2020 Publication History

Abstract

Scale-out stream processing engines (SPEs) are powering large big data applications on high velocity data streams. Industrial setups require SPEs to sustain outages, varying data rates, and low-latency processing. SPEs need to transparently reconfigure stateful queries during runtime. However, state-of-the-art SPEs are not ready yet to handle on-the-fly reconfigurations of queries with terabytes of state due to three problems. These are network overhead for state migration, consistency, and overhead on data processing. In this paper, we propose Rhino, a library for efficient reconfigurations of running queries in the presence of very large distributed state. Rhino provides a handover protocol and a state migration protocol to consistently and efficiently migrate stream processing among servers. Overall, our evaluation shows that Rhino scales with state sizes of up to TBs, reconfigures a running query 15 times faster than the state-of-the-art, and reduces latency by three orders of magnitude upon a reconfiguration.

Supplementary Material

MP4 File (3318464.3389723.mp4)

Presentation Video

Download
152.56 MB

References

[1]

Tyler Akidau, Alex Balikov, Kaya Bekiroug lu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, and Sam Whittle. 2013. MillWheel: fault-tolerant stream processing at internet scale. PVLDB (2013).

[2]

Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, et al. 2015. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. PVLDB (2015).

[3]

Alexander Alexandrov, Rico Bergmann, Stephan Ewen, J. Freytag, Fabian Hueske, Arvid Heise, Odej Kao, Marcus Leich, Ulf Leser, Volker Markl, Felix Naumann, Mathias Peters, Astrid Rheinl"ander, Matthias Sax, Sebastian Schelter, Mareike Höger, Kostas Tzoumas, and Daniel Warneke. 2014. The Stratosphere Platform for Big Data Analytics. The VLDB Journal (2014).

[4]

Peter A. Alsberg and John D. Day. 1976. A Principle for Resilient Sharing of Distributed Resources. In ICSE.

[5]

Paris Carbone, Stephan Ewen, Gyula Fóra, Seif Haridi, Stefan Richter, and Kostas Tzoumas. 2017. State Management in Apache Flink: Consistent Stateful Distributed Stream Processing. PVLDB Endow. (2017).

[6]

Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2013. Integrating Scale out and Fault Tolerance in Stream Processing Using Operator State Management. In ACM SIGMOD.

[7]

Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2014. Making State Explicit for Imperative Big Data Processing. In USENIX ATC.

[8]

Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert Henry, Robert Bradshaw, and Nathan. 2010. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In ACM SIGPLAN.

[9]

Badrish Chandramouli, Guna Prasaad, Donald Kossmann, Justin Levandoski, James Hunter, and Mike Barnett. 2018. FASTER: A Concurrent Key-Value Store with In-Place Updates. SIGMOD.

Digital Library

[10]

K. Mani Chandy and Leslie Lamport. 1985. Distributed Snapshots: Determining Global States of Distributed Systems. ACM TOCS (1985).

Digital Library

[11]

Confluent. 2017. Running Kafka in Production. https://docs.confluent.io/current/kafka/deployment.html

[12]

Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon's Highly Available Key-value Store. ACM SIGOPS (2007).

Digital Library

[13]

Facebook. 2012. RocksDB.org. Facebook Open Source. https://rocksdb.org/

[14]

Facebook. 2017. RocksDB Tuning Guide. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

[15]

Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2014. Making state explicit for imperative big data processing. In 2014 USENIX ATC).

[16]

Apache Flink. 2015. Apache Flink Configuration. https://ci.apache.org/projects/flink/flink-docs-master/

[17]

Avrilia Floratou, Ashvin Agrawal, Bill Graham, Sriram Rao, and Karthik Ramasamy. 2017. Dhalion: Self-Regulating Stream Processing in Heron. PVLDB (2017).

Digital Library

[18]

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The Google File System. In ACM SOSP.

[19]

Thomas Heinze, Yuanzhen Ji, Lars Roediger, Valerio Pappalardo, Andreas Meister, Zbigniew Jerzak, and Christof Fetzer. 2015. FUGU: Elastic Data Stream Processing with Latency Constraints. IEEE Data Eng. Bull. (2015).

[20]

Moritz Hoffmann, Andrea Lattuada, Frank McSherry, Vasiliki Kalavri, and Timothy Roscoe. 2019 a. Megaphone: Latency-conscious state migration. https://github.com/strymon-system/megaphone

[21]

Moritz Hoffmann, Andrea Lattuada, Frank McSherry, Vasiliki Kalavri, and Timothy Roscoe. 2019 b. Megaphone: Latency-conscious State Migration for Distributed Streaming Dataflows. VLDB (2019).

[22]

Jeong-Hyon Hwang, Magdalena Balazinska, Alexander Rasin, Ugur Cetintemel, Michael Stonebraker, and Stan Zdonik. 2005. High-Availability Algorithms for Distributed Stream Processing. In IEEE ICDE.

[23]

Gabriela Jacques-Silva, Ran Lei, Luwei Cheng, Guoqiang Jerry Chen, Kuen Ching, Tanji Hu, Yuan Mei, Kevin Wilfong, Rithin Shetty, Serhat Yilmaz, Anirban Banerjee, Benjamin Heintz, Shridar Iyer, and Anshul Jaiswal. 2018. Providing Streaming Joins As a Service at Facebook. PVLDB (2018).

[24]

Vasiliki Kalavri, John Liagouris, Moritz Hoffmann, Desislava Dimitrova, Matthew Forshaw, and Timothy Roscoe. 2018. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In USENIX OSDI.

[25]

David Karger, Eric Lehman, Tom Leighton, Rina Panigrahy, Matthew Levine, and Daniel Lewin. 1997. Consistent Hashing and Random Trees. In ACM STOC.

[26]

Jeyhun Karimov, Tilmann Rabl, Asterios Katsifodimos, Roman Samarev, Henri Heiskanen, and Volker Markl. 2018. Benchmarking Distributed Stream Data Processing Systems. In IEEE ICDE 2018.

[27]

Klaviyo. 2019. Apache Flink Performance Optimization. https://klaviyo.tech/flinkperf-c7bd28acc67

[28]

H. T. Kung, Trevor Blackwell, and Alan Chapman. 1994. Credit-Based Flow Control for ATM Networks: Credit Update Protocol, Adaptive Credit Allocation and Statistical Multiplexing. In ACM SIGCOMM.

[29]

Avinash Lakshman and Prashant Malik. 2010. Cassandra: A Decentralized Structured Storage System. ACM SIGOPS (2010).

[30]

Luo Mai, Kai Zeng, Rahul Potharaju, Le Xu, Shivaram Venkataraman, Paolo Costa, Terry Kim, Saravanan Muthukrishnan, Vamsi Kuppa, Sudheer Dhulipalla, and Sriram Rao. 2018. Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems. VLDB (2018).

Digital Library

[31]

Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Mart'in Abadi. 2013. Naiad: A Timely Dataflow System. In ACM SOSP.

Digital Library

[32]

Raghunath Nambiar, Meikel Poess, Andrew Masland, H. Reza Taheri, Andrew Bond, Forrest Carman, and Michael Majdalany. 2013. TPC State of the Council 2013. In 5th TPC Technology Conference on Performance Characterization and Benchmarking - Volume 8391.

[33]

Muhammad Anis Uddin Nasir, Gianmarco De Francisci Morales, David Garc'i a-Soriano, Nicolas Kourtellis, and Marco Serafini. 2015. The power of both choices: Practical load balancing for distributed stream processing engines. In IEEE ICDE.

[34]

Netflix. 2017. Keystone Real-time Stream Processing Platform. https://medium.com/netflix-techblog/keystone-real-time-stream-processing-platform-a3ee651812a

[35]

Shadi A. Noghabi, Kartik Paramasivam, Yi Pan, Navina Ramesh, Jon Bringhurst, Indranil Gupta, and Roy H. Campbell. 2017. Samza: Stateful Scalable Stream Processing at LinkedIn. PVLDB (2017).

[36]

Brian M. Oki and Barbara H. Liskov. 1988. Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems. In ACM PODC.

[37]

Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. 2010. The hadoop distributed file system. In IEEE MSST.

[38]

Tencent. 2018. Oceanus: A one-stop platform for real time stream processing. https://www.ververica.com/blog/oceanus-platform-powered-by-apache-flink

[39]

Quoc-Cuong To, Juan Soto, and Volker Markl. 2018. A Survey of State Management in Big Data Processing Systems. The VLDB Journal (2018).

[40]

Pete Tucker, Kristin Tufte, Vassilis Papadimos, and David Maier. 2004. NEXMark - A Benchmark for Queries over Data Streams DRAFT. (2004).

[41]

Uber. 2018. Introducing AthenaX, Uber Engineering's Open Source Streaming Analytics Platform. https://eng.uber.com/athenax/

[42]

Robbert van Renesse and Fred B. Schneider. 2004. Chain Replication for Supporting High Throughput and Availability. In USENIX OSDI.

Digital Library

[43]

Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. 2006. Ceph: A Scalable, High-performance Distributed File System. In USENIX OSDI.

[44]

Chenggang Wu, Jose Faleiro, Yihan Lin, and Joseph Hellerstein. 2019. Anna: A kvs for any scale. IEEE TKDE (2019).

[45]

Yingjun Wu and Kian-Lee Tan. 2015. ChronoStream: Elastic stateful stream computation in the cloud. In IEEE ICDE.

[46]

Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. 2013. Discretized Streams: Fault-tolerant Streaming Computation at Scale. In ACM SOSP.

Digital Library

[47]

Matei Zaharia, Reynold S Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J Franklin, et al. 2016. Apache Spark: A unified engine for big data processing. CACM (2016).

Digital Library

[48]

Steffen Zeuch, Ankit Chaudhary, Bonaventura Del Monte, Haralampos Gavriilidis, Dimitrios Giouroukis, Philipp M Grulich, Sebastian Breß, Jonas Traub, and Volker Markl. 2020. The NebulaStream Platform: Data and Application Management for the Internet of Things. In CIDR.

[49]

Yali Zhu, Elke Rundensteiner, and George Heineman. 2004. Dynamic Plan Migration for Continuous Queries over Data Streams. In ACM SIGMOD.

Cited By

Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Zhang ZJin PXie XWang XLiu RWan S(2024)Online Nonstop Task Management for Storm-Based Distributed Stream Processing EnginesJournal of Computer Science and Technology10.1007/s11390-021-1629-939:1(116-138)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s11390-021-1629-9
Show More Cited By

Index Terms

Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs
        MapReduce-based systems
      2. Stream management

Recommendations

Rethinking Stateful Stream Processing with RDMA
SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

Remote Direct Memory Access (RDMA) hardware has bridged the gap between network and main memory speed and thus invalidated the common assumption that network is often the bottleneck in distributed data processing systems. However, high-speed networks do ...
Integrating scale out and fault tolerance in stream processing using operator state management
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

As users of "big data" applications expect fresh results, we witness a new breed of stream processing systems (SPS) that are designed to scale to large numbers of cloud-hosted machines. Such systems face new challenges: (i) to benefit from the "pay-as-...
A new operator for efficient stream-relation join processing in data streaming engines
CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge Management

In the last decade, Stream Processing Engines (SPEs) have emerged as a new processing paradigm that can process huge amounts of data while retaining low latency and high-throughputs. Yet, it is often necessary to join streaming data with traditional ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

June 2020

2925 pages

ISBN:9781450367356

DOI:10.1145/3318464

General Chairs:
David Maier
Portland State University, USA
,
Rachel Pottinger
University of British Columbia, Canada
,
Program Chairs:
AnHai Doan
University of Wisconsin, USA
,
Wang-Chiew Tan
Megagon Labs, USA
,
Publications Chairs:
Abdussalam Alawini
University of Illinois at Urbana-Champaign, USA
,
Hung Q. Ngo
RelationalAI, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

German Ministry for Education and Research
German Federal Ministry for Economic Affairs and Energy

Conference

SIGMOD/PODS '20

Sponsor:

SIGMOD

SIGMOD/PODS '20: International Conference on Management of Data

June 14 - 19, 2020

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,254
Total Downloads

Downloads (Last 12 months)159
Downloads (Last 6 weeks)13

Reflects downloads up to

Other Metrics

View Author Metrics

Citations

Cited By

Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Zhang ZJin PXie XWang XLiu RWan S(2024)Online Nonstop Task Management for Storm-Based Distributed Stream Processing EnginesJournal of Computer Science and Technology10.1007/s11390-021-1629-939:1(116-138)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s11390-021-1629-9
Rank JHerget JHein AKrcmar H(2023)Evaluating Task-Level CPU Efficiency for Distributed Stream Processing SystemsBig Data and Cognitive Computing10.3390/bdcc70100497:1(49)Online publication date: 10-Mar-2023
https://doi.org/10.3390/bdcc7010049
Mao YChen ZZhang YWang MFang YZhang GShi RMa R(2023)StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154316:12(3501-3514)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611543
Esteves JCosta RZhou YAlmeida AKemme BRiviere ESchiavoni VPasin M(2023)An exploratory analysis of methods for real-time data deduplication in streaming processesProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596898(91-102)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596898
Jamieson SForshaw MKemme BRiviere ESchiavoni VPasin M(2023)On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting MethodsProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596886(68-79)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596886
She ZMao YXiang HWang XMa R(2023)StreamSwitch: Fulfilling Latency Service-Layer Agreement for Stateful StreamingIEEE INFOCOM 2023 - IEEE Conference on Computer Communications10.1109/INFOCOM53939.2023.10228903(1-10)Online publication date: 17-May-2023
https://doi.org/10.1109/INFOCOM53939.2023.10228903
Karimov JJacobsen H(2023)SASPAR: Shared Adaptive Stream Partitioning2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00076(922-935)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00076
Zhang YZhang FLi HZhang SDu X(2023)CompressStreamDB: Fine-Grained Adaptive Stream Processing without Decompression2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00038(408-422)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00038
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents