Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines
Authors: 
Zhou Zhang, Pei-Quan Jin, Xi-Ke Xie, Xiao-Liang Wang, Rui-Cheng Liu, Shou-Hong WanAuthors Info & Claims
Zhou Zhang, Pei-Quan Jin, Xi-Ke Xie, Xiao-Liang Wang, Rui-Cheng Liu, Shou-Hong WanAuthors Info & ClaimsPages 116 - 138
Abstract
Most distributed stream processing engines (DSPEs) do not support online task management and cannot adapt to time-varying data flows. Recently, some studies have proposed online task deployment algorithms to solve this problem. However, these approaches do not guarantee the Quality of Service (QoS) when the task deployment changes at runtime, because the task migrations caused by the change of task deployments will impose an exorbitant cost. We study one of the most popular DSPEs, Apache Storm, and find out that when a task needs to be migrated, Storm has to stop the resource (implemented as a process of Worker in Storm) where the task is deployed. This will lead to the stop and restart of all tasks in the resource, resulting in the poor performance of task migrations. Aiming to solve this problem, in this paper, we propose N-Storm (Nonstop Storm), which is a task-resource decoupling DSPE. N-Storm allows tasks allocated to resources to be changed at runtime, which is implemented by a thread-level scheme for task migrations. Particularly, we add a local shared key/value store on each node to make resources aware of the changes in the allocation plan. Thus, each resource can manage its tasks at runtime. Based on N-Storm, we further propose Online Task Deployment (OTD). Differing from traditional task deployment algorithms that deploy all tasks at once without considering the cost of task migrations caused by a task re-deployment, OTD can gradually adjust the current task deployment to an optimized one based on the communication cost and the runtime states of resources. We demonstrate that OTD can adapt to different kinds of applications including computation- and communication-intensive applications. The experimental results on a real DSPE cluster show that N-Storm can avoid the system stop and save up to 87% of the performance degradation time, compared with Apache Storm and other state-of-the-art approaches. In addition, OTD can increase the average CPU usage by 51% for computation-intensive applications and reduce network communication costs by 88% for communication-intensive applications.
References
[1]
Fu X W, Ghaffar T, Davis J C, Lee D. EdgeWise: A better stream processing engine for the edge. In Proc. the 2019 USENIX Annual Technical Conference, Jul. 2019, pp.929–946.
[2]
Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel J M, Kulkarni S, Jackson J, Gade K, Fu M S, Donham J, Bhagat N, Mittal S, Ryaboy D. Storm@Twitter. In Proc. the 2014 ACM SIGMOD International Conference on Management of Data, Jun. 2014, pp.147–156.
[3]
Kulkarni S, Bhagat N, Fu M S, Kedigehalli V, Kellogg C, Mittal S, Patel J M, Ramasamy K, Taneja S. Twitter heron: Stream processing at scale. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 2015, pp.239–250.
[4]
Fu M S, Agrawal A, Floratou A, Graham B, Jorgensen A, Li R H, Lu N, Ramasamy K, Rao S, Wang C. Twitter heron: Towards extensible streaming engines. In Proc. the 33rd IEEE International Conference on Data Engineering, Apr. 2017, pp.1165–1172.
[5]
Zhang Z, Jin P Q, Wang X L, Liu R C, Wan S H. NStorm: Efficient thread-level task migration in Apache Storm. In Proc. the 21st International Conference on High Performance Computing and Communications, the 17th IEEE International Conference on Smart City, the 5th IEEE International Conference on Data Science and Systems, Aug. 2019, pp.1595–1602.
[6]
Xu J L, Chen Z H, Tang J, Su S. T-Storm: Traffic-aware online scheduling in Storm. In Proc. the 34th IEEE International Conference on Distributed Computing Systems, Jun. 30–Jul. 3, 2014, pp.535–544.
[7]
Zhang J, Li C L, Zhu L Y, Liu Y P. The real-time scheduling strategy based on traffic and load balancing in Storm. In Proc. the 18th International Conference on High Performance Computing and Communications, the 14th IEEE International Conference on Smart City, the 2nd IEEE International Conference on Data Science and Systems, Dec. 2016, pp.372–379.
[8]
Peng B Y, Hosseini M, Hong Z H, Farivar R, Campbell R. R-Storm: Resource-aware scheduling in Storm. In Proc. the 16th Annual Middleware Conference, Nov. 2015, pp.149–161.
[9]
Mai L, Zeng K, Potharaju R, Xu L, Suh S, Venkataraman S, Costa P, Kim T, Muthukrishnan S, Kuppa V, Dhulipalla S, Rao S. Chi: A scalable and programmable control plane for distributed stream processing systems. Proceedings of the VLDB Endowment, 2018, 11(10): 1303–1316.
[10]
Nasir M A U, De Francisci Morales G, García-Soriano D, Kourtellis N, Serafini M. The power of both choices: Practical load balancing for distributed stream processing engines. In Proc. the 31st IEEE International Conference on Data Engineering, Apr. 2015, pp.137–148.
[11]
Aniello L, Baldoni R, Querzoni L. Adaptive online scheduling in Storm. In Proc. the 7th ACM International Conference on Distributed Event-Based Systems, Jun. 2013, pp.207–218.
[12]
Cardellini V, Lo Presti F, Nardelli M, and Russo GR Optimal operator deployment and replication for elastic distributed data stream processing Concurrency and Computation: Practice and Experience 2018 30 9 e4334
[13]
Li J, Pu C, Chen Y, Gmach D, Milojicic D. Enabling elastic stream processing in shared clusters. In Proc. the 9th IEEE International Conference on Cloud Computing, Jun. 27–Jul. 2, 2016, pp.108–115.
[14]
Weng Z J, Guo Q, Wang C K, Meng X F, He B S. AdaStorm: Resource efficient Storm with adaptive configuration. In Proc. the 33rd IEEE International Conference on Data Engineering, Apr. 2017, pp.1363–1364.
[15]
Farahabady M R H, Samani H R D, Wang Y D, Zomaya A Y, Tari Z. A QoS-aware controller for Apache Storm. In Proc. the 15th IEEE International Symposium on Network Computing and Applications, Oct. 26–Nov. 2, 2016, pp.334–342.
[16]
Jiang J W, Zhang Z P, Cui B, Tong Y H, Xu N. Stro-MAX: Partitioning-based scheduler for real-time stream processing system. In Proc. the 22nd International Conference on Database Systems for Advanced Applications, Mar. 2017, pp.269–288.
[17]
Nardelli M, Cardellini V, Grassi V, and Lo PF Efficient operator placement for distributed data stream processing applications IEEE Trans. Parallel and Distributed Systems 2019 30 8 1753-1767
[18]
Eskandari L, Mair J, Huang Z Y, Eyers D. I-Scheduler: Iterative scheduling for distributed stream processing systems. Future Generation Computer Systems, 2021, 117: 219–233. .
[19]
Chatzistergiou A, Viglas S D. Fast heuristics for near-optimal task allocation in data stream processing over clusters. In Proc. the 23rd ACM International Conference on Information and Knowledge Management, Nov. 2014, pp.1579–1588.
[20]
Qian W J, Shen Q N, Qin J, Yang D, Yang Y H, Wu Z H. S-Storm: A slot-aware scheduling strategy for even scheduler in Storm. In Proc. the 18th International Conference on High Performance Computing and Communications, the 14th IEEE International Conference on Smart City, the 2nd IEEE International Conference on Data Science and Systems, Dec. 2016, pp.623–630.
[21]
Fang J H, Zhang R, Fu T Z J, Zhang Z J, Zhou A Y, Zhu J H. Parallel stream processing against workload skewness and variance. In Proc. the 26th International Symposium on High-Performance Parallel and Distributed Computing, Jun. 2017, pp.15–26.
[22]
Li CL, Zhang J, and Luo YL Real-time scheduling based on optimized Topology and communication traffic in distributed real-time computation platform of Storm Journal of Network and Computer Applications 2017 87 100-115
[23]
Sun DW, Zhang GY, Yang SL, Zheng WM, Khan SU, and Li KQ Re-Stream: Real-time and energy-efficient resource scheduling in big data stream computing environments Information Sciences 2015 319 92-112
[24]
Sax M J, Castellanos M, Chen Q M, Hsu M. Aeolus: An optimizer for distributed intra-node-parallel streaming systems. In Proc. the 29th IEEE International Conference on Data Engineering, Apr. 2013, pp.1280–1283.
[25]
Fu TZJ, Ding JB, Ma RTB, Winslett M, Yang Y, and Zhang ZJ DRS: Auto-scaling for real-time stream analytics IEEE/ACM Trans. Networking 2017 25 6 3338-3352
[26]
Kahveci B and Gedik B Joker: Elastic stream processing with organic adaptation Journal of Parallel and Distributed Computing 2020 137 205-223
[27]
Floratou A, Agrawal A, Graham B, Rao S, Ramasamy K. Dhalion: Self-regulating stream processing in Heron. Proceedings of the VLDB Endowment, 2017, 10(12): 1825–1836.
[28]
Lombardi F, Aniello L, Bonomi S, and Querzoni L Elastic symbiotic scaling of operators and resources in stream processing systems IEEE Trans. Parallel and Distributed Systems 2018 29 3 572-585
[29]
Kalavri V, Liagouris J, Hoffmann M, Dimitrova D, Forshaw M, Roscoe T. Three steps is all you need: Fast, accurate, automatic scaling decisions for distributed streaming dataflows. In Proc. the 13th USENIX Conference on Operating Systems Design and Implementation, Oct. 2018, pp.783–798.
[30]
Marangozova-Martin V, de Palma N, and El Rheddane A Multi-level elasticity for data stream processing IEEE Trans. Parallel and Distributed Systems 2019 30 10 2326-2337
[31]
Wang CK, Meng XF, Guo Q, Weng ZJ, and Yang C Automating characterization deployment in distributed data stream management systems IEEE Trans. Knowledge and Data Engineering 2017 29 12 2669-2681
[32]
Yang M S, Ma R T B. Smooth task migration in Apache Storm. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 2015, pp.2067–2068.
[33]
Shukla A, Simmhan Y. Toward reliable and rapid elasticity for streaming dataflows on clouds. In Proc. the 38th IEEE International Conference on Distributed Computing Systems, Jul. 2018, pp.1096–1106.
[34]
Fernandez R C, Migliavacca M, Kalyvianaki E, Pietzuch P. Integrating scale out and fault tolerance in stream processing using operator state management. In Proc. the 2013 ACM SIGMOD International Conference on Management of Data, Jun. 2013, pp.725–736.
[35]
Wu Y J, Tan K L. ChronoStream: Elastic stateful stream computation in the cloud. In Proc. the 31st IEEE International Conference on Data Engineering, Apr. 2015, pp.723–734.
[36]
Gedik B, Schneider S, Hirzel M, and Wu KL Elastic scaling for data stream processing IEEE Trans. Parallel and Distributed Systems 2014 25 6 1447-1463
[37]
Noghabi S A, Paramasivam K, Pan Y, Ramesh N, Bringhurst J, Gupta I, Campbell R H. Samza: Stateful scalable stream processing at LinkedIn. Proceedings of the VLDB Endowment, 2017, 10(12): 1634–1645.
[38]
Venkataraman S, Panda A, Ousterhout K, Armbrust M, Ghodsi A, Franklin M J, Recht B, Stoica I. Drizzle: Fast and adaptable stream processing at scale. In Proc. the 26th Symposium on Operating Systems Principles, Oct. 2017, pp.374–389.
[39]
Wang L, Fu T Z J, Ma R T B, Winslett M, Zhang Z J. Elasticutor: Rapid elasticity for realtime stateful stream processing. In Proc. the 2019 International Conference on Management of Data, Jun. 2019, pp.573–588.
[40]
Hoffmann M, Lattuada A, McSherry F. Megaphone: Latency-conscious state migration for distributed streaming dataflows. Proceedings of the VLDB Endowment, 2019, 12(9): 1002–1015.
[41]
Del Monte B, Zeuch S, Rabl T, Markl V. Rhino: Efficient management of very large distributed state for stream processing engines. In Proc. the 2020 ACM SIGMOD International Conference on Management of Data, Jun. 2020, pp.2471–2486.
Recommendations
Smooth Task Migration in Apache Storm
SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of DataTask migration happens when distributed data processing systems scale in real-time. To handle the task migration process more gracefully, we propose three task migration methods: (i) worker level migration, (ii) executor level migration, and (iii) ...
Task allocation for distributed stream processing
IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer CommunicationsThere is a growing demand for live, on-the-fly processing of increasingly large amounts of data. In order to ensure the timely and reliable processing of streaming data, a variety of distributed stream processing architectures and platforms have been ...
A Cluster-Based Approach to Consensus Based Distributed Task Allocation
PDP '14: Proceedings of the 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based ProcessingThis paper presents a novel extension to the Consensus-Based Bundle Algorithm (CBBA), which we have named Cluster-Formed Consensus-Based Bundle Algorithm (CFCBBA). CF-CBBA is designed to reduce the amount of communication required to complete a ...
Comments
Information & Contributors
Information
Published In
© Institute of Computing Technology, Chinese Academy of Sciences 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 30 January 2024
Accepted: 06 January 2022
Received: 30 May 2021
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in