Self-Regulating Streaming Systems: Challenges and Opportunities

In recent years, stream processing systems have been deployed in almost every organization due to the explosion of large-scale analytics applications. Our discussions with users of these systems within Microsoft and Twitter have revealed that a major challenge with these frameworks is to tune them in order to meet the required performance and also maintain this level of performance over time. In this paper, we present the open problems and challenges in supporting streaming systems that self-regulate. Such systems automatically adjust their configuration to meet service level objectives (SLOs) even in the presence of external load variations or internal faults such as slow hardware. To address some of these challenges, we propose using machine learning techniques such as supervised learning and reinforcement learning which can potentially further improve the application management lifecycle. We believe that exploring machine learning in the context of self-regulating streaming systems is a rich area for future research with can impact the ways streaming applications are managed.