Component Profiling and Prediction Models for QoS-Aware Self-Adapting DSMS Framework
Pages 88 - 94
Abstract
Quality of Service (QoS) has been identified as an important attribute of system performance of Data Stream Management Systems (DSMS). A DSMS should have the ability to allocate physical computing resources between different submitted queries and fulfil QoS specifications in a fair and square manner. System scheduling strategies need to be adjusted dynamically to utilise available physical resources to guarantee the end-to-end quality of service levels. In this paper, we present a proactive method that utilises a multi-level component profiling approach to build prediction models that anticipate several QoS violations and performance degradations. The models are constructed using several incremental machine learning algorithms that are enhanced with ensemble learning and abnormal detection techniques. The approach performs accurate predictions in near real-time with accuracy up to 85% and with abnormal detection techniques, the accuracy reaches 100%. This is a major component within a proposed QoS-Aware Self-Adapting Data Stream Management Framework.
References
[1]
[1]Noghabi, S.A., Paramasivam, K., Pan, Y., Ramesh, N., Bringhurst, J., Gupta, I. and Campbell, R.H., 2017. Samza stateful scalable stream processing at LinkedIn. Proceedings of the VLDB Endowment, pp.1634-1645.
[2]
[2]Gorawski, M., Gorawska, A. and Pasterak, K., 2014. A survey of data stream processing tools. In Information sciences and systems 2014 (pp. 295-303). Springer, Cham.
[3]
[3]Liu, X., Iftikhar, N. and Xie, X., 2014, July. Survey of real-time processing systems for big data. In Proceedings of the 18th International Database Engineering & Applications Symposium (pp. 356-361).
[4]
[4]Terry, D., Goldberg, D., Nichols, D. and Oki, B., 1992. Continuous queries over append-only databases. Acm. Sigmod Record, 21(2), pp.321-330.
[5]
[5]Pratama, M., Ashfahani, A., Ong, Y.S., Ramasamy, S. and Lughofer, E., 2018. Autonomous deep learning: Incremental learning of denoising autoencoder for evolving data streams arXiv preprint arXiv:1809.09081.
[6]
[6]Akidau, T., 2015. The world beyond batch: Streaming 101. A High-Level Tour of Modern Data-Processing Concepts. Blog entry.
[7]
[7] Isah, H., Abughofa, T., Mahfuz, S., Ajerla, D., Zulkernine, F. and Khan, S., 2019. A survey of distributed data stream processing frameworks. IEEE Access, 7, pp.154300-154316.
[8]
[8]Wang, C., Meng, X., Guo, Q., Weng, Z. and Yang, C., 2017. Automating characterization deployment in distributed data stream management systems. IEEE Transactions on Knowledge and Data Engineering, 29(12), pp.2669-2681.
[9]
[9]Iqbal, M.H. and Soomro, T.R., 2015. Big data analysis: Apache storm perspective. International journal of computer trends and technology, 19(1), pp.9-14.
[10]
[10]Chauhan, J., Chowdhury, S.A. and Makaroff, D., 2012, November. Performance evaluation of Yahoo! S4: A first look. In 2012 Seventh International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (pp. 58-65). IEEE.
[11]
[11]Noghabi, S.A., Paramasivam, K., Pan, Y., Ramesh, N., Bringhurst, J., Gupta, I. and Campbell, R.H., 2017. Samza: stateful scalable stream processing at LinkedIn. Proceedings of the VLDB Endowment, 10(12), pp.1634-1645.
[12]
[12]Wang, C., Meng, X., Guo, Q., Weng, Z. and Yang, C., 2017. Automating characterization deployment in distributed data stream management systems. IEEE Transactions on Knowledge and Data Engineering, 29(12), pp.2669-2681.
[13]
[13]Holmes, G., Donkin, A. and Witten, I.H., 1994, November. Weka: A machine learning workbench. In Proceedings of ANZIIS’94-Australian New Zealnd Intelligent Information Systems Conference (pp. 357-361). IEEE.
[14]
[14] Gama, J., Medas, P. and Rodrigues, P., 2005, March. Learning decision trees from dynamic data streams. In Proceedings of the 2005 ACM Symposium on Applied computing (pp. 573-577).
[15]
[15] Bifet, A. and Gavalda, R., 2009, August. Adaptive learning from evolving data streams. In International Symposium on Intelligent Data Analysis (pp. 249-260). Springer, Berlin, Heidelberg.
[16]
[16] Kabakchieva, D., 2012. Student performance prediction by using data mining classification algorithms. International journal of computer science and management research, 1(4), pp.686-690.
[17]
[17] Li, Z., Zhou, F., Chen, F. and Li, H., 2017. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835.
[18]
[18] Krawczyk, B., Minku, L.L., Gama, J., Stefanowski, J. and Wo?niak, M., 2017. Ensemble learning for data stream analysis: A survey. Information Fusion, 37, pp.132-156.
[19]
[19] Khoshkbarforoushha, A., Ranjan, R., Gaire, R., Abbasnejad, E., Wang, L. and Zomaya, A.Y., 2016. Distribution based workload modelling of continuous queries in clouds. IEEE Transactions on Emerging Topics in Computing, 5(1), pp.120-133.
[20]
[20]Wei, Y., Prasad, V., Son, S.H. and Stankovic, J.A., 2006, December. Prediction-based QoS management for real-time data streams. In 2006 27th IEEE International Real-Time Systems Symposium (RTSS’06) (pp. 344-358). IEEE.
[21]
[21]Jiang, Q. and Chakravarthy, S., 2004, July. Scheduling strategies for processing continuous queries over streams. In British National Conference on Databases (pp. 16-30). Springer, Berlin, Heidelberg.
[22]
[22]Gupta, R., Gupta, H. and Mohania, M., 2012, December. Cloud computing and big data analytics: what is new from databases perspective?. In International Conference on Big Data Analytics (pp. 42-61). Springer, Berlin, Heidelberg.
[23]
[23] Baru, C. and Rabl, T., 2016. Application-level benchmarking of big data systems. In Big Data Analytics (pp. 189-199). Springer, New Delhi.
[24]
[24]Nikravesh, A.Y., Ajila, S.A. and Lung, C.H., 2015, May. Towards an autonomic auto-scaling prediction system for cloud resource provisioning. In 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (pp. 35-45). IEEE.
[25]
[25] Yagnik, T., Chen F., and Kasraian, L. 2021, 18 April. QoS-Aware Self-Adapting Resource Utilisation Framework For Distributed Stream Management Systems. In ALLDATA 2021, The Seventh International Conference on Big Data, Small Data, Linked Data and Open Data, Porto, Portugal. IARIA, 2021 (pp. 1-9)
Index Terms
- Component Profiling and Prediction Models for QoS-Aware Self-Adapting DSMS Framework
Index terms have been assigned to the content through auto-classification.
Recommendations
Framework for parameter mapping to provide end-to-end QoS guarantees in IntServ/DiffServ architectures
With the numerous emerging real-time and multimedia applications, there has been much interest for developing mechanisms which enable real-time services over the Internet. These applications have QoS requirements. From the end-user's point of view, QoS ...
QoS-aware bandwidth provisioning for IP network links
Current bandwidth provisioning procedures for IP network links are mostly based on simple rules of thumb, using coarse traffic measurements made on a time scale of e.g., 5 or 15min. A crucial question, however, is whether such coarse measurements give ...
A framework for providing differentiated QoS guarantees in IP-based network
The paper describes the traffic handling mechanisms implemented in the AQUILA pilot QoS IP network [AQUILA Project Consortium (2001)] [11]. The AQUILA architecture enhances the DiffServ concept [A Conceptual Model for DiffServ Routers (2000), An ...
Comments
Information & Contributors
Information
Published In
August 2021
122 pages
ISBN:9781450390408
DOI:10.1145/3481646
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 November 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICCBDC 2021
ICCBDC 2021: 2021 5th International Conference on Cloud and Big Data Computing
August 13 - 15, 2021
Liverpool, United Kingdom
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 47Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format