Cost-efficient Workflow as a Service using Containers
Abstract
Workflows are special applications used to solve complex scientific problems. The emerging Workflow as a Service (WaaS) model provides scientists with an effective way of deploying their workflow applications in Cloud environments. The WaaS model can execute multiple workflows in a multi-tenant Cloud environment. Scheduling the tasks of the workflows in the WaaS model has several challenges. The scheduling approach must properly utilize the underlying Cloud resources and satisfy the users’ Quality of Service (QoS) requirements for all the workflows. In this work, we have proposed a heurisine-sensitive workflows in a containerized Cloud environment for the WaaS model. We formulated the problem of minimizing the MIPS (million instructions per second) requirement of tasks while satisfying the deadline of the workflows as a non-linear optimization problem and applied the Lagranges multiplier method to solve it. It allows us to configure/scale the containers’ resources and reduce costs. We also ensure maximum utilization of VM’s resources while allocating containers to VMs. Furthermore, we have proposed an approach to effectively scale containers and VMs to improve the schedulability of the workflows at runtime to deal with the dynamic arrival of the workflows. Extensive experiments and comparisons with other state-of-the-art works show that the proposed approach can significantly improve resource utilization, prevent deadline violation, and reduce the cost of renting Cloud resources for the WaaS model.
References
[1]
Zhao, Y., Li, Y., Raicu, I., Lu, S., Lin, C., Zhang, Y., Tian, W., Xue, R.: A service framework for scientific workflow management in the Cloud. IEEE Trans. Serv. Comput. 8(6), 930–944 (2014)
[2]
Zhao, Y., Li, Y., Raicu, I., Lu, S., Tian, W., Liu, H.: Enabling scalable scientific workflow management in the cloud. Futur. Gener. Comput. Syst. 46, 3–16 (2015)
[3]
Ahmad, Z., Nazir, B., Umer, A.: A fault-tolerant workflow management system with quality-of-service-aware scheduling for scientific workflows in cloud computing. Int. J. Commun. Syst. 34(1), 4649 (2021)
[4]
Hilman, M.H., Rodriguez, M.A., Buyya, R.: Resource-sharing policy in multi-tenant scientific workflow as a service platform (2019). Available from
[5]
Stavrinides, G.L., Karatza, H.D.: An energy-efficient, Qos-aware and cost-effective scheduling approach for real-time workflow applications in cloud computing systems utilizing DVFS and approximate computations. Futur. Gener. Comput. Syst. 96, 216–226 (2019)
[6]
Qin, Y., Wang, H., Yi, S., Li, X., Zhai, L.: An energy-aware scheduling algorithm for budget-constrained scientific workflows based on multi-objective reinforcement learning. J. Supercomput. 76(1), 455–480 (2020)
[7]
Tarafdar, A., Karmakar, K., Khatua, S., Das, R.K.: Energy-efficient scheduling of deadline-sensitive and budget-constrained workflows in the cloud. In: International conference on distributed computing and internet technology. Springer, pp. 65–80 (2021)
[8]
Rodriguez, M.A., Buyya, R.: Scheduling dynamic workloads in multi-tenant scientific workflow as a service platforms. Futur. Gener. Comput. Syst. 79, 739–750 (2018)
[9]
Taibi, D., Lenarduzzi, V., Pahl, C.: Architectural patterns for microservices: a systematic mapping study. In: CLOSER 2018: Proceedings of the 8th international conference on cloud computing and services science. SciTePress, Funchal (2018)
[10]
Wang, J., Korambath, P., Altintas, I., Davis, J., Crawl, D.: Workflow as a service in the cloud: architecture and scheduling algorithms. Procedia Comput. Sci. 29, 546–556 (2014)
[11]
Esteves, S., Veiga, L.: Waas: workflow-as-a-service for the cloud with scheduling of continuous and data-intensive workflows. Comput. J. 59(3), 371–383 (2016)
[12]
Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.-H., Vahi, K.: Characterization of scientific workflows. In: 2008 third workshop on workflows in support of large-scale science. IEEE, pp. 1–10 (2008)
[13]
Rizvi, N., Ramesh, D.: Fair budget constrained workflow scheduling approach for heterogeneous clouds. Clust. Comput. 23(4), 3185–3201 (2020)
[14]
Karmakar, K., Das, R.K., Khatua, S.: Resource scheduling of workflow tasks in cloud environment. In: 2019 IEEE international conference on advanced networks and telecommunications systems (ANTS). IEEE, pp. 1–6 (2019)
[15]
Karmakar, K., Das, R.K., Khatua, S.: Resource scheduling for tasks of a workflow in cloud environment. In: International conference on distributed computing and internet technology. Springer, pp. 214–226 (2020)
[16]
Tarafdar, A., Karmakar, K., Das, R.K., Khatua, S.: Multi-criteria scheduling of scientific workflows in the workflow as a service platform. Comput. Electr. Eng. 105, 108458 (2023)
[17]
Qasha, R., Cala, J., Watson, P.: Dynamic deployment of scientific workflows in the cloud using container virtualization. In: 2016 IEEE international conference on cloud computing technology and science (CloudCom). IEEE, pp. 269–276 (2016)
[18]
Liu, K., Aida, K., Yokoyama, S., Masatani, Y.: Flexible container-based computing platform on cloud for scientific workflows. In: 2016 international conference on cloud computing research and innovations (ICCCRI). IEEE, pp. 56–63 (2016)
[19]
Chen H, Zhu X, Liu G, and Pedrycz W Uncertainty-aware online scheduling for real-time workflows in cloud service environment IEEE Trans. Serv. Comput. 2018 14 4 1167-1178
[20]
B Burkat, K., Pawlik, M., Balis, B., Malawski, M., Vahi, K., Rynge, M., da Silva, R.F., Deelman, E.: Serverless Containers–rising viable approach to Scientific Workflows. In: 17th International Conference on eScience (eScience). IEEE, pp. 40-49 (2021)
[21]
Ranjan, R., Thakur, I.S., Aujla, G.S., Kumar, N., Zomaya, A.Y.: Energy-efficient workflow scheduling using container-based virtualization in software-defined data centers. IEEE Trans. Ind. Inform. 16(12), 7646–7657 (2020)
[22]
Bao, L., Wu, C., Bu, X., Ren, N., Shen, M.: Performance modeling and workflow scheduling of microservice-based applications in clouds. IEEE Trans. Parallel Distrib. Syst. 30(9), 2114–2129 (2019)
[23]
Al-Dhuraibi, Y., Paraiso, F., Djarallah, N., Merle, P.: Autonomic vertical elasticity of docker containers with Elasticdocker. In: 2017 IEEE 10th international conference on cloud computing (CLOUD). IEEE, pp. 472–479 (2017)
[24]
Paraiso, F., Challita, S., Al-Dhuraibi, Y., Merle, P.: Model-driven management of docker containers. In: 2016 IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp. 718–725 (2016)
[25]
Wu, Q., Datla, V.V.: On performance modeling and prediction in support of scientific workflow optimization. In: 2011 IEEE world congress on services. IEEE, pp. 161–168 (2011)
[26]
Sadjadi, S.M., Shimizu, S., Figueroa, J., Rangaswami, R., Delgado, J., Duran, H., Collazo-Mojica, X.J.: A modeling approach for estimating execution time of long-running scientific applications. In: 2008 IEEE international symposium on parallel and distributed processing. IEEE, pp. 1–8 (2008)
[27]
Saeedizade, E., Ashtiani, M.: Ddbws: A dynamic deadline and budget-aware workflow scheduling algorithm in workflow-as-a-service environments. J. Supercomput. 77(12), 14525–14564 (2021)
[28]
Arabnejad, V., Bubendorfer, K., Ng, B.: Dynamic multi-workflow scheduling: a deadline and cost-aware approach for commercial clouds. Futur. Gener. Comput. Syst. 100, 98–108 (2019)
[29]
Silva, R.F., Pottier, L., Coleman, T., Deelman, E., Casanova, H.: Workflowhub: community framework for enabling scientific workflow research and development. In: 2020 IEEE/ACM workflows in support of large-scale science (WORKS). IEEE, pp. 49–56 (2020)
[30]
Piraghaj, S.F., Dastjerdi, A.V., Calheiros, R.N., Buyya, R.: Containercloudsim: an environment for modeling and simulation of containers in cloud data centers. Softw. Pract. Experience 47(4), 505–521 (2017)
[31]
Mao, M., Humphrey, M.: A performance study on the VM startup time in the cloud. In: 2012 IEEE fifth international conference on cloud computing. IEEE, pp. 423–430 (2012)
[32]
Da Silva, R.F., Chen, W., Juve, G., Vahi, K., Deelman, E.: Community resources for enabling research in distributed scientific workflows. In: 2014 IEEE 10th international conference on e-science, vol. 1. IEEE, pp. 177–184 (2014)
[33]
Silva, R.F.d., Chen, W., Juve, G., Vahi, K., Deelman, E.: Community resources for enabling research in distributed scientific workflows. In: Proceedings of the 2014 IEEE 10th international conference on e-science-volume 01, pp. 177–184 (2014)
Recommendations
Multi-criteria scheduling of scientific workflows in the Workflow as a Service platform
AbstractThe increasing use of workflows in various fields of science has led to the development of the Workflow as a Service (WaaS) platform, which provides workflow execution as a service to the users. The WaaS platform must use appropriate resource ...
Graphical abstractDisplay Omitted
Highlights- Multi-criteria scheduling approach for the Workflow as a Service platform.
- Optimal MIPS for tasks decides sharing of virtual machine resources with containers.
- Resource scaling policy to improve the schedulability of the workflows.
Time-Cost Efficient Scheduling Algorithms for Executing Workflow in Infrastructure as a Service Clouds
Cloud Computing enables delivery of IT resources over the Internet and follows the pay-as-you-go billing model. The cloud infrastructures can be used as an appropriate environment for executing of workflow applications. To execute workflow applications ...
Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos
CCGrid '17: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid ComputingWorkflows are a widely used abstraction for describing large scientific applications and running them on distributed systems. However, most workflow systems have been silent on the question of what execution environment each task in the workflow is ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Nature B.V. 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 11 March 2024
Accepted: 12 January 2024
Received: 03 October 2023
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Jan 2025