Article

Scheduling concurrent workflows in HPC cloud through exploiting schedule gaps

Authors:

Kuo-Chan Huang,

Po-Jen ShihAuthors Info & Claims

ICA3PP'11: Proceedings of the 11th international conference on Algorithms and architectures for parallel processing - Volume Part I

Pages 282 - 293

Published: 24 October 2011 Publication History

Abstract

Many large-scale scientific applications are usually constructed as workflows due to large amounts of interrelated computation and communication. Workflow scheduling has long been a research topic in parallel and distributed computing. However, most previous research focuses on single workflow scheduling. As cloud computing emerges, users can now have easy access to on-demand high performance computing resources, usually called HPC cloud. Since HPC cloud has to serve many users simultaneously, it is common that many workflows submitted from different users are running concurrently. Therefore, how to schedule concurrent workflows efficiently becomes an important issue in HPC cloud environments. Due to the dependency and communication costs between tasks in a workflow, there usually are gaps formed in the schedule of a workflow. In this paper, we propose a method which exploits such schedule gaps to efficiently schedule concurrent workflows in HPC cloud. The proposed scheduling method was evaluated with a series of simulation experiments and compared to the existing method in the literature. The results indicate that our method can deliver good performance and outperform the existing method significantly in terms of average makespan, up to 18% performance improvement.

References

[1]

Bittencourt, L.F., Madeira, E.R.M.: Towards the Scheduling of Multiple Workflows on Computational Grids. Journal of Grid Computing 8, 419-441 (2009).

[2]

Kwok, Y.K., Ahmad, I.: Static Scheduling Algorithms for Allocating Directed Task Graphs to Multiprocessors. ACM Computing Surveys 31(4), 406-471 (1999).

Digital Library

[3]

Adam, T.L., Chandy, K.M., Dickson, J.R.: A Comparison of List Schedules for Parallel Processing Systems. Communications of the ACM 17(12), 685-690 (1974).

Digital Library

[4]

Bittencourt, L.F., Madeira, E.R.M.: Fulfilling Task Dependence Gaps for Workflow Scheduling on Grids. In: 3rd IEEE International Conference on Signal-Image Technology and Internet Based Systems, pp. 468-475 (2007).

Digital Library

[5]

Stavrinides, G.L., Karatza, H.D.: Scheduling Multiple Task Graphs in Heterogeneous Distributed Real-Time Systems by Exploiting Schedule Holes with Bin Packing Techniques. Simulation Modelling Practice and Theory, vol 19(1), 540-552 (2011).

[6]

Bittencourt, L.F., Sakellariou, R., Madeira, E.R.M.: DAG Scheduling Using a Lookahead Variant of the Heterogeneous Earliest Finish Time Algorithm. In: 18th 'Conference on Parallel, Distributed and Network-based Processing, pp. 27-34 (2010).

Digital Library

[7]

Wieczorek, M., Prodan, R., Hoheisel, A.: Taxonomies of the Multi-Criteria Grid Workflow Scheduling Problem. In: Grid Middleware and Services, pp. 237-264 (2008).

[8]

Rahman, M., Ranjan, R., Buyya, R.: Cooperative and Decentralized Workflow Scheduling in Global Grids. Future Generation Computer Systems 26, 753-768 (2010).

Digital Library

[9]

Ding, F., Zhang, R., Ruan, K., Lin, J., Zhao, Z.: A QoS-based Scheduling Approach for Complex Workflow Applications. In: 5th Annual ChinaGrid Conference, pp. 67-73 (2010).

Digital Library

[10]

Bittencourt, L.F., Madeira, E.R.M.: A Performance-Oriented Adaptive Scheduler for Dependent Tasks on Grids. Concurrency and Computation: Practice and Experience 20, 1029-1049 (2008).

Digital Library

[11]

Gary, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NPCompleteness. W.H. Freeman and Co, New York (1979).

Digital Library

[12]

Ullman, J.D.: NP-Complete Scheduling Problems. Journal of Computer and Systems Sciences 10, 384-393 (1975).

Digital Library

[13]

Zhao, H., Sakellarious, R.: Scheduling Multiple DAGs onto Heterogeneous Systems. In: 15th Heterogeneous Computing Workshop, 14 pp (2006).

Digital Library

[14]

Yu, Z., Shi, W.: A Planner-Guided Scheduling Strategy for Multiple Workflow Applications. In: 37th International Conference on Parallel Processing Workshops, pp. 8- 12 (2008).

Digital Library

[15]

N'takpé, T., Suter, F.: Concurrent Scheduling of Parallel Task Graphs on Multi-Clusters Using Constrained Resource Allocations. In: IEEE International Symposium on Parallel and Distributed Processing, pp. 1-8 (2009).

Digital Library

[16]

Kwok, Y., Ahmad, I.: Dynamic Critical-Path Scheduling: An Effective Technique for Allocation Task Graphs to Multi-processors. IEEE Transactions on Parallel and Distributed Systems 7(5), 506-521 (1996).

Digital Library

[17]

Sih, G.C., Lee, E.A.: A Compile-Time Scheduling Heuristic for Interconnection-Constrained Heterogeneous Processor Architectures. IEEE Transactions on Parallel and Distributed Systems 4(2), 175-186 (1993).

Digital Library

[18]

EI-Rewini, H., Lewis, T.G.: Scheduling Parallel Program Tasks onto Arbitrary Target Machines. Journal of Parallel and Distributed Computing 9, 138-153 (1990).

Digital Library

[19]

Yang, T., Gerasoulis, A.: DSC: Scheduling Parallel Tasks on an Unbounded Number of Processors. IEEE Transactions on Parallel and Distributed Systems 5(9), 951-967 (1994).

Digital Library

[20]

Park, G., Shirazi, B., Marquis, J.: DFRN: A New Approach for Duplication Based Scheduling for Distributed Memory Multi-processor Systems. In: International Conference on Parallel Processing, pp. 157-166 (1997).

Digital Library

[21]

Mandal, A., Kennedy, K., Koelbel, C., Marin, G., Mellor-Crummey, J., Liu, B., Johnsson, L.: Scheduling Strategies for Mapping Application Workflows onto the Grid. In: 14th IEEE Symposium on High Performance Distributed Computing, pp. 125-134 (2005).

Digital Library

[22]

Braun, T.D., Siegel, H.J., Beck, N., Bölöni, L.L., Maheswaran, M., Reuther, A.I., Robertson, J.P., Theys, M.D., Yao, B., Hensgen, D., Freund, R.F.: A Comparison of Eleven Static Heuristics for Mapping a Class of Independent Tasks onto Heterogeneous Distributed Computing Systems. Journal of Parallel and Distributed Computing 61(6), 810-837 (2001).

Digital Library

[23]

Hofmann, P., Woods, D.: Cloud Computing: The Limits of Public Clouds for Business Applications. IEEE Internet Computing, 90-93 (November 2010).

Digital Library

[24]

Wei, Y., Blake, M.B.: Service-Oriented Computing and Cloud Computing: Challenges and Opportunities. IEEE Internet Computing, 72-75 (November 2010).

Digital Library

[25]

Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the Clouds: A Berkeley View of Cloud Computing. Technical report no. UCB/EECS-2009-28, EECS Department, University of California, Berkeley (2009).

[26]

salesforce.com, http://www.salesforce.com

[27]

Gmail, http://gamil.com

[28]

Google App Engine, http://code.google.com/intl/en/appengine

[29]

Microsoft Azure Platform, http://www.microsoft.com/windowsazure/

[30]

Amazon Elastic Compute Cloud, http://aws.amazon.com/ec2/

[31]

Akioka, S., Muraoka, Y.: HPC Benchmarks on Amazon EC2. In: 24th IEEE International Conference on Advanced Information Networking and Applications Workshops, pp. 1029-1034 (2010).

Digital Library

[32]

Kim, H.: el-Khamra, Y., Jha, S., Parashar, M.: An Autonomic Approach to Integrated HPC Grid and Cloud Usage. In: 5th IEEE International Conference on e-Science, pp. 366-373 (2009).

Digital Library

Cited By

Rodriguez MBuyya R(2018)Scheduling dynamic workloads in multi-tenant scientific workflow as a service platformsFuture Generation Computer Systems10.1016/j.future.2017.05.00979:P2(739-750)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.future.2017.05.009
Arabnejad HBarbosa J(2018)Reprint of Multi-QoS constrained and Profit-aware scheduling approach for concurrent workflows on heterogeneous systemsFuture Generation Computer Systems10.1016/j.future.2016.11.01678:P1(402-412)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.future.2016.11.016
Primas BGarraghan PDjemame KShakhlevich NJiang CRana OAntonopoulos N(2016)Resource boxingProceedings of the 9th International Conference on Utility and Cloud Computing10.1145/2996890.2996897(138-147)Online publication date: 6-Dec-2016
https://dl.acm.org/doi/10.1145/2996890.2996897
Show More Cited By

Recommendations

DSM: a dynamic scheduling method for concurrent workflows in cloud environment
Abstract
Cloud computing, emerged as a commercial service model, has been widely concerned in both industry and academia. Massive workflow applications could be performed simultaneously on the cloud platforms, which significantly benefits from the ...
Scheduling Scientific Workflows Elastically for Cloud Computing
CLOUD '11: Proceedings of the 2011 IEEE 4th International Conference on Cloud Computing

Most existing workflow scheduling algorithms only consider a computing environment in which the number of compute resources is bounded. Compute resources in such an environment usually cannot be provisioned or released on demand of the size of a ...
Scheduling Workflows in Multi-cluster Environments
WAINA '13: Proceedings of the 2013 27th International Conference on Advanced Information Networking and Applications Workshops

Scientific applications modeled as workflows can exhibit both task and data parallelism. Scheduling these workflows in a multi-cluster environment is challenging due to the large number of task mapping possibilities. Therefore, several heuristics have ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ICA3PP'11: Proceedings of the 11th international conference on Algorithms and architectures for parallel processing - Volume Part I

October 2011

493 pages

ISBN:9783642246494

Editors:
Yang Xiang
Deakin University, School of Information Technology, Burwood, VIC, Australia
,
Wanlei Zhou
Deakin University, School of Information Technology, Burwood, VIC, Australia
,
Alfredo Cuzzocrea
ICAR-CNR and University of Calabria, Rende (CS), Italy
,
Michael Hobbs
Deakin University, School of Information Technology, Geelong, VIC, Australia

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 24 October 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rodriguez MBuyya R(2018)Scheduling dynamic workloads in multi-tenant scientific workflow as a service platformsFuture Generation Computer Systems10.1016/j.future.2017.05.00979:P2(739-750)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.future.2017.05.009
Arabnejad HBarbosa J(2018)Reprint of Multi-QoS constrained and Profit-aware scheduling approach for concurrent workflows on heterogeneous systemsFuture Generation Computer Systems10.1016/j.future.2016.11.01678:P1(402-412)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.future.2016.11.016
Primas BGarraghan PDjemame KShakhlevich NJiang CRana OAntonopoulos N(2016)Resource boxingProceedings of the 9th International Conference on Utility and Cloud Computing10.1145/2996890.2996897(138-147)Online publication date: 6-Dec-2016
https://dl.acm.org/doi/10.1145/2996890.2996897
Vasile MPop FTutueanu RCristea VKołodziej J(2015)Resource-aware hybrid scheduling algorithm in heterogeneous distributed computingFuture Generation Computer Systems10.1016/j.future.2014.11.01951:C(61-71)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1016/j.future.2014.11.019

View Options

View options

Figures

Tables

Media

View Table of Conten