article

Free access

Effective distributed scheduling of parallel workloads

Authors:

Andrea C. Dusseau,

Remzi H. Arpaci,

David E. CullerAuthors Info & Claims

ACM SIGMETRICS Performance Evaluation Review, Volume 24, Issue 1

Pages 25 - 36

https://doi.org/10.1145/233008.233020

Published: 15 May 1996 Publication History

Abstract

We present a distributed algorithm for time-sharing parallel workloads that is competitive with coscheduling. Implicit scheduling allows each local scheduler in the system to make independent decisions that dynamically coordinate the scheduling of cooperating processes across processors. Of particular importance is the blocking algorithm which decides the action of a process waiting for a communication or synchronization event to complete. Through simulation of bulk-synchronous parallel applications, we find that a simple two-phase fixed-spin blocking algorithm performs well; a two-phase adaptive algorithm that gathers run-time data on barrier wait-times performs slightly better. Our results hold for a range of machine parameters and parallel program characteristics. These findings are in direct contrast to the literature that states explicit coscheduling is necessary for fine-grained programs. We show that the choice of the local scheduler is crucial, with a priority-based scheduler performing two to three times better than a round-robin scheduler. Overall, we find that the performance of implicit scheduling is near that of coscheduling (+/- 35%), without the requirement of explicit, global coordination.

References

[1]

R. H. Arpaci, D. E. Culler, A. Krishnamurthy, S. Steinberg, and K. Yelick. Empirical Evaluation of the CRAY-T3D: A Compiler Perspective. In Proceedsngs of the 2~.2nd Annual Internatsonal Symposzum on Computer Archztecture, June 1995.

Digital Library

[2]

R. H. Arpaci, A. C. Dusseau, A. M. Vahdat, L. T. Liu, T. E. Anderson, and D. A. Patterson. The Interaction of Parallel and Sequential Workloads on a Network of Workstations. In Proceedzngs of A CM SIGMETRICS'95/PERFORMANCE'95 Joznt International Conference on Measurement and Modetzng of Computer Systems, pages 267-278, May 1995.

Digital Library

[3]

K. Arvind. Probabalistic Clock 21ynchronization in Distributed Systems. IEEE Transactzons on Parallel and Distributed Systems, 5(5):474-87, May 1994.

Digital Library

[4]

M. J. Atallah, C. L. Black, D. C. Marinescu, H. J. Siegel, and T. L. Casavant. Models and Algorithms for Co-scheduling Compute-Intensive Tasks on a Network of Workstations Journal of Parallel and Distrbuted Computing, 16-319-327, 1992.

[5]

R. Chandra, S. Devine, B. Verghese, A. Gupta, and M Rosenblum. Scheduling and Page Migration for Multiprocessor Compute Servers. In Internatzonal Conference on Archztectural Support for Programming Languages and Operating Systems, Oct. 1994

Digital Library

[6]

J. Chapin, M. Rosenblum, S. Devine, T. Lahiri, D Teodoslu, and A Gupta. Hive: Fault Containment for Shared-Memory Multiprocessors. In Proceedings of the Fzfteenth ACM Symposzum on Operating Systems Pmnc,ples, pages 12-25, Dec. 1995

Digital Library

[7]

S-H. Chlang, R K. Mansharamani, and M. K. Vernon. Use of Application Characteristics and Limited Preemption for Run-To- Completion Parallel Processor Scheduhng Policies. In Proceed-,ngs of the 1994 ACM SIGMETRICS Conference on Measurement and Modehng of Computer Systems, pages 34-44, Feb. 1994.

Digital Library

[8]

M. Crovella, P. Das, C. Dubnicki, T. LeBlanc, and E. Markatos. Multiprogramming on Multiprocessors. In Proceed,ngs of the Third IEEE Symposium on Parallel and D~stmbuted Processzng, pages 590-597, Dec. 1991.

[9]

D. Culler, L. T. Lm, R. Martin, and C. Yoshikawa LogP Performance Assessment of Fast Network Interfaces. IEEE Mzcro, Feb. 1996.

Digital Library

[10]

R. Cypher, A Ho, S. Konstantinidou, and P. Messina. Architectural Requirements of Parallel Scientific Applications with Explicit Communication. In 20th Annual Internatzonal Symposium on Computer Architecture, May 1993

Digital Library

[11]

J. J. Dongarra, J. R. Bunch, C. B. Moler, and G W. Stewart LINPACK User's Guide. SIAM, Philadelphia, Penn, 1979.

[12]

A. Dusseau, D. Culler, K. Schauser, and R. Martin Fast Parallel Sorting under LogP: Experience with the CM-5 IEEE Transactions on Parallel and D~stmbuted Systems, 1996. To Appear.

Digital Library

[13]

S. Evans, K. Clarke, D. Singleton, and B. Smaalders. Optimizing Unix Resource Scheduling for User Interaction. In 1993 Summer Usen,x, pages 205-218. USENIX, June 1993.

[14]

D. G. Feitelson and L. Rudolph. Distributed Hierarchical Control for Parallel Processing. IEEE Computer, 23(5):65-77, May 1990.

Digital Library

[15]

D. G. Feltelson and L Rudolph. Gang Scheduling Performance Benefits for Fine-Grained Synchronization Journal of Parallel and D,stmbuted Computing, 16(4):306-318, Dec. 1992

[16]

D. G. Feitelson and L Rudolph. Coscheduhng Based on Run- Time Identificatmn of Actlwty Working Sets International Journal of Parallel Programming, 23(2):136-160, Apr. 1995.

Digital Library

[17]

D. Ghosal, G Serazzi, and S. K. Trlpathl The Processor Working Set and Its Use in Scheduling Mult~processor Systems. IEEE Transactions on Software Eng~neemng, 17(5):443-453, May t991.

Digital Library

[18]

B Goodheart and J Cox The Magic Garden Explained: The Internals of UNIX System V Release 4. Prentice Hall, 1994.

Digital Library

[19]

A. Gupta, A. Tucker, and S. Urushibaru. The Impact of Operating System Scheduling Policies and Synchronization Methods on the Performance of Parallel Applications. In Proceed,ngs of the 199i A CM SIGMETRICS Conference on Measurement and Modehng of Computer Systems, pages 120-131, May 1991.

Digital Library

[20]

High Performance Fortran Forum. High Performance Fortran language specification version 1 0 Draft, Jan. 1993

[21]

M, D. Hill, J. R. L~ru~, S. K. l~~inh~rdt, ~nd D. A. Wood. Cooperative Shared Memory: Software and Hardware for Scalable Multiprocessors ACM Transactions on Computer Systems, 11(4).300-18, Nov. 1993.

Digital Library

[22]

A. Karlin, K. Li, M. Manasse, and S Owicki Empirical studies of competitive spinning for a shared-memory multiprocessor. In Thirteenth A CM Symposzum on Operating Systems Pmnc,ples, Oct. 1991.

Digital Library

[23]

K. Keeton, T. Anderson, and D. Patterson. LogP Quantified: The Case for Low-Overhead Local Area Networks. In Proceedings of Hot Interconnects II{, Aug. 1995

[24]

L. I. Kontothanassm and R. W. Wismewski. Using Scheduler Information to Achmve Optimal Barrier Synchronization Performance. In Proceedzngs of the ACM SIGPLAN Symposium on Pmnczples and Practice of Parallel Programming, May 1993

Digital Library

[25]

S. J. Leffler, M. K McKusick, M. J Karels, and J. S. Quarterman. The Design and lmptementatzon of the ~4.3BSD UNIX Operating System Addison-Wesley, 1990.

Digital Library

[26]

S. T. Leutenegger and X.-H. Sun. Distributed Computing Feasibility in a Non-Dedicated Homogenous Distributed System. In Proceedzngs of Supercomput~ng 93, pages 143-152, 1993.

Digital Library

[27]

S. T. Leutenegger and M. K. Vernon. The Performance of Multiprogrammed Multlprocessor Scheduhng Policies In Proceedings of the A CM SIGMETRICS Conference, pages 226-236, May 1990.

Digital Library

[28]

R. P. Martin HPAM An Active Message Layer for a Network of Workstations. In Proceedzngs of Hot Interconnects {I, July 1994.

[29]

C. McCann, R. Vaswam, and J. Zahorjan. A Dynamic Processor Allocation Policy for Multi-programmed Shared-Memory Multiprocessors. In A CM Transactions on Computer Systems, volume 11, pages 146-178, May 1993.

Digital Library

[30]

C. McCann and J. Zahorjan Processor Allocation Policies for Message-Passing Parallel Computers. In Proceedings of the 1994 A CM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 19-32, Feb. 1994.

Digital Library

[31]

C. McCann and J. Zahorjan Scheduling Memory Constrained Jobs on Distributed Memory Parallel Computers. In Proceedings of ACM SIGMETRICS'95/PERFORMANCE'95 Joint International Conference on Measurement and Modehng of Computer Systems, pages 208-219, May 1995.

Digital Library

[32]

L McVoy and C. Staelin. lmbench: Portable Tools for Performance Analysis. In Proceedings of the i996 W, nter USENIX, Jan. 1996.

Digital Library

[33]

V. K. Naik, S. K Setm, and M S Squillante. Performance Analysis of Job Scheduling Policies m Parallel Supercomputing Enwronments. In Proceedings of Supercomputmg '93, pages 824-833, Nov. 1993.

Digital Library

[34]

A. Olson and K. G. Shin. Fault-tolerant Clock Synchronization in Large Multlcomputer Systems. IEEE Transactions on Parallel and D~stmbuted Systems, 5(9) 912-923, Sept. 1994.

Digital Library

[35]

J. K. Ousterhout. Scheduling Techniques for Concurrent Systems. In Th,rd International Conference on D~stmbuted Comput,ng Systems, pages 22-30, May 1982

[36]

V G Peris, M. S. Squillante, and V. K. Naik. Analysis of the Impact of Memory in Dmtributed Parallel Processing Sytems In 1994 A CM SIGMETRICS Conference on Measurement and Modehng of Computer Systems, pages 5-18, Feb 1994.

Digital Library

[37]

K. C Sevclk Characterizations of Parailelmm m Applications and their Use in Scheduhng. in Proceedings of the 1989 A CM SIGMETRICS Conference on Measurement and Modehng of Computer Systems, pages 171-180, May 1989.

Digital Library

[38]

J P Singh, W.-D Weber, and A. Gupta. SPLASH: Stanford Parallel Applications for Shared Memory. Computer Architecture News, 20(1):5-44, Mar. 1992.

Digital Library

[39]

P G Sobalvarro and W. E. Weihl Demand-based Coscheduling of Parallel Jobs on Multiprogrammed Multlprocessors. In Proceedzngs of the IPPS '95 Workshop on Job Scheduhng Strategzes for Parallel Processing, pages 63-75, Apr 1995

Digital Library

[40]

T yon Elcken, A Basu, V. Buch, and W. Vogels. U-Net' A User- Level Network Interface for Parallel and Distributed Computing. In Proceedings of the F,fteenth A CM Symposium on Operating System Pmnc~ples, pages 40-51, Dec 1995

Digital Library

[41]

T von E~cken, D. E. Culler, S C. Goldstem, and K E Schauser. Active Messages' a Mechamsm for Integrated Communication ~nd Comlou~ion. In Pr~vwd,r~g~ ef th~ 19th Ir,~vrnatlenal Symposium on Computer ArcMtecture, May 1992

Digital Library

[42]

R W. Wisniewski, L I. Kontothanassis, and M L Scott. High Performance Synchromzat~on Algomthms for Multiprogrammed Multiprocessors In Proceed,rigs of the Fifth A CM SIGPLAN Sympos,um on Principles and Practice of Parallel Programm~ng, pages 199-206, July 1995.

Digital Library

[43]

J. Zahor9an and E. D. Lazowska. Spinning Versus Blocking m Parallel ~y~tems with Uncertain~y. In Proceedings of the IFIP International Seminar on Performance of D,stmbuted and Parallel Systems, pages 455-472, Dec. 1988.

Cited By

Grosu GNistor SVasile MCiobanu RDobre CPop F(2023)Local and Global Scheduling in Mobile Drop Computing2023 22nd International Symposium on Parallel and Distributed Computing (ISPDC)10.1109/ISPDC59212.2023.00026(85-92)Online publication date: Jul-2023
https://doi.org/10.1109/ISPDC59212.2023.00026
Jia WZhang JShan JLi JDing XGavrilovska AAltınbüken DBinnig C(2022)Achieving low latency in public edges by hiding workloads mutual interferenceProceedings of the 13th Symposium on Cloud Computing10.1145/3542929.3563459(477-492)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3542929.3563459
Tian BHuang JMozafari BSchoenebeck G(2018)Contention-aware lock scheduling for transactional databasesProceedings of the VLDB Endowment10.1145/3187009.317774011:5(648-662)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1145/3187009.3177740
Show More Cited By

Index Terms

Effective distributed scheduling of parallel workloads

Recommendations

Effective distributed scheduling of parallel workloads
SIGMETRICS '96: Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

We present a distributed algorithm for time-sharing parallel workloads that is competitive with coscheduling. Implicit scheduling allows each local scheduler in the system to make independent decisions that dynamically coordinate the scheduling of ...
Simulation Based Job Scheduling Optimization for Batch Workloads
ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

We present a simulation based approach for scheduling jobs that are part of a batch workflow. Our objective is to minimize the makespan, defined as completion time of the last job to leave the system in a batch workflow with dependencies. The existing ...
Parallel job scheduling algorithms

Comments

Information & Contributors

Information

Published In

cover image ACM SIGMETRICS Performance Evaluation Review

ACM SIGMETRICS Performance Evaluation Review Volume 24, Issue 1

May 1996

273 pages

ISSN:0163-5999

DOI:10.1145/233008

Editors:
Blaine E. Gaither
Hewlett-Packard, Fort Collins, CO
,
Daniel A. Reed
Univ. of Illinois, Urbana

Issue’s Table of Contents

SIGMETRICS '96: Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
May 1996
279 pages
ISBN:0897917936
DOI:10.1145/233013
Chairman:
Daniel A. Reed
Univ. of Illinois, Urbana
,
Editor:
Blaine D. Gaither
Hewlett-Packard, Fort Collins, CO

Copyright © 1996 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 May 1996

Published in SIGMETRICS Volume 24, Issue 1

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

110
Total Citations
View Citations
1,785
Total Downloads

Downloads (Last 12 months)192
Downloads (Last 6 weeks)34

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Grosu GNistor SVasile MCiobanu RDobre CPop F(2023)Local and Global Scheduling in Mobile Drop Computing2023 22nd International Symposium on Parallel and Distributed Computing (ISPDC)10.1109/ISPDC59212.2023.00026(85-92)Online publication date: Jul-2023
https://doi.org/10.1109/ISPDC59212.2023.00026
Jia WZhang JShan JLi JDing XGavrilovska AAltınbüken DBinnig C(2022)Achieving low latency in public edges by hiding workloads mutual interferenceProceedings of the 13th Symposium on Cloud Computing10.1145/3542929.3563459(477-492)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3542929.3563459
Tian BHuang JMozafari BSchoenebeck G(2018)Contention-aware lock scheduling for transactional databasesProceedings of the VLDB Endowment10.1145/3187009.317774011:5(648-662)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1145/3187009.3177740
Tian BHuang JMozafari BSchoenebeck G(2018)Contention-aware lock scheduling for transactional databasesProceedings of the VLDB Endowment10.1145/3177732.317774011:5(648-662)Online publication date: 5-Oct-2018
https://dl.acm.org/doi/10.1145/3177732.3177740
Wu SPeng YJin H(2016)Time Donating Barrier for efficient task scheduling in competitive multicore systemsFuture Generation Computer Systems10.1016/j.future.2015.04.00554(469-477)Online publication date: Jan-2016
https://doi.org/10.1016/j.future.2015.04.005
Song Wu Haibao Chen Sheng Di Bingbing Zhou Zhenjiang Xie Hai Jin Xuanhua Shi (2015)Synchronization-Aware Scheduling for Virtual Clusters in CloudIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2014.235901726:10(2890-2902)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1109/TPDS.2014.2359017
Das SG. SAshok V(2014)Learning Based Performance and Power Efficient Cluster Resource Manager for CPU-GPU ClusterProceedings of the 2014 Fourth International Conference of Emerging Applications of Information Technology10.1109/EAIT.2014.58(161-166)Online publication date: 19-Dec-2014
https://dl.acm.org/doi/10.1109/EAIT.2014.58
Sajjapongse KWang XBecchi MParashar MWeissman JEpema DFigueiredo R(2013)A preemption-based runtime to efficiently schedule multi-process applications on heterogeneous clusters with GPUsProceedings of the 22nd international symposium on High-performance parallel and distributed computing10.1145/2462902.2462911(179-190)Online publication date: 17-Jun-2013
https://dl.acm.org/doi/10.1145/2462902.2462911
Ravi VBecchi MJiang WAgrawal GChakradhar S(2013)Scheduling concurrent applications on a cluster of CPU-GPU nodesFuture Generation Computer Systems10.1016/j.future.2013.06.00229:8(2262-2271)Online publication date: 1-Oct-2013
https://dl.acm.org/doi/10.1016/j.future.2013.06.002
Brelsford DChochia GFalk NMarthi KSure RBobroff NFong LSeelam S(2013)Partitioned Parallel Job Scheduling for Extreme Scale ComputingJob Scheduling Strategies for Parallel Processing10.1007/978-3-642-35867-8_9(157-177)Online publication date: 2013
https://doi.org/10.1007/978-3-642-35867-8_9
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents