research-article

Open access

Load Balancing Guardrails: Keeping Your Heavy Traffic on the Road to Low Response Times

Authors:

Mor Harchol-BalterAuthors Info & Claims

Proceedings of the ACM on Measurement and Analysis of Computing Systems, Volume 3, Issue 2

Article No.: 42, Pages 1 - 31

https://doi.org/10.1145/3341617.3326157

Published: 19 June 2019 Publication History

Abstract

Load balancing systems, comprising a central dispatcher and a scheduling policy at each server, are widely used in practice, and their response time has been extensively studied in the theoretical literature. While much is known about the scenario where the scheduling at the servers is First-Come-First-Served (FCFS), to minimize mean response time we must use Shortest-Remaining-Processing-Time (SRPT) scheduling at the servers. Much less is known about dispatching polices when SRPT scheduling is used. Unfortunately, traditional dispatching policies that are used in practice in systems with FCFS servers often have poor performance in systems with SRPT servers. In this paper, we devise a simple fix that can be applied to any dispatching policy. This fix, called guardrails, ensures that the dispatching policy yields optimal mean response time under heavy traffic when used in a system with SRPT servers. Any dispatching policy, when augmented with guardrails, becomes heavy-traffic optimal. Our results yield the first analytical bounds on mean response time for load balancing systems with SRPT scheduling at the servers.

References

[1]

Samuli Aalto, Urtzi Ayesta, and Rhonda Righter. 2009. On the Gittins index in the M/G/1 queue. Queueing Systems, Vol. 63, 1 (2009), 437--458.

Digital Library

[2]

E. Altman, U. Ayesta, and B. J. Prabhu. 2011. Load balancing in processor sharing systems. Telecommunication Systems, Vol. 47, 1 (01 Jun 2011), 35--48.

Digital Library

[3]

Nir Avrahami and Yossi Azar. 2003. Minimizing Total Flow Time and Total Completion Time with Immediate Dispatching. In Proceedings of the Fifteenth Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA '03). ACM, New York, NY, USA, 11--18.

Digital Library

[4]

Eitan Bachmat and Hagit Sarfati. 2008. Analysis of Size Interval Task Assignment Policies. SIGMETRICS Perform. Eval. Rev., Vol. 36, 2 (Aug. 2008), 107--109.

Digital Library

[5]

T. Bonald, M. Jonckheere, and A. Proutiére. 2004. Insensitive Load Balancing. In Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS '04/Performance '04). ACM, New York, NY, USA, 367--377.

Digital Library

[6]

F. Bonomi. 1990. On job assignment for a parallel system of processor sharing queues. IEEE Trans. Comput., Vol. 39, 7 (July 1990), 858--869.

Digital Library

[7]

Maury Bramson, Yi Lu, and Balaji Prabhakar. 2012. Asymptotic independence of queues under randomized load balancing. Queueing Systems, Vol. 71, 3 (01 Jul 2012), 247--292.

Digital Library

[8]

Rodolpho G. de Siqueira and Daniel R. Figueiredo. 2017. A Control-based Load Balancing Algorithm with Flow Control for Dynamic and Heterogeneous Servers. In Anais do XXXV Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos. SBC, Porto Alegre, RS, Brasil. http://portaldeconteudo.sbc.org.br/index.php/sbrc/article/view/2626

[9]

Douglas G. Down and Rong Wu. 2006. Multi-layered round robin routing for parallel servers. Queueing Systems, Vol. 53, 4 (01 Aug 2006), 177--188.

Digital Library

[10]

Hanhua Feng, Vishal Misra, and Dan Rubenstein. 2005. Optimal state-free, size-aware dispatching for heterogeneous M/G/-type systems. Performance Evaluation, Vol. 62, 1 (2005), 475 -- 492. Performance 2005.

Digital Library

[11]

John C. Gittins, Kevin D. Glazebrook, and Richard Weber. 2011. Multi-armed Bandit Allocation Indices .John Wiley & Sons.

[12]

Varun Gupta, Mor Harchol Balter, Karl Sigman, and Ward Whitt. 2007. Analysis of join-the-shortest-queue routing for web server farms. Performance Evaluation, Vol. 64, 9 (2007), 1062--1081. Performance 2007.

Digital Library

[13]

Mor Harchol-Balter. 2013. Performance Modeling and Design of Computer Systems: Queueing Theory in Action 1st ed.). Cambridge University Press, New York, NY, USA.

Digital Library

[14]

Mor Harchol-Balter, Mark E. Crovella, and Cristina D. Murta. 1999. On Choosing a Task Assignment Policy for a Distributed Server System. J. Parallel and Distrib. Comput., Vol. 59, 2 (1999), 204--228.

Digital Library

[15]

Mor Harchol-Balter, Alan Scheller-Wolf, and Andrew R. Young. 2009. Surprising Results on Task Assignment in Server Farms with High-variability Workloads. In Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems (SIGMETRICS '09). ACM, New York, NY, USA, 287--298.

Digital Library

[16]

Mor Harchol-Balter, Bianca Schroeder, Nikhil Bansal, and Mukesh Agrawal. 2003. Size-based Scheduling to Improve Web Performance. ACM Trans. Comput. Syst., Vol. 21, 2 (May 2003), 207--233.

Digital Library

[17]

Esa Hyytiä, Aleksi Penttinen, and Samuli Aalto. 2012. Size- and state-aware dispatching problem with queue-specific job sizes. European Journal of Operational Research, Vol. 217, 2 (2012), 357 -- 370.

[18]

Stefano Leonardi and Danny Raz. 2007. Approximating total flow time on parallel machines. J. Comput. System Sci., Vol. 73, 6 (2007), 875--891.

Digital Library

[19]

Quan-Lin Li, John C. S. Lui, and Yang Wang. 2011. A Matrix-Analytic Solution for Randomized Load Balancing Models with PH Service Times .Springer Berlin Heidelberg, Berlin, Heidelberg, 240--253.

[20]

Minghong Lin, Adam Wierman, and Bert Zwart. 2011. Heavy-traffic analysis of mean response time under Shortest Remaining Processing Time. Performance Evaluation (2011).

[21]

Zhen Liu and Rhonda Righter. 1998. Optimal Load Balancing on Distributed Homogeneous Unreliable Processors. Operations Research, Vol. 46, 4 (1998), 563--573. ://www.jstor.org/stable/168596

Digital Library

[22]

Richard R. Weber. 1978. On the optimal assignment of customers to parallel servers. Journal of Applied Probability, Vol. 15, 2 (1978), 406--413.

[23]

Adam Wierman and Mor Harchol-Balter. 2003. Classifying scheduling policies with respect to unfairness in an M/GI/1. In ACM SIGMETRICS Performance Evaluation Review, Vol. 31. ACM, 238--249.

Digital Library

[24]

Adam Wierman, Mor Harchol-Balter, and Takayuki Osogami. 2005. Nearly insensitive bounds on SMART scheduling. In ACM SIGMETRICS Performance Evaluation Review, Vol. 33. ACM, 205--216.

Digital Library

[25]

Wayne Winston. 1977. Optimality of the shortest line discipline. Journal of Applied Probability, Vol. 14, 1 (1977), 181--189.

[26]

Ronald W. Wolff. 1982. Poisson arrivals see time averages. Operations Research, Vol. 30, 2 (1982), 223--231.

Digital Library

[27]

Xingyu Zhou, Jian Tan, and Ness Shroff. 2018. Flexible load balancing with multi-dimensional state-space collapse: Throughput and heavy-traffic delay optimality. Performance Evaluation, Vol. 127--128 (2018), 176 -- 193.

Cited By

Xie RGrosof IScully Z(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingACM SIGMETRICS Performance Evaluation Review10.1145/3673660.365505952:1(7-8)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3673660.3655059
Xie RGrosof IScully ZGaretto MMarin ACiucu FFanti GRighter R(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingAbstracts of the 2024 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3652963.3655059(7-8)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3652963.3655059
Xie RGrosof IScully Z(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36390358:1(1-36)Online publication date: 21-Feb-2024
https://dl.acm.org/doi/10.1145/3639035
Show More Cited By

Index Terms

Recommendations

Optimal Scheduling in the Multiserver-job Model under Heavy Traffic
POMACS

Multiserver-job systems, where jobs require concurrent service at many servers, occur widely in practice. Essentially all of the theoretical work on multiserver-job systems focuses on maximizing utilization, with almost nothing known about mean response ...
Load Balancing Guardrails: Keeping Your Heavy Traffic on the Road to Low Response Times

Load balancing systems, comprising a central dispatcher and a scheduling policy at each server, are widely used in practice, and their response time has been extensively studied in the theoretical literature. While much is known about the scenario where ...
Load Balancing Guardrails: Keeping Your Heavy Traffic on the Road to Low Response Times
SIGMETRICS '19: Abstracts of the 2019 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer Systems

Load balancing systems, comprising a central dispatcher and a scheduling policy at each server, are widely used in practice, and their response time has been extensively studied in the theoretical literature. While much is known about the scenario where ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Measurement and Analysis of Computing Systems

Proceedings of the ACM on Measurement and Analysis of Computing Systems Volume 3, Issue 2

June 2019

683 pages

EISSN:2476-1249

DOI:10.1145/3341617

Editors:
Augustin Chaintreau
Columbia University
,
Thomas Bonald
Telecom ParisTech
,
Nick Duffield
Texas A&M University

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 June 2019

Published in POMACS Volume 3, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
1,488
Total Downloads

Downloads (Last 12 months)1,126
Downloads (Last 6 weeks)773

Reflects downloads up to 22 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie RGrosof IScully Z(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingACM SIGMETRICS Performance Evaluation Review10.1145/3673660.365505952:1(7-8)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3673660.3655059
Xie RGrosof IScully ZGaretto MMarin ACiucu FFanti GRighter R(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingAbstracts of the 2024 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3652963.3655059(7-8)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3652963.3655059
Xie RGrosof IScully Z(2024)Heavy-Traffic Optimal Size- and State-Aware DispatchingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36390358:1(1-36)Online publication date: 21-Feb-2024
https://dl.acm.org/doi/10.1145/3639035
Hong YScully Z(2024)Performance of the Gittins policy in the G/G/1 and G/G/k, with and without setup timesPerformance Evaluation10.1016/j.peva.2023.102377163:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.peva.2023.102377
der Boor MBorst SVan Leeuwaarden JMukherjee D(2022)Scalable Load Balancing in Networked SystemsSIAM Review10.1137/20M132374664:3(554-622)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1137/20M1323746
Harchol-Balter M(2021)Open problems in queueing theory inspired by datacenter computingQueueing Systems: Theory and Applications10.1007/s11134-020-09684-697:1-2(3-37)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1007/s11134-020-09684-6
Elahi MMarin ARossi SWilliamson C(2020)Frequency scaling in multilevel queuesPerformance Evaluation10.1016/j.peva.2020.102140143(102140)Online publication date: Nov-2020
https://doi.org/10.1016/j.peva.2020.102140
Horváth IScully ZVan Houdt B(2019)Mean Field Analysis of Join-Below-Threshold Load Balancing for Resource Sharing ServersProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33667053:3(1-21)Online publication date: 17-Dec-2019
https://dl.acm.org/doi/10.1145/3366705
Grosof IScully ZHarchol-Balter MNahum EBonald TDuffield N(2019)Load Balancing GuardrailsAbstracts of the 2019 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3309697.3331514(9-10)Online publication date: 20-Jun-2019
https://dl.acm.org/doi/10.1145/3309697.3331514

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents