research-article

The X-flex cross-platform scheduler: who's the fairest of them all?

Authors:

Viswanath Nagarajan,

Robert Saccone,

Kirsten Hildrum,

Kanthi SarpatwarAuthors Info & Claims

Industry papers: Proceedings of the Middleware Industry Track

Article No.: 1, Pages 1 - 7

https://doi.org/10.1145/2676727.2676728

Published: 08 December 2014 Publication History

Abstract

We introduce the X-Flex cross-platform scheduler. X-Flex is intended as an alternative to the Dominant Resource Fairness (DRF) scheduler currently employed by both YARN and Mesos. There are multiple design differences between X-Flex and DRF. For one thing, DRF is based on an instantaneous notion of fairness, while X-Flex monitors instantaneous fairness in order to take a long-term view. The definition of instantaneous fairness itself is different among the two schedulers. Furthermore, the packing of containers into processing nodes in DRF is done online, while in X-Flex it is performed offline in order to improve packing quality. Finally, DRF is essentially an extension to multiple dimensions of the Fair MapReduce scheduler. As such it makes scheduling decisions at a very low level. X-Flex, on the other hand, takes the perspective that some frameworks have sufficient structure to make higher level scheduling decisions. So X-Flex allows this, and also gives platforms a great deal of autonomy over the degree of sharing they will permit with other platforms. We describe the technical details of X-Flex and provide experiments to show its excellent performance.

References

[1]

Y. Azar, I. Cohen, S. Kamara and B. Shepherd. Tight Bounds for Online V Bin Packing. In Proceedings of STOC, 2013.

Digital Library

[2]

J. Boyar, K. Larsen and M. Nielsen. The Accommodating Function - A Generalization of the Competitive Ratio. In Proceedings of IADS, 1999.

Digital Library

[3]

C. Chekura and S. Khanna A PTAS for the Multiple Knapsack Problem. In Proceedings of SODA, 2000.

Digital Library

[4]

R. Cohen, L. Katzir and D. Raz. An Efficient Approximation for the Generalized Assignment Problem. Information Processing Letters, 100(4): 162--166, 2006.

Digital Library

[5]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. ACM Transactions on Computer Systems, 51(1):107--113, 2008.

Digital Library

[6]

C. Delimitrou and C. Kozyrakis. Quasar: Resource-Efficient and QoS-Aware Cluster Management. In Proceedings of ASPLOS, 2014.

Digital Library

[7]

W. De Pauw, J. Wolf and A. Balmin. Visualizing Jobs with Shared Resources in Distributed Environments. In IEEE Working Conference on Software Visualization, 2013.

[8]

D. Dolev, D. Feitelson, J. Halpern, R. Kupferman and N. Linial. No Justified Complaints: Fair Sharing of Multiple Resources. In Proceedings of ITCS, 2012.

Digital Library

[9]

M. Drozdowski. Scheduling for Parallel Processing. Springer.

Digital Library

[10]

www.wikipedia.org/wiki/Greta_Garbo

[11]

A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker and I. Stoica. Dominant Resource Fairness: Fair Allocation of Multiple Resource Types. In Proceedings of NSDI, 2011.

Digital Library

[12]

R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao and A. Akella. Multi-Resource Packing for Cluster Schedulers. In Proceedings of ACM SIGCOMM, 2014.

Digital Library

[13]

B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. Joseph, R. Katz, S. Shenker and I. Stoica. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In Proceedings of NSDI, 2011.

Digital Library

[14]

Z. Nabi, R. Wagle and E. Bouillet. The Best of Two Worlds: Integrating IBM InfoSphere Streams with Apache YARN. In Proceedings of IEEE Big Data, 2014.

[15]

V. Nagarajan, J. Wolf, A. Balmin and K. Hildrum. FlowFlex: Malleable Scheduling for Flows of MapReduce Jobs. In Proceedings of Middleware, 2013

[16]

K. Ousterhout, P. Wendell, M. Zaharia and I. Stoica. Sparrow: Distributed, Low Latency Scheduling. In Proceedings of SOSP, 2013.

Digital Library

[17]

D. Parkes, A. Procaccia and N. Shah. Beyond Dominant Resource Fairness: Extensions, Limitations, and Indivisibilities. In Proceedings of EC, 2012.

Digital Library

[18]

M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek and J. Wilkes. Omega: Flexible, Scalable Schedulers for Large Compute Clusters. In Proceedings of EuroSys, 2013.

Digital Library

[19]

G. Staples. TORQUE Resource Manager. In Proceedings of Supercomputing, 2006.

Digital Library

[20]

IBM Infosphere Streams www.ibm.com/software/products/en/infosphere-streams.

[21]

IBM Infosphere Streams/Resource Managers Project https://github.com/IBMStreams/resourceManagers.

[22]

D. Thain, T. Tannenbaum, Todd and M. Livny. Distributed Computing in Practice: The Condor Experience: Research Articles. Concurrency and Computation: Practice & Experience, 17:(2-4), 323--356, 2005.

Digital Library

[23]

V. Vavilapalli, A. Murthy, C. Douglis, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed and E. Baldeschwieler. Apache Hadoop YARN: Yet Another Resource Negotiator. In Proceedings of SoCC, 2013.

Digital Library

[24]

V. Vazirani. Approximation Algorithms. Springer.

[25]

J. Wolf, A. Balmin, D. Rajan, K. Hildrum, R. Khandekar, S. Parekh, K.-L. Wu and R. Vernica. On the Optimization of Schedules for MapReduce Workloads in the Presence of Shared Scans. VLDB Journal, 21(5): 589--609, 2012.

Digital Library

[26]

J. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, K.-L. Wu and L. Fleischer. SODA: An Optimizing Scheduler for Large-Scale Stream-Based Distributed Computing Systems. In Proceedings of Middleware, 2008.

Digital Library

[27]

J. Wolf, Z. Nabi, V. Nagarajan, R. Saccone, R. Wagle, K. Hildrum, E. Pring and K. Sarpatwar. The X-Flex Cross-Platform Scheduler. IBM RC, 2014.

Digital Library

[28]

J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh, K. L. Wu and A. Balmin. Flex: A Slot Allocation Scheduling Optimizer for MapReduce Workloads. In Proceedings of Middleware, 2010.

Digital Library

[29]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker and I. Stoica. Resilient Distributed Datasets: A Fault-tolerant Abstraction for In-memory Cluster Computing. In Proceedings of USENIX -- Networked Systems Design and Implementation, 2012.

Digital Library

Cited By

Hanif MLee C(2019)Jargon of Hadoop MapReduce scheduling techniques: a scientific categorizationThe Knowledge Engineering Review10.1017/S026988891800037134Online publication date: 15-Mar-2019
https://doi.org/10.1017/S0269888918000371
Wu XLoiseau P(2015)Algorithms for scheduling deadline-sensitive malleable tasks2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)10.1109/ALLERTON.2015.7447050(530-537)Online publication date: Sep-2015
https://doi.org/10.1109/ALLERTON.2015.7447050

Index Terms

The X-flex cross-platform scheduler: who's the fairest of them all?
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management
        Scheduling

Recommendations

Evaluation and Analysis of Capacity Scheduler and Fair Scheduler in Hadoop Framework on Big Data Technology
AIVR 2018: Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual Reality

Apache Hadoop is an open source framework that implements MapReduce. It is scalable, reliable, and fault tolerant. Scheduling is an important process in Hadoop MapReduce. It is because scheduling has responsibility to allocate resources for running ...
A hard real-time scheduler for Spark on YARN
CCGrid '18: Proceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Apache Spark is a fast and general engine for large-scale data processing using distributed memory. It provides different deploy modes to meet the needs of different users and Spark on YARN is the most popular deploy mode. Different deploy modes have ...
Pro Smartphone Cross-Platform Development: iPhone, Blackberry, Windows Mobile and Android Development and Distribution

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

Industry papers: Proceedings of the Middleware Industry Track

December 2014

37 pages

ISBN:9781450332194

DOI:10.1145/2676727

Conference Chairs:
Thierry Coupaye
Orange Labs, France
,
Michael A. Kozuch
Intel Labs

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Middleware '14

Middleware '14: 15th International Middleware Conference

December 8 - 12, 2014

Bordeaux, France

Acceptance Rates

Industry papers Paper Acceptance Rate 5 of 23 submissions, 22%;

Overall Acceptance Rate 5 of 23 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
158
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)2

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hanif MLee C(2019)Jargon of Hadoop MapReduce scheduling techniques: a scientific categorizationThe Knowledge Engineering Review10.1017/S026988891800037134Online publication date: 15-Mar-2019
https://doi.org/10.1017/S0269888918000371
Wu XLoiseau P(2015)Algorithms for scheduling deadline-sensitive malleable tasks2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)10.1109/ALLERTON.2015.7447050(530-537)Online publication date: Sep-2015
https://doi.org/10.1109/ALLERTON.2015.7447050

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents