research-article

Free access

Sincronia: near-optimal network design for coflows

Authors:

Saksham Agarwal,

Shijin Rajakrishnan,

Akshay Narayan,

Rachit Agarwal,

David Shmoys, and

Amin VahdatAuthors Info & Claims

SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

August 2018

Pages 16 - 29

https://doi.org/10.1145/3230543.3230569

Published: 07 August 2018 Publication History

Abstract

We present Sincronia, a near-optimal network design for coflows that can be implemented on top on any transport layer (for flows) that supports priority scheduling. Sincronia achieves this using a key technical result --- we show that given a "right" ordering of coflows, any per-flow rate allocation mechanism achieves average coflow completion time within 4X of the optimal as long as (co)flows are prioritized with respect to the ordering.

Sincronia uses a simple greedy mechanism to periodically order all unfinished coflows; each host sets priorities for its flows using corresponding coflow order and offloads the flow scheduling and rate allocation to the underlying priority-enabled transport layer. We evaluate Sincronia over a real testbed comprising 16-servers and commodity switches, and using simulations across a variety of workloads. Evaluation results suggest that Sincronia not only admits a practical, near-optimal design but also improves upon state-of-the-art network designs for coflows (sometimes by as much as 8X).

References

[1]

2018. Sincronia Repository. https://github.com/sincronia-coflow.

[2]

Saksham Agarwal, Shijin Rajakrishnan, Akshay Narayan, Rachit Agarwal, David Shmoys, and Amin Vahdat. 2018. Sincronia: Near-Optimal Network Design for Coflows. In Tech Report.

Digital Library

[3]

Saba Ahmadi, Samir Khuller, Manish Purohit, and Sheng Yang. 2017. On scheduling coflows. In MOS IPCO.

[4]

Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013. pFabric: Minimal Near-optimal Datacenter Transport. In ACM SIGCOMM.

Digital Library

[5]

Nikhil Bansal and Subhash Khot. 2010. Inapproximability of hyper-graph vertex cover and applications to scheduling problems. In EATCS ICALP.

Digital Library

[6]

Kwok Ho Chan, Jozef Babiarz, and Fred Baker. 2006. Configuration Guidelines for DiffServ Service Classes. https://tools/ietf.org/html/rfc4594.

[7]

Mosharaf Chowdhury and Ion Stoica. 2012. Coflow: A networking abstraction for cluster applications. In ACM HotNets.

Digital Library

[8]

Mosharaf Chowdhury and Ion Stoica. 2015. Efficient coflow scheduling without prior knowledge. In ACM SIGCOMM.

Digital Library

[9]

Mosharaf Chowdhury, Matei Zaharia, Justin Ma, Michael I Jordan, and Ion Stoica. 2011. Managing data transfers in computer clusters with orchestra. In ACM SIGCOMM.

Digital Library

[10]

Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica. 2014. Efficient coflow scheduling with varys. In ACM SIGCOMM.

Digital Library

[11]

Jeffrey Dean and Sanjay Ghemawat. 2004. MapReduce: simplified data processing on large clusters. In USENIX OSDI.

Digital Library

[12]

Fahad R Dogar, Thomas Karagiannis, Hitesh Ballani, and Antony Rowstron. 2014. Decentralized task-aware scheduling for data center networks. In ACM SIGCOMM.

Digital Library

[13]

Peter X Gao, Akshay Narayan, Gautam Kumar, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2015. phost: Distributed near-optimal datacenter transport over commodity network fabric. In ACM CoNEXT.

Digital Library

[14]

Naveen Garg, Amit Kumar, and Vinayaka Pandit. 2007. Order scheduling models: hardness and algorithms. In IARCS FSTTCS.

Digital Library

[15]

Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew Moore, Gianni Antichi, and Marcin Wojcik. 2017. Re-architecting datacenter networks and stacks for low latency and high performance. In ACM SIGCOMM.

Digital Library

[16]

Chi-Yao Hong, Matthew Caesar, and P Godfrey. 2012. Finishing flows quickly with preemptive scheduling. In ACM SIGCOMM.

Digital Library

[17]

Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In ACM EuroSys.

Digital Library

[18]

Hamidreza Jahanjou, Erez Kantor, and Rajmohan Rajaraman. 2017. Asymptotically Optimal Approximation Algorithms for Coflow Scheduling. In ACM SPAA.

Digital Library

[19]

Samir Khuller, Jingling Li, Pascal Sturmfels, Kevin Sun, and Prayaag Venkat. 2018. Select and Permute: An Improved Online Framework for Scheduling to Minimize Weighted Completion Time. In LATIN.

[20]

Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M Hellerstein. 2012. Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment, 5(8): 716--727.

Digital Library

[21]

Grzegorz Malewicz, Matthew H Austern, Aart JC Bik, James C Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: a system for large-scale graph processing. In ACM SIGMOD.

Digital Library

[22]

Zhen Qiu, Cliff Stein, and Yuan Zhong. 2015. Minimizing the total weighted completion time of coflows in datacenter networks. In ACM SPAA.

Digital Library

[23]

Thomas A. Roemer. 2006. A note on the complexity of the concurrent open shop problem. In Journal of Scheduling, 9(4): 389--396. Springer.

Digital Library

[24]

Sushant Sachdeva and Rishi Saket. 2013. Optimal inapproximability for scheduling problems via structural hardness for hypergraph vertex cover. In IEEE CCC.

[25]

Christo Wilson, Hitesh Ballani, Thomas Karagiannis, and Ant Rowtron. 2011. Better never than late: Meeting deadlines in datacenter networks. In ACM SIGCOMM.

Digital Library

[26]

Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, and Jon Currey. 2008. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In USENIX OSDI.

Digital Library

[27]

Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In USENIX NSDI.

Digital Library

[28]

Hong Zhang, Li Chen, Bairen Yi, Kai Chen, Mosharaf Chowdhury, and Yanhui Geng. 2016. CODA: Toward automatically identifying and scheduling coflows in the dark. In ACM SIGCOMM.

Digital Library

[29]

Yangming Zhao, Kai Chen, Wei Bai, Minlan Yu, Chen Tian, Yanhui Geng, Yiming Zhang, Dan Li, and Sheng Wang. 2015. Rapier: Integrating routing and scheduling for coflow-aware data center networks. In IEEE INFOCOM.

Cited By

Ferreira MAtre NSherry JSobrinho JKuznetsov PGelles ROlivetti D(2024)Impossibility Results for Data-Center Routing with Congestion Control and Unsplittable FlowsProceedings of the 43rd ACM Symposium on Principles of Distributed Computing10.1145/3662158.3662777(358-368)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3662158.3662777
Wang XShen HTian H(2024)Scheduling Coflows in Hybrid Optical-Circuit and Electrical-Packet Switches With Performance GuaranteeIEEE/ACM Transactions on Networking10.1109/TNET.2024.335424532:3(2299-2314)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3354245
Brun OEl-Azouzi RLuu QDe Pellegrini FPrabhu BRichier C(2024)Weighted Scheduling of Time-Sensitive CoflowsIEEE Transactions on Cloud Computing10.1109/TCC.2024.338451412:2(644-658)Online publication date: Apr-2024
https://doi.org/10.1109/TCC.2024.3384514
Show More Cited By

Index Terms

Sincronia: near-optimal network design for coflows
1. Networks
  1. Network protocols
    1. Network protocol design
2. Theory of computation
  1. Design and analysis of algorithms
    1. Approximation algorithms analysis
      1. Scheduling algorithms

Recommendations

Efficient coflow scheduling with Varys
SIGCOMM '14: Proceedings of the 2014 ACM conference on SIGCOMM

Communication in data-parallel applications often involves a collection of parallel flows. Traditional techniques to optimize flow-level metrics do not perform well in optimizing such collections, because the network is largely agnostic to application-...
Read More
PROSE: Multi-round fair coflow scheduling without prior knowledge
Abstract
Coflow scheduling plays a crucial role in enhancing network-level performance in datacenter applications. Existing works on coflow scheduling predominantly concentrate on clairvoyant schedulers, which assume prior knowledge of coflow ...
Highlights
- Fair non-clairvoyant coflow scheduler for non-cooperative environments.
- Two-...
Read More
Saath: Speeding up CoFlows by Exploiting the Spatial Dimension
CoNEXT '17: Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies

CoFlow scheduling improves data-intensive application performance by improving their networking performance. State-of-the-art CoFlow schedulers in essence approximate the classic online Shortest-Job-First (SJF) scheduling, designed for a single CPU, in ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

August 2018

604 pages

ISBN:9781450355674

DOI:10.1145/3230543

General Chairs:
Sergey Gorinsky
IMDEA, Spain
,
János Tapolcai
BME, Hungary

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '18

Sponsor:

SIGCOMM

SIGCOMM '18: ACM SIGCOMM 2018 Conference

August 20 - 25, 2018

Budapest, Hungary

Acceptance Rates

Overall Acceptance Rate 554 of 3,547 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
2,583
Total Downloads

Downloads (Last 12 months)172
Downloads (Last 6 weeks)22

Other Metrics

View Author Metrics

Citations

Cited By

Ferreira MAtre NSherry JSobrinho JKuznetsov PGelles ROlivetti D(2024)Impossibility Results for Data-Center Routing with Congestion Control and Unsplittable FlowsProceedings of the 43rd ACM Symposium on Principles of Distributed Computing10.1145/3662158.3662777(358-368)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3662158.3662777
Wang XShen HTian H(2024)Scheduling Coflows in Hybrid Optical-Circuit and Electrical-Packet Switches With Performance GuaranteeIEEE/ACM Transactions on Networking10.1109/TNET.2024.335424532:3(2299-2314)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3354245
Brun OEl-Azouzi RLuu QDe Pellegrini FPrabhu BRichier C(2024)Weighted Scheduling of Time-Sensitive CoflowsIEEE Transactions on Cloud Computing10.1109/TCC.2024.338451412:2(644-658)Online publication date: Apr-2024
https://doi.org/10.1109/TCC.2024.3384514
Chen C(2024)Efficient Approximation Algorithms for Scheduling Coflows With Total Weighted Completion Time in Identical Parallel NetworksIEEE Transactions on Cloud Computing10.1109/TCC.2023.334072912:1(116-129)Online publication date: Jan-2024
https://doi.org/10.1109/TCC.2023.3340729
Wang XXu SZhao Y(2024)SARS: Towards minimizing average Coflow Completion Time in MapReduce systemsComputer Networks10.1016/j.comnet.2024.110429247(110429)Online publication date: Jun-2024
https://doi.org/10.1016/j.comnet.2024.110429
Katebzadeh MCosta PGrot BFedorova ANarayanan DDi Luna GQuerzoni L(2023)Saba: Rethinking Datacenter Network Allocation from Application's PerspectiveProceedings of the Eighteenth European Conference on Computer Systems10.1145/3552326.3587450(623-638)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.1145/3552326.3587450
Oubaydallah YIbrahimi KAzouzi R(2023)Recent Advances in Data Intensive Applications: Survey2023 10th International Conference on Wireless Networks and Mobile Communications (WINCOM)10.1109/WINCOM59760.2023.10322920(1-6)Online publication date: 26-Oct-2023
https://doi.org/10.1109/WINCOM59760.2023.10322920
Jiang WLi HYan YJi FHuang JWang JZhang T(2023)Consistent Low Latency Scheduler for Distributed Key-Value StoresIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.331577734:12(3012-3027)Online publication date: Dec-2023
https://doi.org/10.1109/TPDS.2023.3315777
Ruan CWang JJiang WZhang T(2023)Scheduling Coflows by Online Identification in Data Center NetworkIEEE Transactions on Emerging Topics in Computing10.1109/TETC.2023.331551211:4(1057-1069)Online publication date: Oct-2023
https://doi.org/10.1109/TETC.2023.3315512
Liu LGao CWang PHuang HLi JXu HZhang W(2023)Bottleneck-Aware Non-Clairvoyant Coflow Scheduling With FaiIEEE Transactions on Cloud Computing10.1109/TCC.2021.312836011:1(1011-1025)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TCC.2021.3128360
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents