Article

C3: cutting tail latency in cloud data stores via adaptive replica selection

Authors:

Anja FeldmannAuthors Info & Claims

NSDI'15: Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation

Pages 513 - 527

Published: 04 May 2015 Publication History

Abstract

Achieving predictable performance is critical for many distributed applications, yet difficult to achieve due to many factors that skew the tail of the latency distribution even in well-provisioned systems. In this paper, we present the fundamental challenges involved in designing a replica selection scheme that is robust in the face of performance fluctuations across servers. We illustrate these challenges through performance evaluations of the Cassandra distributed database on Amazon EC2. We then present the design and implementation of an adaptive replica selection mechanism, C3, that is robust to performance variability in the environment. We demonstrate C3's effectiveness in reducing the latency tail and improving throughput through extensive evaluations on Amazon EC2 and through simulations. Our results show that C3 significantly improves the latencies along the mean, median, and tail (up to 3 times improvement at the 99.9th percentile) and provides higher system throughput.

References

[1]

Akka. http://akka.io/, accessed Sept 25, 2014.

[2]

M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. Data Center TCP (DCTCP). In SIGCOMM, 2010.

Digital Library

[3]

Amazon ELB. http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html, accessed Sept 24, 2014.

[4]

G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the Outliers in Map-reduce Clusters Using Mantri. In OSDI, 2010.

Digital Library

[5]

Apache Cassandra. http://cassandra.apache.org/, accessed June 10, 2013.

[6]

Apache Cassandra Use Cases. http://planetcassandra. org/apache-cassandra-use-cases/, accessed Sept 25, 2014.

[7]

G. R. Arce. Nonlinear Signal Processing: A Statistical Approach. Wiley, 2004.

[8]

Astyanax. https://github.com/Netflix/astyanax, accessed Jan 5, 2015.

[9]

B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, and M. Paleczny. Workload Analysis of a Large-scale Key-value Store. In SIGMETRICS, 2012.

Digital Library

[10]

C. F. Bispo. The single-server scheduling problem with convex costs. Queueing Systems, 73(3), 2013.

Digital Library

[11]

J. Brutlag. Speed Matters, accessed Sept 24, 2014. http://googleresearch.blogspot.com/2009/06/speed-matters.html.

[12]

Cassandra Documentation. http://www.datastax.com/documentation/cassandra/2.0, accessed Sept 25, 2014.

[13]

F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst., 26(2), June 2008.

Digital Library

[14]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In SoCC, 2010.

Digital Library

[15]

DB-Engines Ranking of Wide Column Stores. http: //db-engines.com/en/ranking/wide+column+ store, accessed Sept 25, 2014.

[16]

J. Dean and L. A. Barroso. The Tail At Scale. Communications of the ACM, 56:74-80, 2013.

Digital Library

[17]

G. Decandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's Highly Available Key-value Store. In SOSP, 2007.

Digital Library

[18]

J. Ellis. How not to benchmark Cassandra: a case study, 2014. http://www.datastax.com/dev/blog/how-not-to-benchmark-cassandra-a-case-study.

[19]

A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca. Jockey: Guaranteed Job Latency in Data Parallel Clusters. In EuroSys, 2012.

Digital Library

[20]

W. D. Gray and D. Boehm-Davis. Milliseconds matter: An introduction to microstrategies and to their use in describing and predicting interactive behavior. Journal of Experimental Psychology: Applied, 6, 2000.

[21]

A. Gulati, I. Ahmad, and C. A. Waldspurger. PARDA: Proportional Allocation of Resources for Distributed Storage Access. In FAST, 2009.

Digital Library

[22]

S. Ha, I. Rhee, and L. Xu. CUBIC: A New TCP-Friendly High-Speed TCP Variant. SIGOPS Oper. Syst. Rev., 42(5), 2008.

Digital Library

[23]

V. Jalaparti, P. Bodik, S. Kandula, I. Menache, M. Rybalkin, and C. Yan. Speeding up Distributed Request-Response Workflows. In SIGCOMM, 2013.

Digital Library

[24]

C. Kalantzis. Eventual Consistency != Hopeful Consistency, talk at Cassandra Summit, 2013. https://www.youtube.com/watch?v=A6qzx_HE3EU.

[25]

C. Kalantzis. Revisiting 1 Million Writes per second, 2014. http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html.

[26]

M. Kambadur, T. Moseley, R. Hank, and M. A. Kim. Measuring Interference Between Live Datacenter Applications. In SC, 2012.

Digital Library

[27]

R. Kapoor, G. Porter, M. Tewari, G. M. Voelker, and A. Vahdat. Chronos: Predictable Low Latency for Data Center Applications. In SoCC, 2012.

Digital Library

[28]

J. Li, N. K. Sharma, D. R. K. Ports, and S. D. Gribble. Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency. In SoCC, 2014.

Digital Library

[29]

A. Liljencrantz. How Not to Use Cassandra, talk at Cassandra Summit, 2013. https://www.youtube.com/watch?v=0u-EKJBPrj8.

[30]

C. R. Lumb and R. Golding. D-SPTF: Decentralized Request Distribution in Brick-based Storage Systems. SIGOPS Oper. Syst. Rev., 38(5):37-47, Oct. 2004.

Digital Library

[31]

J. C. McCullough, J. Dunagan, A. Wolman, and A. C. Snoeren. Stout: An Adaptive Interface to Scalable Cloud Storage. In USENIX ATC, 2010.

Digital Library

[32]

M. Mitzenmacher. How Useful Is Old Information? IEEE Trans. Parallel Distrib. Syst., 11(1), Jan. 2000.

Digital Library

[33]

M. Mitzenmacher. The power of two choices in randomized load balancing. IEEE Trans. Parallel Distrib. Syst., 12(10), Oct. 2001.

Digital Library

[34]

Nginx. http://nginx.org/en/docs/http/load_ balancing.html, accessed Sept 24, 2014.

[35]

R. Nishtala, H. Fugal, S. Grimm, M. Kwiatkowski, H. Lee, H. C. Li, R. McElroy, M. Paleczny, D. Peek, P. Saab, D. Stafford, T. Tung, and V. Venkataramani. Scaling Memcache at Facebook. In NSDI, 2013.

Digital Library

[36]

K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Sparrow: Distributed, Low Latency Scheduling. In SOSP, 2013.

Digital Library

[37]

Riak. AWS Performance Tuning, accessed Sept 24, 2014. http://docs.basho.com/riak/latest/ops/tuning/aws/.

[38]

Riak. Load Balancing and Proxy Configuration, accessed Sept 24, 2014. http://docs.basho.com/riak/1.4.0/cookbooks/Load-Balancing-and-Proxy-Configuration/.

[39]

M. Roussopoulos and M. Baker. Practical Load Balancing for Content Requests in Peer-to-Peer Networks. Distributed Computing, 18(6), 2006.

[40]

S. Sanfilippo. Redis latency spikes and the 99th percentile, 2014. http://antirez.com/news/83.

[41]

J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. VLDB Endowment, 3(1-2), Sept. 2010.

Digital Library

[42]

D. Shue, M. J. Freedman, and A. Shaikh. Performance Isolation and Fairness for Multi-tenant Cloud Storage. In OSDI, 2012.

Digital Library

[43]

S. Souders. Velocity and the Bottom Line, 2009. http://radar.oreilly.com/2009/07/velocity-making-your-site-fast.html.

[44]

C. Stewart, A. Chakrabarti, and R. Griffith. Zoolander: Efficiently Meeting Very Strict, Low-Latency SLOs. In USENIX ICAC, 2013.

[45]

R. Sumbaly, J. Kreps, L. Gao, A. Feinberg, C. Soman, and S. Shah. Serving Large-scale Batch Computed Data with Project Voldemort. In FAST, 2012.

Digital Library

[46]

J. A. van Mieghem. Dynamic Scheduling with Convex Delay Costs: The Generalized $c|mu$ Rule. The Annals of Applied Probability, 5, 1995.

[47]

V. Venkataramani, Z. Amsden, N. Bronson, G. Cabrera III, P. Chakka, P. Dimov, H. Ding, J. Ferris, A. Giardullo, J. Hoon, S. Kulkarni, N. Lawrence, M. Marchukov, D. Petrov, and L. Puzar. TAO: How Facebook Serves the Social Graph. In SIGMOD, 2012.

Digital Library

[48]

A. Vulimiri, P. B. Godfrey, R. Mittal, J. Sherry, S. Ratnasamy, and S. Shenker. Low Latency via Redundancy. In CoNEXT, 2013.

Digital Library

[49]

Z. Wu, C. Yu, and H. V. Madhyastha. CosTLO: Cost-Effective Redundancy for Lower Latency Variance on Cloud Storage Services. In NSDI, 2015.

Digital Library

[50]

Y. Xu, Z. Musgrave, B. Noble, and M. Bailey. Bobtail: Avoiding Long Tails in the Cloud. In NSDI, 2013.

Digital Library

[51]

M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving MapReduce Performance in Heterogeneous Environments. In OSDI, 2008.

Digital Library

[52]

X. Zhang, E. Tune, R. Hagmann, R. Jnagal, V. Gokhale, and J. Wilkes. CPI²: CPU performance isolation for shared compute clusters. In EuroSys, 2013.

Digital Library

[53]

T. Zhu, A. Tumanov, M. A. Kozuch, M. Harchol-Balter, and G. R. Ganger. PriorityMeister: Tail Latency QoS for Shared Networked Storage. In SoCC, 2014.

Digital Library

Cited By

Bothra RGandhi RBhagwan RPadmanabhan VLiang RCarlson SKamath VAcharyya SSueda KChaturmohta SSharma HSchulzrinne HKohler EMaltz DMisra V(2023)Switchboard: Efficient Resource Management for Conferencing ServicesProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604879(1000-1011)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604879
Sun XLi KRen YLin JRen ZFeng SJian YQi ZBittencourt LSill A(2021)Client layer becomes bottleneckProceedings of the 14th IEEE/ACM International Conference on Utility and Cloud Computing Companion10.1145/3492323.3495625(1-6)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.1145/3492323.3495625
Zhu HKaffes KChen ZLiu ZKozyrakis CStoica IJin XLu SHowell J(2020)RackSchedProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488835(1225-1240)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488835
Show More Cited By

C3: cutting tail latency in cloud data stores via adaptive replica selection
1. General and reference
  1. Cross-computing tools and techniques

Recommendations

Austere C³ graphics

The MITRE Corporation, a not for profit Federal Contract Research Center (FCRC), has been involved in Command, Control and Communications (C ³) Systems Engineering for the past twenty years. MITRE's program in support of the Defense Communications ...
Austere C³ graphics
SIGGRAPH '81: Proceedings of the 8th annual conference on Computer graphics and interactive techniques

The MITRE Corporation, a not for profit Federal Contract Research Center (FCRC), has been involved in Command, Control and Communications (C ³) Systems Engineering for the past twenty years. MITRE's program in support of the Defense Communications ...
Compliant Cloud Computing (C3): Architecture and Language Support for User-Driven Compliance Management in Clouds
CLOUD '10: Proceedings of the 2010 IEEE 3rd International Conference on Cloud Computing

Cloud computing represents a promising computing paradigm, where computational power is provided similar to utilities like water, electricity or gas. While most of the Cloud providers can guarantee some measurable non-functional performance metrics e.g.,...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NSDI'15: Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation

May 2015

620 pages

ISBN:9781931971218

Program Chairs:
Paul Barham
Google
,
Arvind Krishnamurthy
University of Washington

Sponsors

VMware
NSF: National Science Foundation
Google Inc.
Microsoft Reasearch: Microsoft Reasearch
CISCO

Publisher

USENIX Association

United States

Publication History

Published: 04 May 2015

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bothra RGandhi RBhagwan RPadmanabhan VLiang RCarlson SKamath VAcharyya SSueda KChaturmohta SSharma HSchulzrinne HKohler EMaltz DMisra V(2023)Switchboard: Efficient Resource Management for Conferencing ServicesProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604879(1000-1011)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604879
Sun XLi KRen YLin JRen ZFeng SJian YQi ZBittencourt LSill A(2021)Client layer becomes bottleneckProceedings of the 14th IEEE/ACM International Conference on Utility and Cloud Computing Companion10.1145/3492323.3495625(1-6)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.1145/3492323.3495625
Zhu HKaffes KChen ZLiu ZKozyrakis CStoica IJin XLu SHowell J(2020)RackSchedProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488835(1225-1240)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488835
Estro TBhandari PWildani AZadok EBadam AChidambaram V(2020)Desperately seeking ... optimal multi-tier cache configurationsProceedings of the 12th USENIX Conference on Hot Topics in Storage and File Systems10.5555/3488733.3488739(6-6)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.5555/3488733.3488739
Uta ACustura ADuplyakin DJimenez IRellermeyer JMaltzahn CRicci RIosup ABhagwan RPorter G(2020)Is big data performance reproducible in modern cloud networks?Proceedings of the 17th Usenix Conference on Networked Systems Design and Implementation10.5555/3388242.3388280(513-528)Online publication date: 25-Feb-2020
https://dl.acm.org/doi/10.5555/3388242.3388280
Chen JZhao MLi ZZhai EQian FChen HLiu YXu TNoh SWelch B(2020)Lock-free collaboration support for cloud storage services with operation inference and transformationProceedings of the 18th USENIX Conference on File and Storage Technologies10.5555/3386691.3386694(13-28)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.5555/3386691.3386694
Peng YVarman PYadgar GNoh S(2019)Fair-EDFProceedings of the 11th USENIX Conference on Hot Topics in Storage and File Systems10.5555/3357062.3357070(6-6)Online publication date: 8-Jul-2019
https://dl.acm.org/doi/10.5555/3357062.3357070
Bashir HFaisal AJamshed MVondras PIftikhar AQazi IDogar FMohaisen AZhang Z(2019)Reducing tail latency using duplicationProceedings of the 15th International Conference on Emerging Networking Experiments And Technologies10.1145/3359989.3365432(246-259)Online publication date: 3-Dec-2019
https://dl.acm.org/doi/10.1145/3359989.3365432
Mitra SMondal SSheoran NDhake NNehra RSimha R(2019)DeepPlaceProceedings of the 10th ACM SIGOPS Asia-Pacific Workshop on Systems10.1145/3343737.3343741(61-68)Online publication date: 19-Aug-2019
https://dl.acm.org/doi/10.1145/3343737.3343741
Choi SPark SShahbaz MPrabhakar BRosenblum M(2019)Toward Scalable Replication Systems with Predictable Tails Using Programmable Data PlanesProceedings of the 3rd Asia-Pacific Workshop on Networking10.1145/3343180.3343181(78-84)Online publication date: 17-Aug-2019
https://dl.acm.org/doi/10.1145/3343180.3343181
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents