research-article

High-bandwidth data dissemination for large-scale distributed systems

Authors:

Alex C. Snoeren,

Charles Killian,

James W. Anderson,

Jeannie Albrecht,

Adolfo Rodriguez,

Erik VandekieftAuthors Info & Claims

ACM Transactions on Computer Systems (TOCS), Volume 26, Issue 1

Article No.: 3, Pages 1 - 61

https://doi.org/10.1145/1328671.1328674

Published: 10 March 2008 Publication History

Abstract

This article focuses on the multireceiver data dissemination problem. Initially, IP multicast formed the basis for efficiently supporting such distribution. More recently, overlay networks have emerged to support point-to-multipoint communication. Both techniques focus on constructing trees rooted at the source to distribute content among all interested receivers. We argue, however, that trees have two fundamental limitations for data dissemination. First, since all data comes from a single parent, participants must often continuously probe in search of a parent with an acceptable level of bandwidth. Second, due to packet losses and failures, available bandwidth is monotonically decreasing down the tree.

To address these limitations, we present Bullet, a data dissemination mesh that takes advantage of the computational and storage capabilities of end hosts to create a distribution structure where a node receives data in parallel from multiple peers. For the mesh to deliver improved bandwidth and reliability, we need to solve several key problems: (i) disseminating disjoint data over the mesh, (ii) locating missing content, (iii) finding who to peer with (peering strategy), (iv) retrieving data at the right rate from all peers (flow control), and (v) recovering from failures and adapting to dynamically changing network conditions. Additionally, the system should be self-adjusting and should have few user-adjustable parameter settings. We describe our approach to addressing all of these problems in a working, deployed system across the Internet. Bullet outperforms state-of-the-art systems, including BitTorrent, by 25-70% and exhibits strong performance and reliability in a range of deployment settings. In addition, we find that, relative to tree-based solutions, Bullet reduces the need to perform expensive bandwidth probing.

References

[1]

Ahlswede, R., Cai, N., Li, S.-Y. R., and Yeung, R. W. 2000. Network information flow. IEEE Trans. Inform. Theor. 46, 4 (Jul.), 1204--1216.

Digital Library

[2]

Albrecht, J., Tuttle, C., Snoeren, A. C., and Vahdat, A. 2006. PlanetLab application management using Plush. ACM SIGOPS Operat. Syst. Rev. 40 1 (Jan.), 33--40.

Digital Library

[3]

Andersen, D. G., Balakrishnan, H., Kaashoek, M. F., and Morris, R. 2001. Resilient overlay networks. In Proceedings of the Symposium on Operating Systems Principles.

Digital Library

[4]

Banerjee, S., Bhattacharjee, B., and Kommareddy, C. 2002. Scalable application layer multicast. In Proceedings of ACM SIGCOMM.

Digital Library

[5]

Bharambe, A., Herley, C., and Padmanabhan, V. 2006. Analyzing and improving a BitTorrent network's performance mechanisms. In Proceedings of IEEE INFOCOM.

[6]

Bickson, D. and Malkhi, D. 2005. The Julia content distribution network. In Proceedings of the Second Workshop on Real, Large Distributed Systems (WORLDS).

Digital Library

[7]

Birman, K., Hayden, M., Ozkasap, O., Xiao, Z., Budiu, M., and Minsky, Y. 1999. Bimodal multicast. ACM Trans. Comput. Syst. 17, 2 (May), 41--88.

Digital Library

[8]

Bloom, B. 1970. Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13, 7 (Jul.), 422--426.

Digital Library

[9]

Broder, A. 1997. On the resemblance and containment of documents. In Proceedings of the Conference on Compression and Complexity of Sequences (SEQUENCES'97).

Digital Library

[10]

Byers, J. W., Considine, J., Mitzenmacher, M., and Rost, S. 2002. Informed content delivery across adaptive overlay networks. In Proceedings of ACM SIGCOMM.

Digital Library

[11]

Byers, J. W., Luby, M., Mitzenmacher, M., and Rege, A. 1998. A digital fountain approach to reliable distribution of bulk data. In Proceedings of ACM SIGCOMM.

Digital Library

[12]

Calvert, K., Doar, M., and Zegura, E. W. 1997. Modeling Internet topology. IEEE Commun. Mag. 35, 6 (Jun.), 160--163.

Digital Library

[13]

Castro, M., Druschel, P., Kermarrec, A.-M., Nandi, A., Rowstron, A., and Singh, A. 2003. Splitstream: High-bandwidth content distribution in cooperative environments. In Proceedings of the 19th ACM Symposium on Operating System Principles.

Digital Library

[14]

Chang, H., Govindan, R., Jamin, S., Shenker, S., and Willinger, W. 2002. Towards capturing representative AS-level Internet topologies. In Proceedings of ACM SIGMETRICS.

Digital Library

[15]

Cherkasova, L. and Lee, J. 2003. FastReplica: Efficient large file distribution within content delivery networks. In Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems.

Digital Library

[16]

Chou, P. A., Wu, Y., and Jain, K. 2003. Practical network coding. In Proceedings of the Allerton Conference on Communication, Control, and Computing.

[17]

Cohen, B. 2003. Incentives build robustness in bittorrent. In Proceedings of the First Workshop on the Economics of Peer-to-Peer Systems.

[18]

Cohen, R. and Kaempfer, G. 2001. A unicast-based approach for streaming multicast. In Proceeding of INFOCOM. 440--448.

[19]

Dabek, F., Li, J., Sit, E., Kaashoek, F., Morris, R., and Blake, C. 2004. Designing a DHT for low latency and high throughput. In Proceedings of the USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI, San Francisco, CA).

Digital Library

[20]

Dalal, Y. K. and Metcalfe, R. M. 1978. Reverse path forwarding of broadcast packets. Commun. ACM 21, 12, 1040--1048.

Digital Library

[21]

Deering, S. E. 1991. Multicast routing in a datagram internetwork. Ph.D. dissertation. Stanford University, Stanford, CA.

Digital Library

[22]

Demers, A., Greene, D., Hauser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., and Terry, D. 1987. Epidemic algorithms for replicated database maintenance. In Proceedings of the Sixth Symposium on Principles of Distributed Computing. 1--12.

Digital Library

[23]

Eugster, P., Guerraoui, R., Handurukande, S., Kouznetsov, P., and Kermarrec, A.-M. 2003. Lightweight probabilistic broadcast. ACM Trans. Comput. Syst. 21, 4, 341--374.

Digital Library

[24]

Eugster, P., Handurukande, S., Guerraoui, R., Kermarrec, A.-M., and Kouznetsov, P. 2001. Lightweight probabilistic broadcast. In Proceedings of The International Conference on Dependable Systems and Networks (DSN).

Digital Library

[25]

Floyd, S., Handley, M., Padhye, J., and Widmer, J. 2000. Equation-based congestion control for unicast applications. In Proceedings of ACM SIGCOMM (Stockholm, Sweden). 43-- 56.

Digital Library

[26]

Floyd, S., Jacobson, V., Liu, C.-G., McCanne, S., and Zhang, L. 1997. A reliable multicast framework for light-weight sessions and application level framing. IEEE / ACM Trans. Network. 5, 6, 784--803.

Digital Library

[27]

Gkantsidis, C., Miller, J., and Rodriguez, P. 2006. Anatomy of a p2p content distribution system with network coding. In Proceedings of the Second International Peer to Peer Symposium (IPTPS).

[28]

Gkantsidis, C. and Rodriguez, P. R. 2005. Network coding for large scale content distribution. In Proceedings of IEEE INFOCOM.

[29]

Goyal, V. K. 2001. Multiple description coding: Compression meets the network. IEEE Signal Process. Mag. 18, 5, 74--93.

[30]

Hu, N. and Steenkiste, P. 2003. Evaluation and characterization of available bandwidth probing techniques. IEEE J. Select are Commun. 21, 6, 879--895.

Digital Library

[31]

Hua Chu, Y., Ganjam, A., Ng, T. S. E., Rao, S. G., Sripanidkulchai, K., Zhan, J., and Zhang, H. 2004. Early experience with an Internet broadcast system based on overlay multicast. In Proceedings of the USENIX 2004 Annual Technical Conference.

Digital Library

[32]

Hua Chu, Y., Rao, S., and Zhang, H. 2000. A case for end system multicast. In Proceedings of the ACM Sigmetrics 2000 International Conference on Measurement and Modeling of Computer Systems.

Digital Library

[33]

Hua Chu, Y., Rao, S. G., Seshan, S., and Zhang, H. 2001. Enabling Conferencing Applications on the Internet using an overlay multicast architecture. In Proceedings of ACM SIGCOMM.

Digital Library

[34]

InformationWeek. 2004. Go online to http://informationweek.com/story/showArticle.jhtml? articleID=50900297.

[35]

Jain, K., Lovasz, L., and Chou, P. A. 2005. Building scalable and robust peer-to-peer overlay networks for broadcasting using network coding. In Proceedings of the ACM Conference on Principles Of Distributed Computing (PODC).

Digital Library

[36]

Jain, K., Mahdian, M., and Salavatipour, M. R. 2003. Packing Steiner trees. In Proceeding of the 14th ACMSIAM Symposium on Discrete Algorithms.

Digital Library

[37]

Jain, M. and Dovrolis, C. 2002. End-to-end available bandwidth: Measurement methodology, dynamics, and relation with TCP Throughput. In Proceedings of ACM SIGCOMM.

Digital Library

[38]

Jannotti, J., Gifford, D. K., Johnson, K. L., Kaashoek, M. F., and James W. O'Toole, J. 2000. Overcast: Reliable multicasting with an overlay network. In Proceedings of the Conference on Operating Systems Design and Implementation (OSDI).

Digital Library

[39]

Katabi, D., Handley, M., and Rohrs, C. 2002. Internet congestion control for high bandwidth-delay product networks. In Proceedings of ACM SIGCOMM.

Digital Library

[40]

Kim, M. S., Lam, S. S., and Lee, D.-Y. 2002. Optimal distribution tree for Internet streaming media. Tech. rep. TR-02-48. Department of Computer Sciences, University of Texas at Austin, Austin, TX.

[41]

Kostić, D., Braud, R., Killian, C., Vandekieft, E., Anderson, J. W., Snoeren, A. C., and Vahdat, A. 2005. Maintaining high bandwidth under dynamic network conditions. In Proceedings of the USENIX 2005 Annual Technical Conference.

Digital Library

[42]

Kostić, D., Rodriguez, A., Albrecht, J., Bhirud, A., and Vahdat, A. 2003a. Using random subsets to build scalable network services. In Proceedings of the USENIX Symposium on Internet Technologies and Systems.

Digital Library

[43]

Kostić, D., Rodriguez, A., Albrecht, J., and Vahdat, A. 2003b. Bullet: High bandwidth data dissemination using an overlay mesh. In Proceedings of the 19th ACM Symposium on Operating System Principles.

Digital Library

[44]

Krohn, M. N., Freedman, M. J., and Mazieres, D. 2004. On-the-fly verification of rateless erasure codes for efficient content distribution. In Proceedings of the IEEE Symposium on Security and Privacy (Oakland, CA).

[45]

Legout, A., Urvoy-Keller, G., and Michiardi, P. 2006. Rarest first and choke algorithms are enough. In IMC'06: Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement. 203--216.

Digital Library

[46]

Luby, M. 2002. LT codes. In Proceedings of 43rd Annual IEEE Symposium on Foundations of Computer Science.

Digital Library

[47]

Luby, M. G., Mitzenmacher, M., Shokrollahi, M. A., Spielman, D. A., and Stemann, V. 1997. Practical loss-resilient codes. In Proceedings of the 29th Annual ACM Symposium on the Theory of Computing (STOC '97). Ass. Comp. Mach. Press, New York, 150-- 159.

Digital Library

[48]

Massoulie, L. and Vojnovic, M. 2005. Coupon replication systems. In Proceedings of ACM SIGMETRICS.

Digital Library

[49]

Maymounkov, P. and Mazieres, D. 2003. Rateless codes and big downloads. In Proceedings of the Second International Peer to Peer Symposium (IPTPS).

[50]

Padhye, J., Firoiu, V., Towsley, D., and Krusoe, J. 1998. Modeling TCP throughput: A simple model and its empirical validation. In Proceedings of ACM SIGCOMM (Vancouver, Canada). 303--314.

Digital Library

[51]

Padmanabhan, V. N., Qiu, L., and Wang, H. J. 2003. Server-based inference of Internet link lossiness. In Proceedings of the IEEE INFOCOM. (San Francisco, CA).

[52]

Padmanabhan, V. N., Wang, H. J., and Chou, P. A. 2003b. Resilient peer-to-peer streaming. In Proceedings of the 11th ICNP (Atlanta, GA).

Digital Library

[53]

Padmanabhan, V. N., Wang, H. J., Chou, P. A., and Sripanidkulchai, K. 2002. Distributing streaming media content using cooperative networking. In Proceedings of ACM/IEEE NOSSDAV.

Digital Library

[54]

Pai, V., Kumar, K., Tamilmani, K., Sambamurthy, V., and Mohr, A. E. 2005. Chainsaw: Eliminating trees from overlay multicast. In Proceedings of I PT PS.

Digital Library

[55]

Park, K. and Pai, V. S. 2006. Scale and performance in the CoBlitz large-file distribution service. In Proceedings of the 3rd USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI).

Digital Library

[56]

Peterson, L., Anderson, T., Culler, D., and Roscoe, T. 2002. A blueprint for introducing disruptive technology into the Internet. In Proceedings of ACM HotNets-I.

[57]

Piatek, M., Isdal, T., Anderson, T., Krishnamurthy, A., and Venkataramani, A. 2007. Do incentives build robustness in BitTorrent&quest; In Proceedings of the 4th USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI).

Digital Library

[58]

Prim, R. C. 1957. Shortest connection networks and some generalizations. In Bell Syst. Tech. J. 36, 6, 1389--1401.

[59]

Qiu, D. and Srikant, R. 2004. Modeling and performance analysis of Bittorrentlike peer-to-peer networks. In Proceedings of ACM SIGCOMM.

Digital Library

[60]

Rodriguez, A., Killian, C., Bhat, S., Kostić, D., and Vahdat, A. 2004a. MACEDON: Methodology for automatically creating, evaluating, and designing overlay networks. In Proceedings of the USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI, San Francisco, CA).

Digital Library

[61]

Rodriguez, A., Kostić, D., and Vahdat, A. 2004b. Scalability in adaptive multi-metric overlays. In Proceedings the 24th International Conference on Distributed Computing Systems (ICDCS).

Digital Library

[62]

Rowstron, A., Kermarrec, A.-M., Castro, M., and Druschel, P. 2001. SCRIBE: The design of a large-scale event notification infrastructure. In Proceedings of the Third International Workshop on Networked Group Communication.

Digital Library

[63]

Saroiu, S., Gummadi, K. P., Dunn, R. J., Gribble, S. D., and Levy, H. M. 2002. An analysis of Internet content delivery systems. In Proceedings of 5th Symposium on Operating Systems Design and Implementation (OSDI).

Digital Library

[64]

Savage, S., Collins, A., Hoffman, E., Snell, J., and Anderson, T. 1999. The end-to-end effects of Internet path selection. In Proceedings of ACM SIGCOMM.

Digital Library

[65]

Sherwood, R., Braud, R., and Bhattacharjee, B. 2004. Slurpie: A cooperative bulk data transfer protocol. In Proceedings of IEEE INFOCOM.

[66]

Shokrollahi, A. 2003. Raptor codes. Tech. rep. DF2003-06-001. Digital Fountain, Inc.

[67]

Snoeren, A. C., Conley, K., and Gifford, D. K. 2001. Mesh-based content routing using XML. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP).

Digital Library

[68]

Sung, Y.-W., Bishop, M., and Rao, S. 2006. Enabling contribution awareness in an overlay broadcasting system. In Proceedings of ACM SIGCOMM.

Digital Library

[69]

Vahdat, A., Yocum, K., Walsh, K., Mahadevan, P., Kostić, D., Chase, J., and Becker, D. 2002. Scalability and accuracy in a large-scale network emulator. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI).

Digital Library

[70]

Wang, L., Park, K., Pang, R., Pai, V. S., and Peterson, L. 2004. Reliability and Security in the CoDeeN content distribution network. In Proceedings of the USENIX 2004 Annual Technical Conference.

Digital Library

[71]

Young, A., Chen, J., Ma, Z., Krishnamurthy, A., Peterson, L., and Wang, R. Y. 2004. Overlay mesh construction using interleaved spanning trees. In Proceedings of IEEE INFOCOM.

[72]

Zhang, X., Liuy, J., Liz, B., and Yum, T.-S. P. 2005. CoolStreaming/DONet: A data-driven overlay network for efficient live media streaming. In Proceedings of IEEE INFOCOM.

Cited By

Wu CHo J(2022)Time-Critical Data Dissemination Under Flash Crowd TrafficIEEE Open Journal of the Computer Society10.1109/OJCS.2022.31494113(11-22)Online publication date: 2022
https://doi.org/10.1109/OJCS.2022.3149411
Tien YLin WHo JHon W(2019)Optimal File Dissemination Scheduling Under a General Binary Tree of Trust Relationship2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE)10.1109/TrustCom/BigDataSE.2019.00016(42-49)Online publication date: Aug-2019
https://doi.org/10.1109/TrustCom/BigDataSE.2019.00016
Mitzenmacher MPagh R(2018)Simple multi-party set reconciliationDistributed Computing10.1007/s00446-017-0316-031:6(441-453)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1007/s00446-017-0316-0
Show More Cited By

Index Terms

High-bandwidth data dissemination for large-scale distributed systems

Recommendations

Bullet: high bandwidth data dissemination using an overlay mesh
SOSP '03: Proceedings of the nineteenth ACM symposium on Operating systems principles

In recent years, overlay networks have become an effective alternative to IP multicast for efficient point to multipoint communication across the Internet. Typically, nodes self-organize with the goal of forming an efficient overlay tree, one that meets ...
Bullet: high bandwidth data dissemination using an overlay mesh
SOSP '03

In recent years, overlay networks have become an effective alternative to IP multicast for efficient point to multipoint communication across the Internet. Typically, nodes self-organize with the goal of forming an efficient overlay tree, one that meets ...
High bandwidth data dissemination for large-scale distributed systems

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems

ACM Transactions on Computer Systems Volume 26, Issue 1

February 2008

153 pages

ISSN:0734-2071

EISSN:1557-7333

DOI:10.1145/1328671

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 March 2008

Accepted: 01 November 2007

Revised: 01 June 2007

Received: 01 March 2006

Published in TOCS Volume 26, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
1,441
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu CHo J(2022)Time-Critical Data Dissemination Under Flash Crowd TrafficIEEE Open Journal of the Computer Society10.1109/OJCS.2022.31494113(11-22)Online publication date: 2022
https://doi.org/10.1109/OJCS.2022.3149411
Tien YLin WHo JHon W(2019)Optimal File Dissemination Scheduling Under a General Binary Tree of Trust Relationship2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE)10.1109/TrustCom/BigDataSE.2019.00016(42-49)Online publication date: Aug-2019
https://doi.org/10.1109/TrustCom/BigDataSE.2019.00016
Mitzenmacher MPagh R(2018)Simple multi-party set reconciliationDistributed Computing10.1007/s00446-017-0316-031:6(441-453)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1007/s00446-017-0316-0
Navarra APalazzo PPinotti CMostarda L(2016)Algorithms for Services with Multiple Levels of Quality2016 30th International Conference on Advanced Information Networking and Applications Workshops (WAINA)10.1109/WAINA.2016.20(306-311)Online publication date: Mar-2016
https://doi.org/10.1109/WAINA.2016.20
Lee HSeibert JFistrovic DKillian CNita-Rotaru C(2015)GatlingACM Transactions on Information and System Security10.1145/271456517:4(1-34)Online publication date: 24-Apr-2015
https://dl.acm.org/doi/10.1145/2714565
Hisamatsu TAsaeda H(2012)Adaptive Overlay Network for High-Bandwidth StreamingJournal of Information Processing10.2197/ipsjjip.20.15420:1(154-166)Online publication date: 2012
https://doi.org/10.2197/ipsjjip.20.154
Chen MWang XQi HShankar M(2012)Adaptive response time control for metadata matching in information dissemination systemsJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2012.08.00358:10(412-425)Online publication date: 1-Nov-2012
https://dl.acm.org/doi/10.1016/j.sysarc.2012.08.003
Skjegstad MMaseng TChaintreau AKowalski D(2011)Low complexity set reconciliation using Bloom filtersProceedings of the 7th ACM ACM SIGACT/SIGMOBILE International Workshop on Foundations of Mobile Computing10.1145/1998476.1998483(33-41)Online publication date: 9-Jun-2011
https://dl.acm.org/doi/10.1145/1998476.1998483
Chen MWang XTaylor B(2011)Achieving Bounded Matching Delay and Maximized Throughput in Information Dissemination ManagementIEEE Transactions on Network and Service Management10.1109/TNSM.2011.012111.000048:1(26-38)Online publication date: Mar-2011
https://doi.org/10.1109/TNSM.2011.012111.00004
Zage DKillian CNita-Rotaru C(2011)Removing the blinders: Using information to mitigate adversaries in adaptive overlays2011 5th International Conference on Network and System Security10.1109/ICNSS.2011.6059969(129-136)Online publication date: Oct-2011
https://doi.org/10.1109/ICNSS.2011.6059969
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents