Article

Contract-based load management in federated distributed systems

Authors:

Magdalena Balazinska,

Hari Balakrishnan,

Mike StonebrakerAuthors Info & Claims

NSDI'04: Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1

Page 15

Published: 29 March 2004 Publication History

Abstract

This paper focuses on load management in loosely-coupled federated distributed systems. We present a distributed mechanism for moving load between autonomous participants using bilateral contracts that are negotiated offline and that set bounded prices for moving load. We show that our mechanism has good incentive properties, efficiently redistributes excess load, and has a low overhead in practice.

Our load management mechanism is especially well-suited for distributed stream-processing applications, an emerging class of data-intensive applications that employ a "continuous query processing" model. In this model, streams of data are processed and composed continuously as they arrive rather than after they are indexed and stored. We have implemented the mechanism in the Medusa distributed stream processing system, and we demonstrate its properties using simulations and experiments.

References

[1]

{1} D. J. Abadi, D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik. Aurora: A new model and architecture for data stream management. The VLDB Journal: The Int. Journal on Very Large Data Bases, Sept. 2003.

Digital Library

[2]

{2} B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In Proc. of 2002 ACM Symposium on Principles of Database Systems, June 2002.

Digital Library

[3]

{3} P. Bhoj, S. Singhal, and S. Chutani. SLA management in federated environments. Technical Report HPL-98-203, Hewlett-Packard Company, 1998.

[4]

{4} R. Buuya, H. Stockinger, J. Giddy, and D. Abramson. Economic models for management of resources in peer-to-peer and grid computing. In Proc. of SPIE Int. Symposium on The Convergence of Information Technologies and Communications (ITCom 2001), Aug. 2001.

[5]

{5} S. Chandrasekaran, A. Deshpande, M. Franklin, and J. Hellerstein. TelegraphCQ: Continuous dataflow processing for an uncertain world. In Proc. of the First Biennial Conference on Innovative Data Systems Research (CIDR'03), Jan. 2003.

Digital Library

[6]

{6} J. Chen, D. J. DeWitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for Internet databases. In Proc. of the 2000 ACM SIGMOD Int. Conference on Management of Data, May 2000.

Digital Library

[7]

{7} M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Çetintemel, Y. Xing, and S. Zdonik. Scalable distributed stream processing. In Proc. of the First Biennial Conference on Innovative Data Systems Research (CIDR'03), Jan. 2003.

[8]

{8} B. Chun, Y. Fu, and A. Vahdat. Bootstrapping a distributed computational economy with peer-to-peer bartering. In Proc. of the Workshop on Economics of Peer-to-Peer Systems, June 2003.

[9]

{9} B. N. Chun. Market-Based Cluster Resource Management. PhD thesis, University of California at Berkeley, 2001.

Digital Library

[10]

{10} D. L. Eager, E. D. Lazowska, and J. Zahorjan. Adaptive load sharing in homogeneous distributed systems. IEEE Transactions on Software Engineering, SE-12(5):662-675, May 1986.

Digital Library

[11]

{11} C. Estan, S. Savage, and G. Varghese. Automatically inferring patterns of resource consumption in network traffic. In Proc. of the ACM SIGCOMM 2003 Conference, Aug. 2003.

Digital Library

[12]

{12} J. Feigenbaum, C. Papadimitriou, R. Sami, and S. Shenker. A BGP-based mechanism for lowest-cost routing. In Proc. of the 21st Symposium on Principles of Distributed Computing, July 2002.

Digital Library

[13]

{13} J. Feigenbaum, C. Papadimitriou, and S. Shenker. Sharing the cost of multicast transmissions. Journal of Computer and System Sciences, 63:21-41, 2001.

Digital Library

[14]

{14} J. Feigenbaum, C. Papadimitriou, and S. Shenker. Distributed algorithmic mechanism design: Recent results and future directions. In Proc. of the 6th Int. Workshop on Discrete Algorithms and Methods for Mobile Computing and Communications, Sept. 2002.

Digital Library

[15]

{15} D. Ferguson, C. Nikolaou, J. Sairamesh, and Y. Yemini. Economic models for allocating resources in computer systems. In S. H. Clearwater, editor, Market based Control of Distributed Systems. World Scientist, Jan. 1996.

Digital Library

[16]

{16} I. T. Foster and C. Kesselman. Computational grids. In Proc. of the Vector and Parallel Processing (VECPAR), June 2001.

Digital Library

[17]

{17} Y. Fu, J. Chase, B. Chun, S. Schwab, and A. Vahdat. SHARP: An architecture for secure resource peering. In 19th ACM Symposium on Operating Systems Principles, Oct. 2003.

Digital Library

[18]

{18} Y. Fu and A. Vahdat. Service level agreement based distributed resource allocation for streaming hosting systems. In Proc. of 7th Int. Workshop on Web Content Caching and Distribution, Aug. 2002.

[19]

{19} R. Gallager. A minimum delay routing algorithm using distributed computation. IEEE Transactions on Communication, COM-25(1), Jan. 1977.

[20]

{20} M. Jackson. Mechanism theory. Forthcoming in Encyclopedia of Life Support Stystems, 2000.

[21]

{21} A. Keller and H. Ludwig. The WSLA framework: Specifying and monitoring service level agreements for Web services. Technical Report RC22456, IBM Corporation, May 2002.

[22]

{22} J. F. Kurose. A microeconomic approach to optimal resource allocation in distributed computer systems. IEEE Transactions on Computers, 38(5):705-717, 1989.

Digital Library

[23]

{23} K. Lai, M. Feldman, I. Stoica, and J. Chuang. Incentives for co-operation in peer-to-peer networks. In Proc. of the Workshop on Economics of Peer-to-Peer Systems, June 2003.

[24]

{24} W. Lehr and L. W. McKnight. Show me the money: Contracts and agents in service level agreement markets. http://itc.mit. edu/itel/docs/2002/show_me_the_money.pdf, 2002.

[25]

{25} T. W. Malone, R. E. Fikes, K. R. Grant, and M. T. Howard. Enterprise: A market-like task scheduler for distributed computing environments. The Ecology of Computation, 1988.

[26]

{26} Mesquite Software, Inc. CSIM 18 user guide. http://www. mesquite.com.

[27]

{27} M. S. Miller and K. E. Drexler. Markets and computation: Agoric open systems. In B. Huberman, editor, The Ecology of Computation . Science & Technology, 1988.

[28]

{28} C. Ng, D. C. Parkes, and M. Seltzer. Strategyproof computing: Systems infrastructures for self-interested parties. In Proc. of the Workshop on Economics of Peer-to-Peer Systems, June 2003.

[29]

{29} C. Ng, D. C. Parkes, and M. Seltzer. Virtual worlds: Fast and strategyproof auctions for dynamic resource allocation. http://www.eecs.harvard.edu/~parkes/ pubs/virtual.pdf, June 2003.

Digital Library

[30]

{30} T.-W. J. Ngan, D. S. Wallach, and P. Druschel. Enforcing fair sharing of peer-to-peer resources. In Proc. of the 2nd Int. Workshop on Peer-to-Peer Systems (IPTPS '03), Feb. 2003.

[31]

{31} N. Nisan and A. Ronen. Computationally feasible VCG mechanisms. In Proc. of the Second ACM Conference on Electronic Commerce (EC00), Oct. 2000.

Digital Library

[32]

{32} N. Nisan and A. Ronen. Algorithmic mechanism design. Games and Economic Behavior, 35, 2001.

[33]

{33} D. Parkes. Iterative Combinatorial Auctions: Achieving Economic and Computational Efficiency (Chapter 2). PhD thesis, Department of Computer and Information Science, University of Pennsylvania, 2001.

Digital Library

[34]

{34} D. Parkes. Price-based information certificates for minimal-revelation combinatorial auctions. Agent Mediated Electronic Commerce IV, LNAI 2531:103-122, 2002.

Digital Library

[35]

{35} L. Peterson, T. Anderson, D. Culler, and T. Roscoe. A blueprint for introducing disruptive technology into the Internet. In Proc. of the First Workshop on Hot Topics in Networks, Oct. 2002.

[36]

{36} M. Roesch. Snort: Lightweight intrusion detection for networks. In Proc. of the 13th Conference on Systems Administration (LISA-99), Nov. 1999.

Digital Library

[37]

{37} A. Sahai, A. Durante, and V. Machiraju. Towards automated SLA management for Web services. Technical Report HPL-2001-310R1, Hewlett-Packard Company, July 2001.

[38]

{38} A. Sahai, S. Graupner, V. Machiraju, and A. van Moorsel. Specifying and monitoring guarantees in commercial grids through SLA. Technical Report HPL-2003-324, Hewlett-Packard Company, Nov. 2002.

[39]

{39} T. W. Sandholm. An implementation of the contract net protocol based on marginal cost calculations. In Proc. of the 12th Int. Workshop on Distributed Artificial Intelligence, pages 295-308, 1993.

[40]

{40} T. W. Sandholm. Contract types for satisficing task allocation: I theoretical results. In AAAI Spring Symposium Series: Satisficing Models, Mar. 1998.

[41]

{41} M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin. Flux: An adaptive partitioning operator for continuous query systems. In Proc. of the 19th Int. Conference on Data Engineering (ICDE 2003), Mar. 2003.

[42]

{42} I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for Internet applications. In Proc. of the ACM SIGCOMM 2001 Conference, pages 149-160, Aug. 2001.

Digital Library

[43]

{43} M. Stonebraker, P. M. Aoki, W. Litwin, A. Pfeffer, A. Sah, J. Sidell, C. Staelin, and A. Yu. Mariposa: a wide-area distributed database system. The VLDB Journal: The Int. Journal on Very Large Data Bases, 5, Jan. 1996.

Digital Library

[44]

{44} The Condor Project. Condor high throughput computing. http: //www.cs.wisc.edu/condor/.

[45]

{45} V. Vishnumurthy, S. Chandrakumar, and E. G. Sirer. KARMA: A secure economic framework for peer-to-peer resource sharing. In Proc. of the Workshop on Economics of Peer-to-Peer Systems, June 2003.

[46]

{46} C. Waldspurger, T. Hogg, B. Huberman, J. Kephart, and W. Stornetta. Spawn: A distributed computational economy. IEEE Transactions on Software Engineering, SE-18(2):103-117, 1992.

Digital Library

Cited By

Begoli EAkidau THueske FHyde JKnight KKnowles KBoncz PManegold SAilamaki ADeshpande AKraska T(2019)One SQL to Rule Them All - an Efficient and Syntactically Idiomatic Approach to Management of Streams and TablesProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3314040(1757-1772)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3314040
Dias de Assuno Mda Silva Veith ABuyya R(2018)Distributed data stream processing and edge computingJournal of Network and Computer Applications10.1016/j.jnca.2017.12.001103:C(1-17)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.jnca.2017.12.001
Kalyvianaki EFiscato MSalonidis TPietzuch PÖzcan FKoutrika GMadden S(2016)THEMISProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2882943(541-553)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2882943
Show More Cited By

Index Terms

Recommendations

A dynamic load balancing mechanism for distributed systems
Abstract
It is desirable in a distributed system to have the system load balanced evenly among the nodes so that the mean job response time is minimized. In this paper, we present a dynamic load balancing mechanism (DLB). It adopts a centralized approach ...
Resource management in large distributed systems

In this paper, we propose that a resource management system for large distributed systems should have two levels --- a lower one, responsible for export and allocation of resources in local distributed systems, and an upper one, which manages special ...
Allocation of processes and files for load balancing in distributed systems

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NSDI'04: Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1

March 2004

404 pages

Sponsors

USENIX Assoc: USENIX Assoc

Publisher

USENIX Association

United States

Publication History

Published: 29 March 2004

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
16
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Begoli EAkidau THueske FHyde JKnight KKnowles KBoncz PManegold SAilamaki ADeshpande AKraska T(2019)One SQL to Rule Them All - an Efficient and Syntactically Idiomatic Approach to Management of Streams and TablesProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3314040(1757-1772)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3314040
Dias de Assuno Mda Silva Veith ABuyya R(2018)Distributed data stream processing and edge computingJournal of Network and Computer Applications10.1016/j.jnca.2017.12.001103:C(1-17)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.jnca.2017.12.001
Kalyvianaki EFiscato MSalonidis TPietzuch PÖzcan FKoutrika GMadden S(2016)THEMISProceedings of the 2016 International Conference on Management of Data10.1145/2882903.2882943(541-553)Online publication date: 26-Jun-2016
https://dl.acm.org/doi/10.1145/2882903.2882943
Lei CRundensteiner E(2014)Robust Distributed Query Processing for Streaming DataACM Transactions on Database Systems10.1145/260213839:2(1-45)Online publication date: 26-May-2014
https://dl.acm.org/doi/10.1145/2602138
Margara AUrbani Jvan Harmelen FBal H(2014)Streaming the WebWeb Semantics: Science, Services and Agents on the World Wide Web10.1016/j.websem.2014.02.00125:C(24-44)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.1016/j.websem.2014.02.001
Zhang-Jian DLee CHwang R(2014)An energy-saving algorithm for cloud resource management using a Kalman filterInternational Journal of Communication Systems10.1002/dac.259927:12(4078-4091)Online publication date: 1-Dec-2014
https://dl.acm.org/doi/10.1002/dac.2599
Upadhyaya PBalazinska MSuciu D(2012)How to price shared optimizations in the cloudProceedings of the VLDB Endowment10.14778/2168651.21686575:6(562-573)Online publication date: 1-Feb-2012
https://dl.acm.org/doi/10.14778/2168651.2168657
Cugola GMargara A(2012)Processing flows of informationACM Computing Surveys10.1145/2187671.218767744:3(1-62)Online publication date: 14-Jun-2012
https://dl.acm.org/doi/10.1145/2187671.2187677
Gounaris AYfoulis CPaton N(2012)Efficient load balancing in partitioned queries under random perturbationsACM Transactions on Autonomous and Adaptive Systems10.1145/2168260.21682657:1(1-27)Online publication date: 4-May-2012
https://dl.acm.org/doi/10.1145/2168260.2168265
Baldoni RBonomi SLodi GQuerzoni LEsposito CGokhale ACotroneo DSchmidt D(2010)Data Dissemination supporting collaborative complex event processingProceedings of the First International Workshop on Data Dissemination for Large Scale Complex Critical Infrastructures10.1145/1862821.1862822(3-8)Online publication date: 27-Apr-2010
https://dl.acm.org/doi/10.1145/1862821.1862822
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents