Cited By
View all- Yıldız İDy JErdoğmuş DOstmo SCampbell JChiang MIoannidis S(2022)Spectral Ranking RegressionACM Transactions on Knowledge Discovery from Data10.1145/353069316:6(1-38)Online publication date: 30-Jul-2022
Stochastic Bregman Parallel Direction Method of Multipliers for Distributed Optimization
Authors: 
Yue Yu, Behçet AçıkmeşeAuthors Info & Claims
Yue Yu, Behçet AçıkmeşeAuthors Info & ClaimsPages 5550 - 5555
Abstract
Bregman parallel direction method of multipliers (BPDMM) efficiently solves distributed optimization over a network, which arises in a wide spectrum of collaborative multi-agent learning applications. In this paper, we generalize BPDMM to stochastic BPDMM, where each iteration only solves local optimization on a randomly selected subset of nodes rather than all the nodes in the network. Such generalization reduce the need for computational resources and allows applications to larger scale networks. We establish both the global convergence and the O(1/T) iteration complexity of stochastic BPDMM. We demonstrate our results via numerical examples.
References
[1]
M. Mesbahi and M. Egerstedt, Graph Theoretic Methods in Multiagent Networks. Princeton University Press, 2010.
[2]
D. Li, K. D. Wong, Y. H. Hu, and A. M. Sayeed, “Detection, classification, and tracking of targets,” IEEE Signal Process. Mag., vol. 19, no. 2, pp. 17–29, 2002.
[3]
B. Açıkmeşe, M. Mandić, and J. L. Speyer, “Decentralized observers with consensus filters for distributed discrete-time linear systems,” Automatica, vol. 50, no. 4, pp. 1037–1052, 2014.
[4]
V. Lesser, C. L. Ortiz Jr, and M. Tambe, Distributed Sensor Networks: A Multiagent Perspective. Springer Science & Business Media, 2012, vol. 9.
[5]
B. Gholami, S. Yoon, and V. Pavlovic, “Decentralized approximate bayesian inference for distributed sensor network.” in AAAI Conf. Artificial Intell., 2016, pp. 1582–1588.
[6]
A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine, “Collective robot reinforcement learning with distributed asynchronous guided policy search,” in Int. Conf. Intell. Robots Syst. IEEE, 2017, pp. 79–86.
[7]
E. Wei and A. Ozdaglar, “Distributed alternating direction method of multipliers,” in Proc. IEEE Conf. Decision Control, 2012, pp. 5445–5450.
[8]
D. Meng, M. Fazel, and M. Mesbahi, “Proximal alternating direction method of multipliers for distributed optimization on weighted graphs,” in Proc. IEEE Conf. Decision Control, 2015, pp. 1396–1401.
[9]
W. Deng, M.-J. Lai, Z. Peng, and W. Yin, “Parallel multi-block ADMM with O(1/k) convergence,” J. Sci. Comput., vol. 71, no. 2, pp. 712–736, 2017.
[10]
H. Wang and A. Banerjee, “Bregman alternating direction method of multipliers,” in Adv. Neural Inform. Process. Syst., 2014, pp. 2816–2824.
[11]
Y. Yu, B. Açıkmes¸e, and M. Mesbahi, “Bregman parallel direction method of multipliers for distributed optimization via mirror averaging,” IEEE Control Syst. Lett., vol. 2, no. 2, pp. 302–306, 2018.
[12]
E. Wei and A. Ozdaglar, “On the O(1/k) convergence of asynchronous distributed alternating direction method of multipliers,” in Proc. Global Conf. Signal Inform. Process. IEEE, 2013, pp. 551–554.
[13]
H. Wang, A. Banerjee, and Z.-Q. Luo, “Parallel direction method of multipliers,” in Adv. Neural Inform. Process. Syst., 2014, pp. 181–189.
[14]
Z. Zhu and A. J. Storkey, “Stochastic parallel block coordinate descent for large-scale saddle point problems.” in Proc. AAAI Conf. Artificial Intell., 2016, pp. 2429–2437.
[15]
Y. Nesterov, “Efficiency of coordinate descent methods on huge-scale optimization problems,” SIAM J. Optim., vol. 22, no. 2, pp. 341–362, 2012.
[16]
P. Richtárik and M. Takáč, “Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function,” Math. Prog., vol. 144, no. 1-2, pp. 1–38, 2014.
[17]
R. T. Rockafellar, Convex analysis. Princeton University Press, 2015.
[18]
S. Bubeck et al., “Convex optimization: Algorithms and complexity,” Found. Trends Mach. Learn., vol. 8, no. 3-4, pp. 231–357, 2015.
[19]
Y. Censor, S. A. Zenios, et al., Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press, 1997.
[20]
R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge University Press, 2012.
[21]
D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. Prentice Hall Englewood Cliffs, NJ, 1989, vol. 23.
[22]
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, 2011.
[23]
D. Jakovetic, J. M. Moura, and J. Xavier, “Distributed augmented lagrangian algorithms: convergence rate,” in Proc. IEEE Global Conf. Signal Inform. Process., 2013, pp. 563–566.
[24]
F. Iutzeler, P. Bianchi, P. Ciblat, and W. Hachem, “Asynchronous distributed optimization using a randomized alternating direction method of multipliers,” in Proc. IEEE Conf. Decision Control, 2013, pp. 3671–3676.
[25]
W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin, “On the linear convergence of the ADMM in decentralized consensus optimization.” IEEE Trans. Signal Process., vol. 62, no. 7, pp. 1750–1761, 2014.
[26]
J. Wang and N. Elia, “Control approach to distributed optimization,” in Proc. Allerton Conf. Commun. Control Comput. IEEE, 2010, pp. 557–561.
[27]
Y. Yu, B. Açıkmes¸e, and M. Mesbahi, “Mass-spring-damper network for distributed averaging and optimization,” arXiv preprint arXiv:1808.01999 [math. OC], 2018.
[28]
A. S. Nemirovsky and D. B. Yudin, Problem complexity and method efficiency in optimization. Wiley, 1983.
[29]
Y. Yu and B. Açıkmeşe, “Stochastic bregman parallel direction method of multipliers for distributed optimization,” arXiv preprint arXiv:1902.09695 [math.OC], 20119.
Recommendations
Comments
Information & Contributors
Information
Published In
7716 pages
Copyright © 2019.
Publisher
IEEE Press
Publication History
Published: 01 December 2019
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025
Other Metrics
Citations
Cited By
View all- Yıldız İDy JErdoğmuş DOstmo SCampbell JChiang MIoannidis S(2022)Spectral Ranking RegressionACM Transactions on Knowledge Discovery from Data10.1145/353069316:6(1-38)Online publication date: 30-Jul-2022