Generalized implicit follow-the-regularized-leader
Article No.: 190, Pages 4826 - 4838
Abstract
We propose a new class of online learning algorithms, generalized implicit Follow-The-Regularized-Leader (FTRL), that expands the scope of FTRL framework. Generalized implicit FTRL can recover known algorithms, such as FTRL with linearized losses and implicit FTRL, and it allows the design of new update rules, as extensions of aProx and Mirror-Prox to FTRL. Our theory is constructive in the sense that it provides a simple unifying framework to design updates that directly improve the worst-case upper bound on the regret. The key idea is substituting the linearization of the losses with a Fenchel-Young inequality. We show the flexibility of the framework by proving that some known algorithms, like the Mirror-Prox updates, are instantiations of the generalized implicit FTRL. Finally, the new framework allows us to recover the temporal variation bound of implicit OMD, with the same computational complexity.
References
[1]
Abernethy, J. D., Hazan, E., and Rakhlin, A. Competing in the dark: An efficient algorithm for bandit linear optimization. In Servedio, R. A. and Zhang, T. (eds.), Proc. of Conference on Learning Theory (COLT), pp. 263-274. Omnipress, 2008.
[2]
Asi, H. and Duchi, J. C. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, 29(3):2257-2290, 2019.
[3]
Bauschke, H. H. and Combettes, P. L. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011.
[4]
Campolongo, N. and Orabona, F. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
[5]
Cesa-Bianchi, N. and Lugosi, G. Prediction, learning, and games. Cambridge University Press, 2006.
[6]
Cesa-Bianchi, N. and Orabona, F. Online learning algorithms. Annual Review of Statistics and Its Application, 8:165-190, 2021.
[7]
Chang, C.-C. and Lin, C.-J. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):1-27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
[8]
Chen, K., Cutkosky, A., and Orabona, F. Implicit parameter-free online learning with truncated linear models. In International Conference on Algorithmic Learning Theory, pp. 148-175. PMLR, 2022a.
[9]
Chen, K., Langford, J., and Orabona, F. Better parameter-free stochastic optimization with ODE updates for coinbetting. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
[10]
Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7:551-585, 2006.
[11]
Duchi, J. C., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
[12]
Fang, H., Harvey, N., Portella, V., and Friedlander, M. Online mirror descent and dual averaging: keeping pace in the dynamic case. In Daum, III, H. and Singh, A. (eds.), Proc. of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3008-3017. PMLR, 13-18 Jul 2020.
[13]
Hazan, E. and Kale, S. Extracting certainty from uncertainty: Regret bounded by variation in costs. In Proc. of the 21st Conference on Learning Theory, 2008.
[14]
Joulani, P., György, A., and Szepesvári, C. A modular analysis of adaptive (non-)convex optimization: Optimism, composite objectives, and variational bounds. In Proc. of the International Conference on Algorithmic Learning Theory (ALT), volume 76, pp. 681-720, 2017.
[15]
Kivinen, J. and Warmuth, M. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1):1-63, January 1997.
[16]
Kulis, B. and Bartlett, P. L. Implicit online learning. In International Conference on Machine Learning, pp. 575- 582, 2010.
[17]
Littlestone, N. Learning quickly when irrelevant attributes abound: a new linear-threshold algorithm. Machine Learning, 2(4):285-318, 1988.
[18]
Martinet, B. Régularisation dinéquations variationnelles par approximations successives. rev. française informat. Recherche Opérationnelle, 4:154-158, 1970.
[19]
McMahan, H. B. A unified view of regularized dual averaging and mirror descent with implicit updates. arXiv preprint arXiv:1009.3240, 2010.
[20]
McMahan, H. B. A survey of algorithms and analysis for adaptive online learning. The Journal of Machine Learning Research, 18(1):3117-3166, 2017.
[21]
McMahan, H. B. and Streeter, M. J. Adaptive bound optimization for online convex optimization. In COLT, 2010.
[22]
Moreau, J.-J. Proximité et dualité dans un espace hilbertien. Bulletin de la Société mathématique de France, 93:273- 299, 1965.
[23]
Nemirovski, A. Prox-method with rate of convergence O(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1):229-251, 2004.
[24]
Nemirovskij, A. S. and Yudin, D. Problem complexity and method efficiency in optimization. Wiley, New York, NY, USA, 1983.
[25]
Orabona, F. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019. Version 6.
[26]
Orabona, F. and Pál, D. Scale-free algorithms for online linear optimization. In International Conference on Algorithmic Learning Theory, pp. 287-301. Springer, 2015.
[27]
Orabona, F. and Pál, D. Coin betting and parameter-free online learning. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 577-585. Curran Associates, Inc., 2016.
[28]
Orabona, F. and Pál, D. Scale-free online learning. Theoretical Computer Science, 716:50-69, 2018. Special Issue on ALT 2015.
[29]
Orabona, F. and Pál, D. Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236, 2021.
[30]
Parikh, N. and Boyd, S. Proximal algorithms. Foundations and Trends in optimization, 1(3):127-239, 2014.
[31]
Rockafellar, R. T. Convex Analysis. Princeton University Press, 1970.
[32]
Rockafellar, R. T. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.
[33]
Shalev-Shwartz, S. Online Learning: Theory, Algorithms, and Applications. PhD thesis, The Hebrew University, 2007.
[34]
Shalev-Shwartz, S. and Singer, Y. A primal-dual perspective of online learning algorithms. Machine Learning, 69:115- 142, 2007a.
[35]
Shalev-Shwartz, S. and Singer, Y. Convex repeated games and Fenchel duality. In Advances in neural information processing systems, pp. 1265-1272, 2007b.
[36]
Shtoff, A. Efficient implementation of incremental proximal-point methods. arXiv preprint arXiv:2205.01457, 2022.
[37]
Streeter, M. and McMahan, H. B. Less regret via online conditioning. arXiv preprintarXiv:1002.4862, 2010.
[38]
Warmuth, M. K. and Jagota, A. K. Continuous and discretetime nonlinear gradient descent: Relative loss bounds and convergence. In Electronic proceedings of the 5th International Symposium on Artificial Intelligence and Mathematics, volume 326, 1997.
Recommendations
Stability-penalty-adaptive follow-the-regularized-leader: sparsity, game-dependency, and best-of-both-worlds
NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing SystemsAdaptivity to the difficulties of a problem is a key property in sequential decision-making problems to broaden the applicability of algorithms. Follow-the-regularized-leader (FTRL) has recently emerged as one of the most promising approaches for ...
Follow the perturbed leader: optimism and fast parallel algorithms for smooth minimax games
NIPS '20: Proceedings of the 34th International Conference on Neural Information Processing SystemsWe consider the problem of online learning and its application to solving minimax games. For the online learning problem, Follow the Perturbed Leader (FTPL) is a widely studied algorithm which enjoys the optimal O (T1/2) worst case regret guarantee for ...
Follow the compressed leader: faster online learning of eigenvectors and faster MMWU
ICML'17: Proceedings of the 34th International Conference on Machine Learning - Volume 70The online problem of computing the top eigenvector is fundamental to machine learning. The famous matrix-multiplicative-weight-update (MMWU) framework solves this online problem and gives optimal regret. However, since MMWU runs very slow due to the ...
Comments
Information & Contributors
Information
Published In
July 2023
43479 pages
Copyright © 2023.
Publisher
JMLR.org
Publication History
Published: 23 July 2023
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Jan 2025