research-article

Experimental Design Networks: A Paradigm for Serving Heterogeneous Learners under Networking Constraints

Authors:

Stratis IoannidisAuthors Info & Claims

IEEE INFOCOM 2022 - IEEE Conference on Computer Communications

Pages 210 - 219

https://doi.org/10.1109/INFOCOM48880.2022.9796907

Published: 02 May 2022 Publication History

Abstract

Significant advances in edge computing capabilities enable learning to occur at geographically diverse locations. In general, the training data needed in those learning tasks are not only heterogeneous but also not fully generated locally. In this paper, we propose an experimental design network paradigm, wherein learner nodes train possibly different Bayesian linear regression models via consuming data streams generated by data source nodes over a network. We formulate this problem as a social welfare optimization problem in which the global objective is defined as the sum of experimental design objectives of individual learners, and the decision variables are the data transmission strategies subject to network constraints. We first show that, assuming Poisson data streams, the global objective is a continuous DR-submodular function. We then propose a Frank-Wolfe type algorithm that outputs a solution within a 1 – 1/e factor from the optimal. Our algorithm contains a novel gradient estimation component which is carefully designed based on Poisson tail bounds and sampling. Finally, we complement our theoretical findings through extensive experiments. Our numerical evaluation shows that the proposed algorithm outperforms several baseline algorithms both in maximizing the global objective and in the quality of the trained models.

References

[1]

N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, “Mobile edge computing: A survey,” IEEE Internet of Things Journal, vol. 5, no. 1, pp. 450–465, 2017.

[2]

B. Yang, X. Cao, X. Li, Q. Zhang, and L. Qian, “Mobile-edge-computing-based hierarchical machine learning tasks distribution for iiot,” IEEE Internet of Things Journal, vol. 7, no. 3, pp. 2169–2180, 2019.

[3]

V. Albino, U. Berardi, and R. M. Dangelico, “Smart cities: Definitions, dimensions, performance, and initiatives,” Journal of urban technology, vol. 22, no. 1, pp. 3–21, 2015.

[4]

M. Mohammadi and A. Al-Fuqaha, “Enabling cognitive smart cities using big data and machine learning: Approaches and challenges,” IEEE Communications Magazine, vol. 56, no. 2, pp. 94–101, 2018.

Digital Library

[5]

S. Boyd, S. P. Boyd, and L. Vandenberghe, Convex Optimization. Cambridge university press, 2004.

[6]

Y. Deshpande and A. Montanari, “Linear bandits in high dimension and recommendation systems,” in 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2012, pp. 1750–1754.

[7]

B. Settles, Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.

[8]

N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, “Data lifecycle challenges in production machine learning: a survey,” ACM SIGMOD Record, vol. 47, no. 2, pp. 17–28, 2018.

Digital Library

[9]

T. Horel, S. Ioannidis, and S. Muthukrishnan, “Budget feasible mechanisms for experimental design,” in Latin American Symposium on Theoretical Informatics. Springer, 2014, pp. 719–730.

[10]

Y. Guo, J. Dy, D. Erdogmus, J. Kalpathy-Cramer, S. Ostmo, J. P. Campbell, M. F. Chiang, and S. Ioannidis, “Accelerated experimental design for pairwise comparisons,” in Proceedings of the 2019 SIAM International Conference on Data Mining. SIAM, 2019, pp. 432–440.

[11]

N. Gast, S. Ioannidis, P. Loiseau, and B. Roussillon, “Linear regression from strategic data sources,” ACM Transactions on Economics and Computation (TEAC), vol. 8, no. 2, pp. 1–24, 2020.

Digital Library

[12]

Y. Guo, P. Tian, J. Kalpathy-Cramer, S. Ostmo, J. P. Campbell, M. F. Chiang, D. Erdogmus, J. G. Dy, and S. Ioannidis, “Experimental design under the bradley-terry model.” in IJCAI, 2018, pp. 2198–2204.

[13]

P. Flaherty, A. Arkin, and M. I. Jordan, “Robust design of biological experiments,” in Advances in Neural Information Processing Systems, 2006, pp. 363–370.

[14]

A. A. Bian, B. Mirzasoleiman, J. Buhmann, and A. Krause, “Guaranteed non-convex optimization: Submodular maximization over continuous domains,” in Artificial Intelligence and Statistics. PMLR, 2017, pp. 111–120.

[15]

L. Tong, Y. Li, and W. Gao, “A hierarchical edge cloud architecture for mobile computing,” in IEEE INFOCOM 2016-IEEE International Conference on Computer Communications, 2016, pp. 1–9.

[16]

K. Poularakis, J. Llorca, A. M. Tulino, I. Taylor, and L. Tassiulas, “Joint service placement and request routing in multi-cell mobile edge computing networks,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications, 2019, pp. 10–18.

[17]

K. Kamran, E. Yeh, and Q. Ma, “Deco: Joint computation, caching and forwarding in data-centric computing networks,” in Proceedings of the Twentieth ACM International Symposium on Mobile Ad Hoc Networking and Computing, 2019, pp. 111–120.

[18]

A. Lalitha, O. C. Kilinc, T. Javidi, and F. Koushanfar, “Peer-to-peer federated learning on graphs,” arXiv preprint arXiv:1901.11173, 2019.

[19]

G. Neglia, G. Calbi, D. Towsley, and G. Vardoyan, “The role of network topology for distributed machine learning,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications, 2019, pp. 2350–2358.

[20]

S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, “When edge meets learning: Adaptive control for resource-constrained distributed machine learning,” in IEEE INFOCOM 2018-IEEE Conference on Computer Communications, 2018, pp. 63–71.

[21]

K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Basar, “Fully decentralized multi-agent reinforcement learning with networked agents,” in International Conference on Machine Learning. PMLR, 2018, pp. 5872–5881.

[22]

S. Wang, Y. Ruan, Y. Tu, S. Wagle, C. G. Brinton, and C. Joe-Wong, “Network-aware optimization of distributed learning for fog computing,” IEEE/ACM Transactions on Networking, 2021.

Digital Library

[23]

F. Pukelsheim, Optimal design of experiments. Society for Industrial and Applied Mathematics, 2006.

[24]

X. Huan and Y. M. Marzouk, “Simulation-based optimal bayesian experimental design for nonlinear systems,” Journal of Computational Physics, vol. 232, no. 1, pp. 288–317, 2013.

Digital Library

[25]

G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher, “An analysis of approximations for maximizing submodular set functions—i,” Mathematical Programming, vol. 14, no. 1, pp. 265–294, 1978.

Digital Library

[26]

G. Calinescu, C. Chekuri, M. Pal, and J. Vondrák, “Maximizing a monotone submodular function subject to a matroid constraint,” SIAM Journal on Computing, vol. 40, no. 6, pp. 1740–1766, 2011.

Digital Library

[27]

T. Soma and Y. Yoshida, “A generalization of submodular cover via the diminishing return property on the integer lattice,” Advances in Neural Information Processing Systems, vol. 28, pp. 847–855, 2015.

[28]

H. Hassani, M. Soltanolkotabi, and A. Karbasi, “Gradient methods for submodular maximization,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 5843–5853.

[29]

A. A. Ageev and M. I. Sviridenko, “Pipage rounding: A new method of constructing algorithms with proven performance guarantee,” Journal of Combinatorial Optimization, vol. 8, no. 3, pp. 307–328, 2004.

[30]

C. Chekuri, J. Vondrák, and R. Zenklusen, “Dependent randomized rounding via exchange properties of combinatorial structures,” in 2010 IEEE 51st Annual Symposium on Foundations of Computer Science. IEEE, 2010, pp. 575–584.

[31]

T. Soma and Y. Yoshida, “Maximizing monotone submodular functions over the integer lattice,” Mathematical Programming, vol. 172, no. 1, pp. 539–563, 2018.

Digital Library

[32]

S. Ioannidis and E. Yeh, “Adaptive caching networks with optimality guarantees,” IEEE/ACM Transactions on Networking, vol. 26, no. 2, pp. 737–750, 2018.

Digital Library

[33]

K. Poularakis and L. Tassiulas, “On the complexity of optimal content placement in hierarchical caching networks,” IEEE Transactions on Communications, vol. 64, no. 5, pp. 2092–2103, 2016.

[34]

S. Ioannidis and E. Yeh, “Jointly optimal routing and caching for arbitrary network topologies,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 6, pp. 1258–1275, 2018.

Digital Library

[35]

K. Kamran, A. Moharrer, S. Ioannidis, and E. Yeh, “Rate allocation and content placement in cache networks,” in IEEE INFOCOM 2021–IEEE Conference on Computer Communications, 2021.

[36]

T. Wu, P. Yang, H. Dai, W. Xu, and M. Xu, “Charging oriented sensor placement and flexible scheduling in rechargeable wsns,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications, 2019, pp. 73–81.

[37]

G. Sallam and B. Ji, “Joint placement and allocation of virtual network functions with budget and capacity constraints,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications, 2019, pp. 523–531.

[38]

Z. Zheng and N. B. Shroff, “Submodular utility maximization for deadline constrained data collection in sensor networks,” IEEE Transactions on Automatic Control, vol. 59, no. 9, pp. 2400–2412, 2014.

[39]

D. Yang, G. Xue, X. Fang, and J. Tang, “Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing,” in Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, 2012, pp. 173–184.

[40]

A. Krause and C. Guestrin, “Beyond convexity: Submodularity in machine learning,” ICML Tutorials, 2008.

[41]

G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning. Springer, 2013, vol. 112.

[42]

R. G. Gallager, Stochastic Processes: Theory for Applications. Cambridge University Press, 2013.

[43]

Y. Liu, Y. Li, L. Su, E. Yeh, and S. Ioannidis, “Experimental design networks: A paradigm for serving heterogeneous learners under networking constraints,” arXiv preprint arXiv:2201.04830, 2022.

[44]

F. P. Kelly, Reversibility and stochastic networks. Cambridge University Press, 2011.

Digital Library

[45]

C. L. Canonne, “A short note on poisson tail bounds,” http://www.cs.columbia.edu/~ccanonne/files/misc/2017-poissonconcentration.pdf.

[46]

N. Alon and J. H. Spencer, The probabilistic method. John Wiley & Sons, 2004.

[47]

J. Friedman, T. Hastie, and R. Tibshirani, The elements of statistical learning. Springer series in statisticsNew York, 2001, vol. 1, no. 10.

[48]

J. Kleinberg, “The small-world phenomenon: An algorithmic perspective,” in Proceedings of the thirty-second Annual ACM Symposium on Theory of Computing, 2000, pp. 163–170.

[49]

D. Rossi and G. Rossini, “Caching performance of content centric networks under multi-path routing (and more),” Relatório técnico, Telecom ParisTech, pp. 1–6, 2011.

[50]

R. Srikant, The Mathematics of Internet Congestion Control. Birkhäuser Boston, MA: Springer Science & Business Media, 2012.

Digital Library

[51]

T. S. Jaakkola and M. I. Jordan, “A variational approach to bayesian logistic regression models and their extensions,” in Sixth International Workshop on Artificial Intelligence and Statistics. PMLR, 1997, pp. 283–294.

Cited By

Li Y(2024)Distributed Experimental Design NetworksACM SIGMETRICS Performance Evaluation Review10.1145/3639830.363983751:3(13-15)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3639830.3639837
Liu YSu LJoe-Wong CIoannidis SYeh ESiew MJi BChiasserini CWu JSubramaniam S(2023)Cache-Enabled Federated Learning SystemsProceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing10.1145/3565287.3610264(1-11)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3565287.3610264

Index Terms

Experimental Design Networks: A Paradigm for Serving Heterogeneous Learners under Networking Constraints
1. Computing methodologies
  1. Machine learning
2. Mathematics of computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Experimental Design Networks: A Paradigm for Serving Heterogeneous Learners Under Networking Constraints
Significant advances in edge computing capabilities enable learning to occur at geographically diverse locations. In general, the training data needed in those learning tasks are not only heterogeneous but also not fully generated locally. In this paper, ...
Opportunistic resource utilization networks-A new paradigm for specialized ad hoc networks

We present opportunistic resource utilization networks or oppnets, a novel paradigm of specialized ad hoc networks. We believe that applications can benefit from using specialized ad hoc networks that provide a natural basis for them, the basis more ...
Ad Hoc Networking

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

IEEE INFOCOM 2022 - IEEE Conference on Computer Communications

May 2022

2237 pages

Copyright © 2022.

Publisher

IEEE Press

Publication History

Published: 02 May 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li Y(2024)Distributed Experimental Design NetworksACM SIGMETRICS Performance Evaluation Review10.1145/3639830.363983751:3(13-15)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3639830.3639837
Liu YSu LJoe-Wong CIoannidis SYeh ESiew MJi BChiasserini CWu JSubramaniam S(2023)Cache-Enabled Federated Learning SystemsProceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing10.1145/3565287.3610264(1-11)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3565287.3610264

View Options

View options

Figures

Tables

Media

View Table of Conten