research-article

Decentralized Ensemble Learning Based on Sample Exchange among Multiple Agents

Authors:

Jiamou Liu, and

Wu ChenAuthors Info & Claims

BSCI '19: Proceedings of the 2019 ACM International Symposium on Blockchain and Secure Critical Infrastructure

July 2019

Pages 57 - 66

https://doi.org/10.1145/3327960.3332383

Published: 02 July 2019 Publication History

Abstract

Ensemble learning aims to train and combine multiple base learners in the hope of improving the overall performance. Existing ensemble algorithms rely on a centralized framework where each base learner has access to the entire training dataset. We combine the technology of blockchains which is mainly used for data validation with ensemble learning and propose a decentralized framework where data are distributed among multiple base learners, who exchange their respective data to improve the collective predictive abilities. We develop two realizations of this framework, based on static and dynamic decision trees, respectively. We evaluate our methods over 20 real-world datasets and compare them against other centralized ensemble methods. Experimental results show that the proposed method obtains improved accuracy scores through sample exchange and achieves competitive performance with state-of-the-art ensemble methods whereas the base learners store only a small fraction of the samples.

References

[1]

Arthur, D., Vassilvitskii, S.: k-means+: The advantages of careful seeding. In: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. pp. 1027--1035. Society for Industrial and Applied Mathematics (2007)

Digital Library

[2]

Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., Others, O.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning textbf3(1), 1--122 (2011)

Digital Library

[3]

Breiman, L.: Random forests. Machine learning textbf45(1), 5--32 (2001)

Digital Library

[4]

Breiman, L.: Classification and regression trees. Routledge (2017)

[5]

Cai, Y., Zheng, H., Liu, J., Yan, B., Su, H., Liu, Y.: Balancing the pain and gain of hobnobbing: Utility-based network building over atributed social networks. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. pp. 193--201. International Foundation for Autonomous Agents and Multiagent Systems (2018)

Digital Library

[6]

Condorcet, M.d.: Essay on the application of analysis to the probability of majority decisions. Paris: Imprimerie Royale (1785)

[7]

Dasarathy, B.V., Sheela, B.V.: A composite classifier system design: Concepts and methodology. Proceedings of the IEEE textbf67(5), 708--713 (1979)

[8]

Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine learning textbf40(2), 139--157 (2000)

Digital Library

[9]

Domingos, P., Hulten, G.: Mining high-speed data streams. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 71--80. ACM (2000)

Digital Library

[10]

Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research textbf15(1), 3133--3181 (2014)

Digital Library

[11]

Freund, Y., Schapire, R.E., et al.: Experiments with a new boosting algorithm. In: Icml. vol. 96, pp. 148--156. Citeseer (1996)

Digital Library

[12]

Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of statistics pp. 1189--1232 (2001)

[13]

Friedman, J.H., Hall, P.: On bagging and nonlinear estimation. Journal of statistical planning and inference textbf137(3), 669--683 (2007)

[14]

Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence textbf12(10), 993--1001 (1990)

Digital Library

[15]

Hoeffding, W.: Probability inequalities for sums of bounded random variables. Journal of the American statistical association textbf58(301), 13--30 (1963)

[16]

Jahrer, M., Töscher, A., Legenstein, R.: Combining predictions for accurate recommender systems. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 693--702. ACM (2010)

Digital Library

[17]

Jain, A.K.: Data clustering: 50 years beyond k-means. Pattern recognition letters textbf31(8), 651--666 (2010)

Digital Library

[18]

Jke drzejowicz, P.: Machine learning and agents. In: O'Shea, J., Nguyen, N.T., Crockett, K., Howlett, R.J., Jain, L.C. (eds.) Agent and Multi-Agent Systems: Technologies and Applications. Springer (2011)

Digital Library

[19]

Khoussainov, B., Liu, J., Khaliq, I.: A dynamic algorithm for reachability games played on trees. In: International Symposium on Mathematical Foundations of Computer Science. pp. 477--488. Springer (2009)

Digital Library

[20]

Koren, Y.: The bellkor solution to the netflix grand prize. Netflix prize documentation textbf81, 1--10 (2009)

[21]

Lichman, M., et al.: Uci machine learning repository (2013)

[22]

Liu, J., Wei, Z.: Community detection based on graph dynamical systems with asynchronous runs. In: 2014 Second International Symposium on Computing and Networking. pp. 463--469. IEEE (2014)

Digital Library

[23]

Louppe, G.: Understanding random forests: From theory to practice. arXiv preprint arXiv:1407.7502 (2014)

[24]

Maron, O., Moore, A.W.: Hoeffding races: Accelerating model selection search for classification and function approximation. In: Advances in neural information processing systems. pp. 59--66 (1994)

Digital Library

[25]

Michael J. Rennock, Alan Cohn, J.B.: Blockchain technology and regulatory investigations. Journal of Practical Law pp. 33--44 (2018)

[26]

Moskvina, A., Liu, J.: How to build your network? a structural analysis. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. pp. 2597--2603. AAAI Press (2016)

Digital Library

[27]

Moskvina, A., Liu, J.: Integrating networks of equipotent nodes. In: International Conference on Computational Social Networks. pp. 39--50. Springer (2016)

[28]

Moskvina, A., Liu, J.: Togetherness: an algorithmic approach to network integration. In: 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). pp. 223--230. IEEE (2016)

Digital Library

[29]

Oshiro, T.M., Perez, P.S., Baranauskas, J.A.: How many trees in a random forest? In: International Workshop on Machine Learning and Data Mining in Pattern Recognition. pp. 154--168. Springer (2012)

Digital Library

[30]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. Journal of machine learning research textbf12(Oct), 2825--2830 (2011)

Digital Library

[31]

Polikar, R.: Ensemble learning. In: Ensemble machine learning, pp. 1--34. Springer (2012)

[32]

Probst, P., Boulesteix, A.L.: To tune or not to tune the number of trees in random forest? arXiv preprint arXiv:1705.05654 (2017)

Digital Library

[33]

Ratsch, G., Onoda, T., Müller, K.R.: Soft margins for adaboost. Machine learning textbf42(3), 287--320 (2001)

Digital Library

[34]

Schapire, R.E.: The strength of weak learnability. Machine learning textbf5(2), 197--227 (1990)

Digital Library

[35]

Tang, Y., Liu, J., Chen, W., Zhang, Z.: Establishing connections in a social network. In: Pacific Rim International Conference on Artificial Intelligence. pp. 1044--1057. Springer (2018)

[36]

Ueda, N., Nakano, R.: Generalization error of ensemble estimators. In: Neural Networks, 1996., IEEE International Conference on. vol. 1, pp. 90--95. IEEE (1996)

[37]

Wolpert, D.H.: Stacked generalization. Neural networks textbf5(2), 241--259 (1992)

Digital Library

[38]

Yan, B., Chen, Y., Liu, J.: Dynamic relationship building: exploitation versus exploration on a social network. In: International Conference on Web Information Systems Engineering. pp. 75--90. Springer (2017)

Digital Library

[39]

Zhou, Z.H.: Ensemble methods: foundations and algorithms. Chapman and Hall/CRC (2012)

Digital Library

Cited By

Ye GChen THung Nguyen QYin H(2024)Heterogeneous decentralised machine unlearning with seed model distillationCAAI Transactions on Intelligence Technology10.1049/cit2.122819:3(608-619)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1049/cit2.12281
Arya MG H(2022)Ensemble Federated Learning for Classifying IoMT Data Streams2022 IEEE 7th International conference for Convergence in Technology (I2CT)10.1109/I2CT54291.2022.9824145(1-5)Online publication date: 7-Apr-2022
https://doi.org/10.1109/I2CT54291.2022.9824145
Rashid MZhang DWang D(2021)HeteroSAS: A Heterogeneous Resource Management Framework for "All-in-the-Air" Social Airborne Sensing in Disaster Response2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS52077.2021.00034(132-139)Online publication date: Jul-2021
https://doi.org/10.1109/DCOSS52077.2021.00034

Index Terms

Decentralized Ensemble Learning Based on Sample Exchange among Multiple Agents
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
    2. Machine learning algorithms
      1. Ensemble methods
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Multi-agent learning

Recommendations

Building Locally Discriminative Classifier Ensemble Through Classifier Fusion Among Nearest Neighbors
PCM 2016: 17th Pacific-Rim Conference on Advances in Multimedia Information Processing - Volume 9916

Many studies on ensemble learning that combines multiple classifiers have shown that, it is an effective technique to improve accuracy and stability of a single classifier. In this paper, we propose a novel discriminative classifier fusion method, which ...
Read More
Diversity-Based Ensemble with Sample Weight Learning
ICPR '14: Proceedings of the 2014 22nd International Conference on Pattern Recognition

Given multiple classifiers, one prevalent approach in classifier ensemble is to diversely combine classifier components (diversity-based ensemble), and a lot of previous works show that this approach can improve accuracy in classification. However, how ...
Read More
Naïve Bayes ensemble learning based on oracle selection
CCDC'09: Proceedings of the 21st annual international conference on Chinese Control and Decision Conference

Aiming at the stability of Naïve Bayes algorithm and overcoming the limitation of the attributes independence assumption in the Naive Bayes learning, we present an ensemble learning algorithm for naive Bayesian classifiers based on oracle selection (...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

BSCI '19: Proceedings of the 2019 ACM International Symposium on Blockchain and Secure Critical Infrastructure

July 2019

134 pages

ISBN:9781450367868

DOI:10.1145/3327960

Program Chairs:
Keke Gai
Beijing Institute of Technology, China
,
Kim-Kwang Raymond Choo
University of Texas at San Antonio, USA
,
Debiao He
Wuhan University, China
,
Sheng Wen
Swinburne University of Technology, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Asia CCS '19

Sponsor:

SIGSAC

Asia CCS '19: ACM Asia Conference on Computer and Communications Security

July 8, 2019

Auckland, New Zealand

Acceptance Rates

BSCI '19 Paper Acceptance Rate 44 of 12 submissions, 367%;

Overall Acceptance Rate 44 of 12 submissions, 367%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
158
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

Ye GChen THung Nguyen QYin H(2024)Heterogeneous decentralised machine unlearning with seed model distillationCAAI Transactions on Intelligence Technology10.1049/cit2.122819:3(608-619)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1049/cit2.12281
Arya MG H(2022)Ensemble Federated Learning for Classifying IoMT Data Streams2022 IEEE 7th International conference for Convergence in Technology (I2CT)10.1109/I2CT54291.2022.9824145(1-5)Online publication date: 7-Apr-2022
https://doi.org/10.1109/I2CT54291.2022.9824145
Rashid MZhang DWang D(2021)HeteroSAS: A Heterogeneous Resource Management Framework for "All-in-the-Air" Social Airborne Sensing in Disaster Response2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS52077.2021.00034(132-139)Online publication date: Jul-2021
https://doi.org/10.1109/DCOSS52077.2021.00034

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents