This thesis presents online versions of the popular bagging and boosting algorithms. We demonstrate theoretically and experimentally that the online versions perform comparably to their original batch counterparts in terms of classification performance. However, our online algorithms yield the typical practical benefits of online learning algorithms when the amount of training data available is large.
Ensemble learning algorithms have become extremely popular over the last several years because these algorithms, which generate multiple base models using traditional machine learning algorithms and combine them into an ensemble model, have often demonstrated significantly better performance than single models. Bagging and boosting are two of the most popular algorithms because of their good empirical results and theoretical support. However, most ensemble algorithms operate in batch mode, i.e., they repeatedly read and process the entire training set. Typically, they require at least one pass through the training set for every base model to be included in the ensemble. The base model learning algorithms themselves may require several passes through the training set to create each base model. In situations where data is being generated continuously, storing data for batch learning is impractical, which makes using these ensemble learning algorithms impossible. These algorithms are also impractical in situations where the training set is large enough that reading and processing it many times would be prohibitively expensive.
This thesis describes online versions of bagging and boosting. Unlike the batch versions, our online versions require only one pass through the training examples in order regardless of the number of base models to be combined. We discuss how we derive the online algorithms from their batch counterparts as well as theoretical and experimental evidence that our online algorithms perform comparably to the batch versions in terms of classification performance. We also demonstrate that our online algorithms have the practical advantage of lower running time, especially for larger datasets. This makes our online algorithms practical for machine learning and data mining tasks where the amount of training data available is very large.
Cited By
- Fall E, Chang K and Chen L (2024). Tree-managed network ensembles for video prediction, Machine Vision and Applications, 35:4, Online publication date: 1-Jul-2024.
- Gohar U, Biswas S and Rajan H Towards Understanding Fairness and its Composition in Ensemble Machine Learning Proceedings of the 45th International Conference on Software Engineering, (1533-1545)
- Li N, Ma L, Zhang T and He M Multi-objective Evolutionary Ensemble Learning for Disease Classification Advances in Swarm Intelligence, (491-500)
- Liu F, Li H, Yang X and Jiang L L3E-HD: A Framework Enabling Efficient Ensemble in High-Dimensional Space for Language Tasks Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, (1844-1848)
- Kafiyan-Safari M and Rouhani M (2022). Adaptive one-pass passive-aggressive radial basis function for classification problems, Neurocomputing, 491:C, (91-103), Online publication date: 28-Jun-2022.
- Bonab H and Can F (2018). GOOWE, ACM Transactions on Knowledge Discovery from Data, 12:2, (1-33), Online publication date: 30-Apr-2018.
- Sharma V and Mahapatra K (2017). MIL based visual object tracking with kernel and scale adaptation, Image Communication, 53:C, (51-64), Online publication date: 1-Apr-2017.
- Aldoan D and Yaslan Y (2017). A comparison study on active learning integrated ensemble approaches in sentiment analysis, Computers and Electrical Engineering, 57:C, (311-323), Online publication date: 1-Jan-2017.
- Ahmad A and Brown G (2015). Random Ordinality Ensembles, Information Sciences: an International Journal, 296:C, (75-94), Online publication date: 1-Mar-2015.
- Gama J, Žliobaitė I, Bifet A, Pechenizkiy M and Bouchachia A (2014). A survey on concept drift adaptation, ACM Computing Surveys, 46:4, (1-37), Online publication date: 1-Apr-2014.
- Binh N Online multiple tasks one-shot learning of object categories and vision Proceedings of the 9th International Conference on Advances in Mobile Computing and Multimedia, (131-138)
- Gama J and Cornuéjols A Resource aware distributed knowledge discovery Ubiquitous knowledge discovery, (40-60)
- Gama J and Cornuéjols A Resource aware distributed knowledge discovery Ubiquitous knowledge discovery, (40-60)
- Chen Y and Chen Y Combining incremental Hidden Markov Model and Adaboost algorithm for anomaly intrusion detection Proceedings of the ACM SIGKDD Workshop on CyberSecurity and Intelligence Informatics, (3-9)
- Masip D, Lapedriza À and Vitrià J (2009). Boosted online learning for face recognition, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 39:2, (530-538), Online publication date: 1-Apr-2009.
- Muhlbaier M and Polikar R An ensemble approach for incremental learning in nonstationary environments Proceedings of the 7th international conference on Multiple classifier systems, (490-500)
- Yilmaz A, Javed O and Shah M (2006). Object tracking, ACM Computing Surveys (CSUR), 38:4, (13-es), Online publication date: 25-Dec-2006.
- Harrington E Online ranking/collaborative filtering using the perceptron algorithm Proceedings of the Twentieth International Conference on International Conference on Machine Learning, (250-257)
- Oza N Boosting with averaged weight vectors Proceedings of the 4th international conference on Multiple classifier systems, (15-24)
Recommendations
Online Ensemble Learning: An Empirical Study
We study resource-limited online learning, motivated by the problem of conditional-branch outcome prediction in computer architecture. In particular, we consider (parallel) time and space-efficient ensemble learners for online settings, empirically ...
A Novel Online Stacked Ensemble for Multi-Label Stream Classification
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge ManagementAs data streams become more prevalent, the necessity for online algorithms that mine this transient and dynamic data becomes clearer. Multi-label data stream classification is a supervised learning problem where each instance in the data stream is ...
Ensemble Extreme Learning Machine for Multi-instance Learning
ICMLC '17: Proceedings of the 9th International Conference on Machine Learning and ComputingMulti-instance learning (MIL) is a classification approach for classifying on a collection of instances which each group is represented as a bag. The main task of MIL is to learn from labels and features of instances to produce a model to predict a ...