research-article

Bayesian Attribute Bagging-Based Extreme Learning Machine for High-Dimensional Classification and Regression

Authors:

Joshua Zhexue Huang,

Philippe Fournier-VigerAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 13, Issue 2

Article No.: 30, Pages 1 - 26

https://doi.org/10.1145/3495164

Published: 07 March 2022 Publication History

Abstract

This article presents a Bayesian attribute bagging-based extreme learning machine (BAB-ELM) to handle high-dimensional classification and regression problems. First, the decision-making degree (DMD) of a condition attribute is calculated based on the Bayesian decision theory, i.e., the conditional probability of the condition attribute given the decision attribute. Second, the condition attribute with the highest DMD is put into the condition attribute group (CAG) corresponding to the specific decision attribute. Third, the bagging attribute groups (BAGs) are used to train an ensemble learning model of extreme learning machines (ELMs). Each base ELM is trained on a BAG which is composed of condition attributes that are randomly selected from the CAGs. Fourth, the information amount ratios of bagging condition attributes to all condition attributes is used as the weights to fuse the predictions of base ELMs in BAB-ELM. Exhaustive experiments have been conducted to compare the feasibility and effectiveness of BAB-ELM with seven other ELM models, i.e., ELM, ensemble-based ELM (EN-ELM), voting-based ELM (V-ELM), ensemble ELM (E-ELM), ensemble ELM based on multi-activation functions (MAF-EELM), bagging ELM, and simple ensemble ELM. Experimental results show that BAB-ELM is convergent with the increase of base ELMs and also can yield higher classification accuracy and lower regression error for high-dimensional classification and regression problems.

References

[1]

Leo Breiman. (2001). Random forests. Machine Learning 45, 1 (2001), 5–32.

Digital Library

[2]

Robert Bryll, Ricardo Gutierrez-Osuna, and Francis Quek. (2003). Attribute bagging: Improving accuracy of classifier ensembles by using random feature subsets. Pattern Recognition 36, 6 (2003), 1291–1302. DOI:

[3]

Jiuwen Cao, Zhiping Lin, Guang-Bin Huang, and Nan Liu. (2012). Voting based extreme learning machine. Information Sciences 185, 1 (2012), 66–77. DOI:

Digital Library

[4]

Girish Chandrashekar and Ferat Sahin. (2014). A survey on feature selection methods. Computers & Electrical Engineering 40, 1 (2014), 16–28. DOI:

Digital Library

[5]

Jianguo Chen, Kenli Li, Zhuo Tang, Kashif Bilal, Shui Yu, Chuliang Weng, and Keqin Li. (2016). A parallel random forest algorithm for big data in a spark cloud computing environment. IEEE Transactions on Parallel and Distributed Systems 28, 4 (2016), 919–933. DOI:/

[6]

C. L. Philip Chen and Zhulin Liu. (2017). Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Transactions on Neural Networks and Learning Systems 29, 1 (2017), 10–24. DOI:

[7]

Daniel Conn, Tuck Ngun, Gang Li, and Christina M. Ramirez. (2019). Fuzzy forests: Extending random forest feature selection for correlated, high-dimensional data. Journal of Statistical Software 91, 1 (2019), 1–25. DOI:

[8]

Stevica Cvetković, Miloš B. Stojanović, and Saša V. Nikolić. (2015). Multi-channel descriptors and ensemble of Extreme Learning Machines for classification of remote sensing images. Signal Processing: Image Communication 39 (2015), 111–120. DOI:

[9]

Janez Demšar. (2006). Statistical comparisons of classifiers over multiple datasets. Journal of Machine Learning Research 7 (Jan. 2006), 1–30. DOI:

[10]

Shifei Ding, Hong Zhu, Weikuan Jia, and Chunyang Su. (2012). A survey on feature extraction for pattern recognition. Artificial Intelligence Review 37, 3 (2012), 169–180. DOI:

[11]

Dheeru Dua and Casey Graff. (2019). UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA.

[12]

Alberto N. >Escalante-B. and Laurenz Wiskott. (2013). How to solve classification and regression problems on high-dimensional data with a supervised extension of slow feature analysis. The Journal of Machine Learning Research 14, 1 (2013), 3683–3719. DOI:

[13]

Benyamin Ghojogh, Maria N. Samad, Sayema Asif Mashhadi, Tania Kapoor, Wahab Ali, Fakhri Karray, and Mark Crowley. (2019). Feature selection and feature extraction in pattern analysis: A literature review. arXiv preprint arXiv:1905.02845 (2019).

[14]

Debopriya Ghosh and Javier Cabrera. (2021). Enriched random forest for high dimensional genomic data. IEEE/ACM Transactions on Computational Biology and Bioinformatics, (2021). DOI:/

[15]

Jisoo Ham, Yangchi Chen, Melba M. Crawford, and Joydeep Ghosh. (2005). Investigation of the random forest framework for classification of hyperspectral data. IEEE Transactions on Geoscience and Remote Sensing 43, 3 (2005), 492–501. DOI:/

[16]

Yanlin He, Zhiqiang Geng, Yuan Xu, and Qunxiong Zhu. (2014). A hierarchical structure of extreme learning machine (HELM) for high-dimensional datasets with noise. Neurocomputing 128 (2014), 407–414. DOI:

[17]

Yulin He, James N. K. Liu, Xizhao Wang, and Yanxing Hu. (2012). Optimal bandwidth selection for re-substitution entropy estimation. Applied Mathematics and Computation 219, 8 (2012), 3425–3460. DOI:

[18]

Zena M. Hira and Duncan F. Gillies. (2015). A review of feature selection and feature extraction methods applied on microarray data. Advances in Bioinformatics, 2015 (2015), Article ID 198363. DOI:

[19]

Tin Kam Ho. (1998). The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 8 (1998), 832–844. DOI:/

[20]

Tin Kam Ho. (1998). Nearest neighbors in random subspaces. In Lecture Notes in Computer Science vol. 1451 (1998), 640-648. DOI:

[21]

Guangbin Huang, Lei Chen, and Chee Kheong Siew. (2006). Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Transactions on Neural Networks 17, 4 (2006), 879–892. DOI:

Digital Library

[22]

Guangbin Huang, Qinyu Zhu, and Cheekheong Siew. (2006). Extreme learning machine: theory and applications. Neurocomputing 70, 1–3 (2006), 489–501. DOI:

[23]

Guangbin Huang, Hongming Zhou, Xiaojian Ding, and Rui Zhang. (2011). Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 42, 2 (2011), 513–529. DOI:/

[24]

Alexandros Iosifidis, Anastastios Tefas, and Ioannis Pitas. (2015). On the kernel extreme learning machine classifier. Pattern Recognition Letters 54, (2015), 11–17. DOI:

[25]

Liyanaarachchi Lekamalage Chamara Kasun, Yan Yang, Guang-Bin Huang, and Zhengyou Zhang. (2016). Dimension reduction with extreme learning machine. IEEE Transactions on Image Processing 25, 8 (2016), 3906–3918. DOI:

[26]

Leijun Li, Qinghua Hu, Xiangqian Wu, and Daren Yu. (2014). Exploration of classification confidence in ensemble learning. Pattern Recognition 47, 9 (2014), 3120–3131. DOI:

[27]

Changhe Li, Shengxiang Yang, and Trung Thanh Nguyen. (2011). A self-learning particle swarm optimizer for global optimization problems. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42, 3 (2011), 627–646. DOI:/

[28]

Xiaoye Li and Hongyu Zhao. (2009). Weighted random subspace method for high dimensional data classification. Statistics and its Interface 2, 2 (2009), 153–159. DOI:/

[29]

Nan Liu and Han Wang. (2010). Ensemble based extreme learning machine. IEEE Signal Processing Letters 17, 8 (2010), 754–757. DOI:

[30]

Peng Liu, Yihua Huang, Lei Meng, Siyuan Gong, and Guopeng Zhang. (2016). Two-stage extreme learning machine for high-dimensional data. International Journal of Machine Learning and Cybernetics 7, 5 (2016), 765–772. DOI:

[31]

Oyebayo Ridwan Olaniran and Mohd Asrul Affendi Bin Abdullah. (2017). BayesRandomForest: An R implementation of Bayesian Random Forest for regression analysis of high-dimensional data. In Proceedings of the 3rd International Conference on Computing, Mathematics and Statistics (ICMS2017). 269-275. DOI:

[32]

Aytuǧ Onan. (2018). An ensemble scheme based on language function analysis and feature engineering for text genre classification. Journal of Information Science 44, 1 (2018), 28–47. DOI:

Digital Library

[33]

Aytuǧ Onan. (2018). Sentiment analysis on Twitter based on ensemble of psychological and linguistic feature sets. Balkan Journal of Electrical and Computer Engineering 6, 2 (2018), 69–77. DOI:

[34]

Aytuǧ Onan, Serdar Korukoǧlu, and Hasan Bulut. (2016). A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification. Expert Systems with Applications 62, (2016), 1–16. DOI:

Digital Library

[35]

Aytuǧ Onan, Serdar Korukoǧlu, and Hasan Bulut. (2016). Ensemble of keyword extraction methods and classifiers in text classification. Expert Systems with Applications 57, (2016), 232–247. DOI:

[36]

Medhat A. Rakha. (2004). On the Moore-Penrose generalized inverse matrix. Applied Mathematics and Computation 158, 1 (2004), 185–200. DOI:

[37]

Emanuel Parzen. (1962). On estimation of a probability density function and mode. The Annals of Mathematical Statistics 33, 3 (1962), 1065–1076. DOI:

[38]

Maria Irmina Prasetiyowati, Nur Ulfa Maulidevi, and Kridanto Surendro. (2020). Feature selection to increase the random forest method performance on high dimensional data. International Journal of Advances in Intelligent Informatics 6, 3 (2020): 303–312. DOI:

[39]

Salman Salloum, Joshua Zhexue Huang, and Yulin He. (2019). Random sample partition: A distributed data model for big data analysis. IEEE Transactions on Industrial Informatics 15, 11 (2019), 5846–5854. DOI:/

[40]

Marina Skurichina. (2002). Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis and Applications 5, 2 (2002), 121–135. DOI:

[41]

Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar. (2015). A survey of modern questions and challenges in feature extraction. JMLR: Workshop and Conference Proceedings 44 (2015), 1–18.

[42]

E. Ke Tang, Ponnuthurai N. Suganthan, and Xin Yao. (2006). An analysis of diversity measures. Machine Learning 65, 1 (2006), 247–271. DOI:https://link.springer.com/content/pdf/10.1007/s10994-006-9449-2.pdf

Digital Library

[43]

Jiexiong Tang, Chenwei Deng, and Guang-Bin Huang. (2015). Extreme learning machine for multilayer perceptron. IEEE Transactions on Neural Networks and Learning Systems 27, 4 (2015), 809–821. DOI:/

[44]

Dacheng Tao, Xiaoou Tang, Xuelong Li, and Xindong Wu. (2006). Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 7 (2006), 1088–1099. DOI:/

[45]

Xizhao Wang and Yulin He. (2016). Learning from uncertainty for big data: future analytical challenges and strategies. IEEE Systems, Man, and Cybernetics Magazine 2, 2 (2016), 26–31. DOI:

[46]

Hong Wang and Lifeng Zhou. (2018). SurvELM: An R package for high dimensional survival analysis with extreme learning machine. Knowledge-Based Systems 160 (2018), 28–33. DOI:

[47]

Hong Wang and Gang Li. (2019). Extreme learning machine Cox model for high-dimensional survival analysis. Statistics in Medicine 38, 12 (2019), 2139–2156. DOI:

[48]

Bing Xue, Mengjie Zhang, Will N. Browne, and Xin Yao. (2015). A survey on evolutionary computation approaches to feature selection. IEEE Transactions on Evolutionary Computation 20, 4 (2015), 606–626. DOI:

Digital Library

[49]

Nan Zhang and Shifei Ding. (2017). Unsupervised and semi-supervised extreme learning machine with wavelet kernel for high dimensional data. Memetic Computing 9, 2 (2017), 129–139. DOI:

[50]

Xiaohan Zhang, Qunxiong Zhu, Yanlin He, and Yuan Xu. (2018). A novel robust ensemble model integrated extreme learning machine with multi-activation functions for energy modeling and analysis: Application to petrochemical industry. Energy 162 (2018), 593–602. DOI:

[51]

Xiuwen Zheng, Judong Shen, Charles Cox, Jonathan C. Wakefield, Margaret G. Ehm, Matthew R. Nelson, and Bruce S. Weir, 2014. HIBAG—HLA genotype imputation with attribute bagging. The Pharmacogenomics Journal 14, 2 (2014), 192–200. DOI:

[52]

Guoqiang Zhong, Lina Wang, Xiao Ling, and Junyu Dong, 2016. An overview on data representation learning: From traditional feature learning to recent deep learning. The Journal of Finance and Data Science 2, 4 (2016), 265–278. DOI:

[53]

Zhihua Zhou, 2012. Ensemble Methods: Foundations and Algorithms. CRC Press, Taylor & Francis Group, Cambridge, UK.

[54]

Qinyu Zhu, A. Kai Qin, Ponnuthurai N. Suganthan, and Guangbin Huang. (2005). Evolutionary extreme learning machine. Pattern Recognition 38, 10 (2005), 1759–1763. DOI:.

Cited By

Mo ZXiang HDi X(2024)Cross- and Context-Aware Attention Based Spatial-Temporal Graph Convolutional Networks for Human Mobility PredictionACM Transactions on Spatial Algorithms and Systems10.1145/3673227Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3673227
Zhou ZZhang JGuan ZHu MLao NMu LLi SMai GHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented GenerationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657673(2749-2754)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657673
Altamimi SAbu Al-Haija Q(2024)Maximizing intrusion detection efficiency for IoT networks using extreme learning machineDiscover Internet of Things10.1007/s43926-024-00060-x4:1Online publication date: 9-Jul-2024
https://doi.org/10.1007/s43926-024-00060-x
Show More Cited By

Index Terms

Bayesian Attribute Bagging-Based Extreme Learning Machine for High-Dimensional Classification and Regression
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Attribute Bagging-Based Extreme Learning Machine
Algorithms and Architectures for Parallel Processing
Abstract
Extreme learning machine (ELM) is a fast training scheme of single-hidden-layer feedforward neural network. How to further improve the prediction stability and accuracy of ELM in an ensemble learning way becomes one of the hot research topics in ...
Ensemble of kernel extreme learning machine based elimination optimization for multi-label classification
Abstract
Multi-label learning is a class of machine learning algorithms that study the classification problem of data associated with multiple labels simultaneously. Ensemble-based method is one of the representative methods in multi-label learning. In ...
A novel ensemble-based wrapper method for feature selection using extreme learning machine and genetic algorithm

This paper presents a novel wrapper feature selection algorithm for classification problems, namely hybrid genetic algorithm (GA)- and extreme learning machine (ELM)-based feature selection algorithm (HGEFS). It utilizes GA to wrap ELM to search for the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 13, Issue 2

April 2022

392 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3508464

Editor:
Huan Liu
Arizona State University, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2022

Accepted: 01 October 2021

Revised: 01 October 2021

Received: 01 July 2021

Published in TIST Volume 13, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Natural Science Foundation of China
Basic Research Foundation of Shenzhen
Scientific Research Foundation of Shenzhen University for Newly-introduced Teachers

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
321
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)3

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mo ZXiang HDi X(2024)Cross- and Context-Aware Attention Based Spatial-Temporal Graph Convolutional Networks for Human Mobility PredictionACM Transactions on Spatial Algorithms and Systems10.1145/3673227Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3673227
Zhou ZZhang JGuan ZHu MLao NMu LLi SMai GHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented GenerationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657673(2749-2754)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657673
Altamimi SAbu Al-Haija Q(2024)Maximizing intrusion detection efficiency for IoT networks using extreme learning machineDiscover Internet of Things10.1007/s43926-024-00060-x4:1Online publication date: 9-Jul-2024
https://doi.org/10.1007/s43926-024-00060-x
Xu ZXiao THe WWang YJiang ZDamiani MRenz MEldawy AKröger PNascimento M(2023)Spatial Knowledge-Infused Hierarchical Learning: An Application in Flood Mapping on Earth ImageryProceedings of the 31st ACM International Conference on Advances in Geographic Information Systems10.1145/3589132.3625591(1-10)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3589132.3625591
Hu WMao SLiu MDong MZhang YLiu T(2023)Two Twin Extreme Learning Machines for Regression and Their Applications in IndustryIECON 2023- 49th Annual Conference of the IEEE Industrial Electronics Society10.1109/IECON51785.2023.10312257(1-6)Online publication date: 16-Oct-2023
https://doi.org/10.1109/IECON51785.2023.10312257
Ma ZWang XHao Y(2023)Development and application of a hybrid forecasting framework based on improved extreme learning machine for enterprise financing riskExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.119373215:COnline publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.119373

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents