Abstract
For better classification, generative models are used to initialize the model and extract features before training a classifier. Typically, separate unsupervised and supervised learning problems are solved. Generative restricted Boltzmann machines and deep belief networks are widely used for unsupervised learning. We developed several supervised models based on deep belief networks in order to improve this two-phase strategy. Modifying the loss function to account for expectation with respect to the underlying generative model, introducing weight bounds, and multi-level programming are all applied in model development. The proposed models capture both unsupervised and supervised objectives effectively. The computational study verifies that our models perform better than the two-phase training approach. In addition, we conduct an ablation study to examine how a different part of our model and a different mix of training samples affect the performance of our models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Larochelle, H., Mandel, M., Pascanu, R., Bengio, Y.: Learning algorithms for the classification restricted Boltzmann machine. J. Mach. Learn. Res. 13, 643–669 (2012)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Larochelle, H., Bengio, Y.: Classification using discriminative restricted Boltzmann machines. In: International Conference on Machine Learning (ICML), Helsinki, Finland, vol. 25, pp. 536–543 (2008)
Elfwing, S., Uchibe, E., Doya, K.: Expected energy-based restricted Boltzmann machine for classification. Neural Networks 64, 29–38 (2015)
Dahl, G.E., Adams, R.P., Larochelle, H.: Training restricted Boltzmann machines on word observations. In: International Conference on Machine Learning (ICML), Edinburgh, Scotland, UK, vol. 29, pp. 679–686 (2012)
Norouzi, M., Ranjbar, M., Mori, G.: Stacks of convolutional restricted Boltzmann machines for shift-invariant feature learning. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR), Miami, FL, USA, pp. 2735–2742 (2009)
Salama, M.A., Eid, H.F., Ramadan, R.A., Darwish, A., Hassanien, A.E.: Hybrid intelligent intrusion detection scheme. Adv. Intell. Soft Comput. 96, 293–303 (2011)
Xing, E.P., Yan, R., Hauptmann, A.G.: Mining associated text and images with dual-wing Harmoniums. In: Conference on Uncertainty in Artificial Intelligence (UAI), vol. 21, Edinburgh, Scotland, pp. 633–641 (2005)
Cho, K., Ilin, A., Raiko, T.: Improved learning algorithms for restricted Boltzmann machines. In: Artificial Neural Networks and Machine Learning (ICANN), vol. 21. Springer, Heidelberg (2011)
Mccallum, A., Pal, C., Druck, G., Wang, X.: Multi-conditional learning: generative/discriminative training for clustering and classification. In: National Conference on Artificial Intelligence (AAAI), vol. 21, pp. 433–439 (2006)
Gehler, P.V., Holub, A. D., Welling, M.: The rate adapting poisson (RAP) model for information retrieval and object recognition. In: International Conference on Machine Learning (ICML), Pittsburgh, PA, USA, vol. 23, pp. 337–344 (2006)
Bengio, Y., Lamblin, P.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 20, pp. 153–160. MIT Press (2007)
Sarikaya, R., Hinton, G.E., Deoras, A.: Application of deep belief networks for natural language understanding. IEEE/ACM Trans. Audio Speech Lang. Process. 22(4), 778–784 (2014)
Lu, N., Li, T., Ren, X., Miao, H.: A deep learning scheme for motor imagery classification based on restricted Boltzmann machines. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 566–576 (2017)
Yu, L., Shi, X., Shengwei, T.: Classification of cytochrome P450 1A2 of inhibitors and noninhibitors based on deep belief network. Int. J. Comput. Intell. Appl. 16, 1750002 (2017)
Li, Y., Nie, X., Huang, R.: Web spam classification method based on deep belief networks. Expert Syst. Appl. 96(1), 261–270 (2018)
Hassan, M.M., Alam, G.R., Uddin, Z., Huda, S., Almogren, A., Fortino, G.: Human emotion recognition using deep belief network architecture. Inf. Fusion 51, 10–18 (2019)
Schmah, T., Hinton, G.E., Zemel, R.S., Small, S.L., Strother, S.: Generative versus discriminative training of RBMs for classification of fMRI images. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 21, pp. 1409–1416. Curran Associates Inc. (2009)
Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)
Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8), 1771–1800 (2002)
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–54 (2006)
Wang, B., Klabjan, D.: Regularization for unsupervised deep neural nets. In: National Conference on Artificial Intelligence (AAAI), vol. 31, pp. 2681–2687 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Koo, J., Klabjan, D. (2020). Improved Classification Based on Deep Belief Networks. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12396. Springer, Cham. https://doi.org/10.1007/978-3-030-61609-0_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-61609-0_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61608-3
Online ISBN: 978-3-030-61609-0
eBook Packages: Computer ScienceComputer Science (R0)