research-article

Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things

Authors:

Binshi XuAuthors Info & Claims

Transactions on Asian and Low-Resource Language Information Processing, Volume 20, Issue 5

Article No.: 72, Pages 1 - 15

https://doi.org/10.1145/3433539

Published: 21 July 2021 Publication History

Abstract

The Internet of Things and big data are currently hot concepts and research fields. The mining, classification, and recognition of big data in the Internet of Things system are the key links that are widely of concern at present. The artificial neural network is beneficial for multi-dimensional data classification and recognition because of its strong feature extraction and self-learning ability. Pre-training is an effective method to address the gradient diffusion problem in deep neural networks and could result in better generalization. This article focuses on the performance of supervised pre-training that uses labelled data. In particular, this pre-training procedure is a simulation that shows the changes in judgment patterns as they progress from primary to mature within the human brain. In this article, the state-of-the-art of neural network pre-training is reviewed. Then, the principles of the auto-encoder and supervised pre-training are introduced in detail. Furthermore, an extended structure of supervised pre-training is proposed. A set of experiments are carried out to compare the performances of different pre-training methods. These experiments include a comparison between the original and pre-trained networks as well as a comparison between the networks with two types of sub-network structures. In addition, a homemade database is established to analyze the influence of pre-training on the generalization ability of neural networks. Finally, an ordinary convolutional neural network is used to verify the applicability of supervised pre-training.

References

[1]

Y. W. Li and L. Ma. 2016. Big data effective information filtering mining of Internet of Things based on SVM. Control Engineering of China 23, 10 (2016), 1533–1537.

[2]

K. M. Bimal and M. A. Gholam. 2012. Differential epidemic model of virus and worms in computer network. International Journal of Network Security 14, 3 (2012), 149–155.

[3]

M. Volmer, B. G. Wolthers, and H. J. Metting et al. 2020. Artificial neural network predictions of urinary calculus compositions analyzed with infrared spectroscopy. Clinical Chemistry 40, 9 (2020), 1692–1697.

[4]

S. Zhou, J. Wang, and J. Wang et al. 2017. Point to set similarity based deep feature learning for person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Honolulu, HI. IEEE Computer Society, 3741–3750.

[5]

L. Arel, D. C. Rose, and T. P. Karnowski. 2010. Deep machine learning: a new frontier in artificial intelligence research. Computational Intelligence Magazine 5, 4 (2010), 13–18.

Digital Library

[6]

D. Erhan, Y. Bengio, A. Courville, et al. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11, 3 (2010), 625–660.

Digital Library

[7]

G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.

[8]

G. E. Hinton, S. Osinder, and Y. W. Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527–1554.

Digital Library

[9]

D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 1 (1985), 147–169.

[10]

E. H. L. Aarts and J. H. M. Korst. 1987. Boltzmann machines and their applications. In PARLE, Parallel Architectures and Languages Europe, J. W. Bakker, A. J. de Nijman, P. C. Treleaven, (Eds.). Springer, Berlin. 34–50.

Digital Library

[11]

P. Vincent, H. Larochelle, and Y. Bengio. 2008. Extracting and composing robust features with denoising auto-encoders. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland. 1096–1103.

Digital Library

[12]

P. Vincent, H. Larochelle, I. Lajoie, et al. 2010. Stacked denoising auto-encoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research 11, 12 (2010), 3371–3408.

Digital Library

[13]

R. Salah, P. Vincent, X. Muller, et al. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning Bellevue, Washington, USA, June 28-July 2. 833–840.

Digital Library

[14]

J. Masci, U. Meier, and D. Ciresan. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proceedings of the 21st International Conference on Artificial Neural Networks. Vol. 6791. Espoo, Finland, June 14-17, 2011. Springer-Verlag, 52--59.

Digital Library

[15]

T. Schlegl, J. Ofner, and G. Langs. 2014. Unsupervised pre-training across image domains improves lung tissue classification. In Medical Computer Vision (MCV’14): Algorithms for Big Data, Cambridge, MA, USA, September 18, 2014. Springer International Publishing, 82--93.

[16]

H. I. Suk, S. W. Lee, and D. Shen. 2013. Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Structure and Function 220, 2 (2023), 841–859.

[17]

P. O. Glauner. 2015. Deep convolutional neural networks for smile recognition. In Deep Learning For Smile Recognition[C]// 12th Conference on Uncertainty Modelling in Knowledge Engineering and Decision Making (FLINS'16), Roubaix, France, August 24-26, 2016. 986--989.

[18]

J. Xu, L. Xiang, Q. Liu, et al. 2016. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Transactions on Medical Imaging 35, 1 (2016), 119.

[19]

A. Santara, D. Maji, D. Tejas, et al. 2016. Faster learning of deep stacked autoencoders on multi-core systems using synchronized layer-wise pre-training. arXiv preprint arXiv:1603.02836.

[20]

Z. Zhang, Y. Song, and H. Qi. 2017. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17), Hawaii, USA July 21-26, 2017. 287--291.

[21]

B. Du, W. Xiong, and J. Wu. 2017. Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics 47, 4 (2017), 1017–1027.

[22]

Y. Bi, P. Wang, X. Guo, et al. 2019. K-means clustering optimizing deep stacked sparse autoencoder. Sensing and Imaging: An International Journal 20, 1 (2019), 6.

[23]

L. Deng and D. Yu. 2013. Deep learning: Methods and applications. Foundations and Trends in Signal Processing 7, 3.

[24]

Y. Sun, H. Mao, Q. Guo, et al. 2016. Learning a good representation with unsymmetrical auto-encoder. Neural Computing and Applications 27, 5 (2016), 1361–1367.

Digital Library

[25]

B. Schölkopf, J. Platt, and T. Hofmann. 2007. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19 (2007), 153--160.

Digital Library

[26]

A. Achille and S. Soatto. 2018. Emergence of invariance and disentanglement in deep representations. In IEEE Information Theory and Applications Workshop (ITA’18), San Diego, CA, February 11–16, 2018. arXiv: 1706.01350v2 [cs.LG] 16, 1–9.

[27]

P. Koprinkova-Hristova, V. Mladenov, and K. N. Kasabov. 2015. How to Pretrain Deep Boltzmann Machines in Two Stages[J]. Springer International Publishing. 10.1007/978-3-319-09903-3 (Chapter 10), 201--219.

[28]

M. Eastwood and C. Jayne. 2013. Restricted Boltzmann machines for pre-training deep Gaussian networks. In Proceedings of International Joint Conference on Neural Networks, Dallas, USA, August 4-9, 2013. 1501--1507.

[29]

G. Desjardins, A. Courville, and Y. Bengio. 2012. On training deep Boltzmann machines. arXiv: 1203.4416v1 [cs.NE] 20, 1–7.

[30]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, 318--362.

Digital Library

Cited By

Liu X(2023)Trends in Informatization of Electronic Music Composition Data in the Context of Distance EducationICST Transactions on Scalable Information Systems10.4108/eetsis.3826Online publication date: 5-Sep-2023
https://doi.org/10.4108/eetsis.3826
Chen STian Y(2023)Construction of network mathematics model for college English mixed teachingApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.1.004478:2(2665-2676)Online publication date: 22-Jun-2023
https://doi.org/10.2478/amns.2023.1.00447
Yao S(2022)Design of Brand Business Model Based on Big Data and Internet of Things Technology ApplicationComputational Intelligence and Neuroscience10.1155/2022/91898052022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/9189805
Show More Cited By

Index Terms

Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networks

In this paper, through extension of the present methods and based on error minimization, two fast and efficient layer-by-layer pre-training methods are proposed for initializing deep neural network (DNN) weights. Due to confrontation with a large number ...
On Generalization of Graph Autoencoders with Adversarial Training
Machine Learning and Knowledge Discovery in Databases. Research Track
Abstract
Adversarial training is an approach for increasing model’s resilience against adversarial perturbations. Such approaches have been demonstrated to result in models with feature representations that generalize better. However, limited works have ... $_{}$ $_{}$ $_{}$ $_{}$
A canonical polyadic deep convolutional computation model for big data feature learning in Internet of Things
Abstract
In recent years, the Internet of Things is more widely deployed with increasing amounts of data gathered. These data are of high volume, velocity, veracity and variety, posing a vast challenge on the data analysis, especially with ...
Highlights
- A canonical polyadic DCCM is introduced to improve the feature learning efficiency.

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 20, Issue 5

September 2021

320 pages

ISSN:2375-4699

EISSN:2375-4702

DOI:10.1145/3467024

Editor:
Imed Zitouni
Google, USA

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2021

Accepted: 01 November 2020

Revised: 01 October 2020

Received: 01 April 2020

Published in TALLIP Volume 20, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Key Research and Development Program of China
Beijing Science and Technology Special Project
Beijing Science and Technology Planning Project Support

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
104
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)4

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu X(2023)Trends in Informatization of Electronic Music Composition Data in the Context of Distance EducationICST Transactions on Scalable Information Systems10.4108/eetsis.3826Online publication date: 5-Sep-2023
https://doi.org/10.4108/eetsis.3826
Chen STian Y(2023)Construction of network mathematics model for college English mixed teachingApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.1.004478:2(2665-2676)Online publication date: 22-Jun-2023
https://doi.org/10.2478/amns.2023.1.00447
Yao S(2022)Design of Brand Business Model Based on Big Data and Internet of Things Technology ApplicationComputational Intelligence and Neuroscience10.1155/2022/91898052022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/9189805
He LZhang XLi ZXiao PWei ZCheng XQu S(2022)A Chinese Named Entity Recognition Model of Maintenance Records for Power Primary Equipment Based on Progressive Multitype Feature FusionComplexity10.1155/2022/81142172022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/8114217
Luo C(2022)Waveform Feature Extraction of Intelligent Singing Skills under the Background of Internet of ThingsMobile Information Systems10.1155/2022/46388012022Online publication date: 28-Jun-2022
https://dl.acm.org/doi/10.1155/2022/4638801
Chen YChen Q(2022)Gymnastics Action Recognition and Training Posture Analysis Based on Artificial Intelligence SensorJournal of Sensors10.1155/2022/16055292022(1-7)Online publication date: 1-Jul-2022
https://doi.org/10.1155/2022/1605529
Zhang TYin RQu ZChen Z(2022)Phase Detection Methods for Alzheimer's disease Based on Improved Convolutional Neural Network Fine-tuning2022 E-Health and Bioengineering Conference (EHB)10.1109/EHB55594.2022.9991518(1-4)Online publication date: 17-Nov-2022
https://doi.org/10.1109/EHB55594.2022.9991518
Tavakoli SGhaffari AKouzehkanan ZHosseini R(2021)New segmentation and feature extraction algorithm for classification of white blood cells in peripheral smear imagesScientific Reports10.1038/s41598-021-98599-011:1Online publication date: 30-Sep-2021
https://doi.org/10.1038/s41598-021-98599-0

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents