Abstract
A life-long classifier that learns incrementally has many challenges such as concept drift, when the class changes in time, and catastrophic forgetting when the earlier learned knowledge is lost. Many successful connectionist solutions are based on an idea that new data are learned only in a part of a network that is relevant to the new data. We leverage this idea and propose a novel method for learning an ensemble of specialized autoencoders. We interpret autoencoders as manifolds that can be trained to contain or exclude given points from the input space. This manifold manipulation allows us to implement a classifier that can suppress catastrophic forgetting and adapt to concept drift. The proposed algorithm is evaluated on an incremental version of the XOR problem and on an incremental version of the MNIST classification where we achieved 0.9 accuracy which is a significant improvement over the previously published results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The centers of \(0.1\times 0.1\) large squares \(A_1, A_2, B_1\), and B2 are (0.3,0.7), (0.7,0.3), (0.3,0.3), and (0.7,0.7), respectively. \(A_3\) is \(0.02\times 0.02\) large centered at (0.3,0.3).
- 2.
It is apparent from Sect. 5.2, as sampling a point that lies in \(\theta \)-wrap is roughly equivalent to getting an actual image of digit by randomly sampling \(28\times 28\) pixels.
References
Gepperth A, Hammer B (2016) Incremental learning algorithms and applications. In: European symposium on artificial neural networks (ESANN), pp 357–368
Hinton GE, McClelland JL, Rumelhart DE (1986) Distributed representations. In: Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pp 77–109. MIT Press, Cambridge
Kirkpatrick J et al (2017) Overcoming catastrophic forgetting in neural networks. Proc Nat Acad Sci 114(13):3521–3526
Li Z, Hoiem D (2016) Learning without forgetting, CoRR, vol. abs/1606.09282
Krawczyk B, Minku LL, Gama J, Stefanowski J, Woźniak M (2017) Ensemble learning for data stream analysis: a survey. Inf Fusion 37:132–156
Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15:1–15:58
Borghesi A, Bartolini A, Lombardi M, Milano M, Benini L (2018) Anomaly detection using autoencoders in high performance computing systems, CoRR, vol. abs/1811.05269
Marchi E, Vesperini F, Squartini S, Schuller B (2017) Deep recurrent neuralnetwork-based autoencoders for acoustic novelty detection. Comput Intell Neurosci 2017
Mustafa AM, Ayoade G, Al-Naami K, Khan L, Hamlen KW, Thuraisingham B, Araujo F (2017) Unsupervised deep embedding for novel class detection over data stream. In: IEEE international conference on big data, pp 1830–1839
Chen J, Sathe S, Aggarwal C, Turaga D (2017) Outlier detection with autoencoder ensembles. In: SIAM international conference on data mining, pp 90–98
Mirsky Y, Doitshman T, Elovici Y, Shabtai A (2018) Kitsune: an ensemble of autoencoders for online network intrusion detection, CoRR, vol. abs/1802.09089
Triki AR, Aljundi R, Blaschko MB, Tuytelaars T (2017) Encoder based lifelong learning, CoRR, vol. abs/1704.01920
Pfülb B, Gepperth A, Abdullah S, Kilian A (2018) Catastrophic forgetting: still a problem for DNNs. In: International conference on artificial neural networks (ICANN), pp 487–497
LeCun Y, Cortes C (2010) MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. Accessed 29 Jan 2019
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization, CoRR, vol. abs/1412.6980
Acknowledgments
This work was supported by the Czech Science Foundation (GAÄŒR) under research project No. 18-18858S.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Szadkowski, R., Drchal, J., Faigl, J. (2020). Autoencoders Covering Space as a Life-Long Classifier. In: Vellido, A., Gibert, K., Angulo, C., MartÃn Guerrero, J. (eds) Advances in Self-Organizing Maps, Learning Vector Quantization, Clustering and Data Visualization. WSOM 2019. Advances in Intelligent Systems and Computing, vol 976. Springer, Cham. https://doi.org/10.1007/978-3-030-19642-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-19642-4_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-19641-7
Online ISBN: 978-3-030-19642-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)