Abstract
One of the goals of machine learning is learning a sequence of tasks more naturally. Continual learning imitates the real learning mode of human beings and continually learns new knowledge without forgetting old knowledge at the same time. In the past decades, considerable attention has been paid to this learning method. However, avoiding forgetting old knowledge in the process of learning new knowledge remains an ongoing challenge due to catastrophic forgetting. Therefore, it is desirable to exploit a new method to improve the stability in a continual learning scenario. In this paper, we specifically focus on the shared feature extraction between two consecutive tasks. To this end, we explore a continual learning paradigm by using the task-wise shared hidden representation alignment module, which contrasts shared representations from the current task and shared representations from reconstruction pseudo samples of previous tasks. Our proposed TSHRA model grasps similarity features provided by the alignment module to learn shared representations more consummately when the model is learning the current task. To verify our proposed model, we conduct experiments on Split-MNIST and Fashion-MNIST. The experimental results show that our proposed TSHRA’s performance is outstanding which justify that the alignment module has a positive effect on learning shared representations among different tasks for the continual learning scenario.
J. Liu—This work was supported by the Science Foundation of China
University of Petroleum, Beijing (No. 2462020YXZZ023).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ring, M.: Continual learning in reinforcement environment, Ph. D. thesis, University of Texas, Austin (1994)
French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)
Shim, D., et al.: Online class-incremental continual learning with adversarial shapley value. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35. no. 11 (2021)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
Shin, H., et al.: Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 (2017)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)
Fernando, C.: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 1–16 (2017)
Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: International Conference on Machine Learning, PMLR (2017)
Van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 (2019)
Bromley, J., et al.: Signature verification using a “siamese” time delay neural network. Int. J. Pattern Recognit. Artif. Intell. 7(4), 669–688 (1993)
Ye, M., et al.: Unsupervised embedding learning via invariant and spreading instance feature. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
Grill, J.-B., et al.: Bootstrap your own latent: a new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)
Chen, X., Kaiming, H.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
Chen, T., et al.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, PMLR (2020)
He, K., et al.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
Wu, Z., et al.: Clear: contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 (2020)
Wu, J., et al.: Self-supervised graph learning for recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)
Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing systems 27 (2014)
Kingma, D.P., et al.: Semi-supervised learning with deep generative models. In: Advances in Neural Information Processing Systems (2014)
Rebuffi, S.-A., et al.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Sokar, G., Mocanu, D.C., Pechenizkiy, M.: Learning invariant representation for continual learning (2021)
LeCun, Y.: The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/ (1998)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. Adv. Neural. Inf. Process. Syst. 30, 6467–6476 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhan, Xh., Liu, Jw., Han, Yn. (2022). Continual Learning by Task-Wise Shared Hidden Representation Alignment. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13530. Springer, Cham. https://doi.org/10.1007/978-3-031-15931-2_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-15931-2_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15930-5
Online ISBN: 978-3-031-15931-2
eBook Packages: Computer ScienceComputer Science (R0)