Abstract
Current self-supervised representation learning methods are mainly based on contrastive learning and proxy tasks. These methods acquire semantically rich features by contrasting samples with invariant transformations (positive pairs) against other samples (negative pairs), and simply discard transformations that degrade performance when used as invariances. However, using only invariant transformations often leads to an over-reliance on invariant transformations, which affects the generalisation ability and robustness of the model, while the large number of negative sample pairs in contrast learning imposes a huge computational overhead. In order to address these issues, we reduce the dependence on invariant transformations by transforming the discarded invariant transformations into equivariant transformations. In contrast learning, we reduce the computational overhead by using only positive pairs to obtain semantically rich features. Specifically, we enhance feature semantic quality by encouraging certain transformations to exhibit non-trivial equivariance on samples of invariant transformations in the form of a proxy task, while preserving original transformation invariance. The model learns the invariant transformations further by learning equivariance at the same time, and our approach can improve the accuracy of the model without changing the structure of the original model. Experimental results show that significant improvements are obtained on several benchmark datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agrawal, P., Carreira, J., Malik, J.: Learning to see by moving. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 37–45 (2015)
Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual information across views. Adv. Neural Inf. Process. Syst. 32 (2019)
Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P.: Geometric deep learning: grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021)
Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arxiv 2020. arXiv preprint arXiv:2003.04297 (2003)
Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 215–223. JMLR Workshop and Conference Proceedings (2011)
Cohen, T., Welling, M.: Group equivariant convolutional networks. In: International Conference on Machine Learning, pp. 2990–2999. PMLR (2016)
Da, K.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Dangovski, R., et al.: Equivariant contrastive learning. arXiv preprint arXiv:2111.00899 (2021)
Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N.: Whitening for self-supervised representation learning. In: International Conference on Machine Learning, pp. 3015–3024. PMLR (2021)
Fetterman, A., Albrecht, J.: Understanding self-supervised and contrastive learning with bootstrap your own latent (BYOL). Untitled AI, August (2020)
Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., Cord, M.: Boosting few-shot visual learning with self-supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8059–8068 (2019)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018)
Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural. Inf. Process. Syst. 33, 21271–21284 (2020)
Guo, G., Wang, H., Bell, D., Bi, Y., Greer, K.: KNN model-based approach in classification. In: Meersman, R., Tari, Z., Schmidt, D.C. (eds.) OTM 2003. LNCS, vol. 2888, pp. 986–996. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-39964-3_62
Guo, H., Ba, Y., Hu, J., Si, L., Qiang, W., Shi, L.: Self-supervised representation learning with meta comprehensive regularization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 1959–1967 (2024)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
Jayaraman, D., Grauman, K.: Learning image representations tied to ego-motion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1413–1421 (2015)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), 3 (2015)
Lenc, K., Vedaldi, A.: Understanding image representations by measuring their equivariance and equivalence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 991–999 (2015)
Loh, C., Christensen, T., Dangovski, R., Kim, S., Soljačić, M.: Surrogate-and invariance-boosted contrastive learning for data-scarce applications in science. Nat. Commun. 13(1), 4223 (2022)
Metzger, S., Srinivas, A., Darrell, T., Keutzer, K.: Evaluating self-supervised pretraining without using labels. arXiv preprint arXiv:2009.07724 (2020)
Peng, X., Wang, K., Zhu, Z., Wang, M., You, Y.: Crafting better contrastive views for siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16031–16040 (2022)
Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: self-supervised learning via redundancy reduction. In: International Conference on Machine Learning, pp. 12310–12320. PMLR (2021)
Zhang, H., Berg, A.C., Maire, M., Malik, J.: SVM-KNN: discriminative nearest neighbor classification for visual category recognition. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 2126–2136. IEEE (2006)
Zhang, L.: Equivariance and invariance for robust unsupervised and semi-supervised learning (2020)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yang, L., Yang, Y., Jin, H. (2024). Self Supervised Contrastive Learning Combining Equivariance and Invariance. In: Zhang, W., Tung, A., Zheng, Z., Yang, Z., Wang, X., Guo, H. (eds) Web and Big Data. APWeb-WAIM 2024. Lecture Notes in Computer Science, vol 14965. Springer, Singapore. https://doi.org/10.1007/978-981-97-7244-5_22
Download citation
DOI: https://doi.org/10.1007/978-981-97-7244-5_22
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-7243-8
Online ISBN: 978-981-97-7244-5
eBook Packages: Computer ScienceComputer Science (R0)