research-article

Dynamic semantic structure distillation for low-resolution fine-grained recognition

Authors:

Mingjiang Liang,

Wei LiuAuthors Info & Claims

Volume 148, Issue C

https://doi.org/10.1016/j.patcog.2023.110216

Published: 17 April 2024 Publication History

Abstract

Low-resolution images are ubiquitous in real applications such as surveillance and mobile photography. However, existing fine-grained approaches usually suffer catastrophic failures when dealing with low-resolution inputs. This is because their learning strategy inherently depends on the semantic structure of the pre-trained model, resulting in poor robustness and generalization. To mitigate this limitation, we propose a dynamic semantic structure distillation learning framework. Our method first facilitates knowledge distillation of diverse semantic structures by perturbing the composition of semantic components and then utilizes a decoupled distillation objective to prevent the loss of primary semantic part relation knowledge. We evaluate our proposed approach on two knowledge distillation tasks: high-to-low resolution and large-to-small model. The experimental results show that our proposed approach significantly outperforms existing methods in low-resolution fine-grained image classification tasks. This indicates that it can effectively distill knowledge from high-resolution teacher models to low-resolution student models. Furthermore, we demonstrate the effectiveness of our approach in general image classification and standard knowledge distillation tasks.

Highlights

•

Introducing the Dynamic Semantic Structure Distillation (DSSD) framework for enhanced fine-grained image classification in low-resolution images.

•

Proposing dynamic semantic structure learning for perceiving semantic relationships, and decoupled knowledge distillation for efficient semantic information transfer.

•

Extensive experimental validation shows DSSD’s superiority over current state-of-the-art methods in two scenarios.

References

[1]

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He, Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1492–1500.

[2]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[3]

Zhang Lu, Shen Jialie, Zhang Jian, Xu Jingsong, Li Zhibin, Yao Yazhou, Yu Litao, Multimodal marketing intent analysis for effective targeted advertising, IEEE Trans. Multimed. 24 (2021) 1830–1843.

[4]

Shen Jialie, Robertson Neil, BBAS: Towards large scale effective ensemble adversarial attacks against deep neural network learning, Inform. Sci. 569 (2021) 469–478.

[5]

Zhuang Chen, Huang Shaoli, Cheng Gong, Ning Jifeng, Multi-criteria selection of rehearsal samples for continual learning, Pattern Recognit. 132 (2022).

[6]

Zhang Xiaopeng, Xiong Hongkai, Zhou Wengang, Lin Weiyao, Tian Qi, Picking deep filter responses for fine-grained image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1134–1142.

[7]

Jianlong Fu, Heliang Zheng, Tao Mei, Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4438–4446.

[8]

Zheng Heliang, Fu Jianlong, Mei Tao, Luo Jiebo, Learning multi-attention convolutional neural network for fine-grained image recognition, in: ICCV, 2017, pp. 5209–5217.

[9]

Weifeng Ge, Xiangru Lin, Yizhou Yu, Weakly supervised complementary parts models for fine-grained image classification from the bottom up, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3034–3043.

[10]

Lianbo Zhang, Shaoli Huang, Wei Liu, Dacheng Tao, Learning a mixture of granularity-specific experts for fine-grained categorization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 8331–8340.

[11]

Flores Carola Figueroa, Gonzalez-Garcia Abel, van de Weijer Joost, Raducanu Bogdan, Saliency for fine-grained object recognition in domains with scarce training data, Pattern Recognit. 94 (2019) 62–73.

[12]

Zhang Lianbo, Huang Shaoli, Liu Wei, Intra-class part swapping for fine-grained image classification, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 3209–3218.

[13]

Wang Qi, Wang JianJun, Deng Hongyu, Wu Xue, Wang Yazhou, Hao Gefei, Aa-trans: Core attention aggregating transformer with information entropy selector for fine-grained visual classification, Pattern Recognit. 140 (2023).

[14]

Liang Mingjiang, Huang Shaoli, Pan Shirui, Gong Mingming, Liu Wei, Learning multi-level weight-centric features for few-shot learning, Pattern Recognit. 128 (2022).

[15]

Zhang Lianbo, Huang Shaoli, Liu Wei, Learning sequentially diversified representations for fine-grained categorization, Pattern Recognit. 121 (2022).

[16]

Zheng Heliang, Fu Jianlong, Zha Zheng-Jun, Luo Jiebo, Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5012–5021.

[17]

Yao Ding, Yanzhao Zhou, Yi Zhu, Qixiang Ye, Jianbin Jiao, Selective sparse sampling for fine-grained image recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6599–6608.

[18]

Hinton Geoffrey, Vinyals Oriol, Dean Jeff, Distilling the knowledge in a neural network, 2015, arXiv preprint arXiv:1503.02531.

[19]

Feng Zhang, Xiatian Zhu, Mao Ye, Fast human pose estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3517–3526.

[20]

Meng Zhong, Li Jinyu, Zhao Yong, Gong Yifan, Conditional teacher-student learning, in: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2019, pp. 6445–6449.

[21]

Jang Hyun Cho, Bharath Hariharan, On the efficacy of knowledge distillation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4794–4802.

[22]

Lake Brenden M., Salakhutdinov Ruslan, Tenenbaum Joshua B., Human-level concept learning through probabilistic program induction, Science 350 (6266) (2015) 1332–1338.

[23]

Ruichi Yu, Ang Li, Vlad I. Morariu, Larry S. Davis, Visual relationship detection with internal and external linguistic knowledge distillation, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1974–1982.

[24]

Yoshua Bengio, Jerome Louradour, Ronan Collobert, Jason Weston, Curriculum learning, in: Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 41–48.

[25]

Jia Kui, Gong Shaogang, Multi-modal tensor face for simultaneous super-resolution and recognition, in: IEEE International Conference on Computer Vision, vol. 2, IEEE, 2005, pp. 1683–1690.

[26]

Hennings-Yeomans Pablo H., Baker Simon, Kumar B.V.K. Vijaya, Simultaneous super-resolution and feature extraction for recognition of low-resolution faces, in: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8.

[27]

Zou Wilman W.W., Yuen Pong C., Very low resolution face recognition problem, IEEE Trans. Image Process. 21 (1) (2011) 327–340.

Digital Library

[28]

Baker Simon, Kanade Takeo, Hallucinating faces, in: IEEE International Conference on Automatic Face and Gesture Recognition, IEEE, 2000, pp. 83–88.

[29]

Liu Ce, Shum Heung-Yeung, Zhang Chang-Shui, A two-step approach to hallucinating faces: Global parametric model and local nonparametric model, in: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, IEEE, 2001, p. I.

[30]

Liu Wei, Lin Dahua, Tang Xiaoou, Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, IEEE, 2005, pp. 478–484.

[31]

Mehdi S.M. Sajjadi, Bernhard Scholkopf, Michael Hirsch, Enhancenet: Single image super-resolution through automated texture synthesis, in: IEEE International Conference on Computer Vision, 2017, pp. 4491–4500.

[32]

Choi Jae Young, Ro Yong Man, Plataniotis Konstantinos N., Color face recognition for degraded face images, IEEE Trans. Syst. Man Cybern. B 39 (5) (2009) 1217–1230.

[33]

Biswas Soma, Bowyer Kevin W., Flynn Patrick J., Multidimensional scaling for matching low-resolution face images, IEEE Trans. Pattern Anal. Mach. Intell. 34 (10) (2011) 2019–2030.

[34]

Wright John, Yang Allen Y., Ganesh Arvind, Sastry S. Shankar, Ma Yi, Robust face recognition via sparse representation, IEEE Trans. Pattern Anal. Mach. Intell. 31 (2) (2008) 210–227.

Digital Library

[35]

Gunturk Bahadir K., Batur Aziz Umit, Altunbasak Yucel, Hayes Monson H., Mersereau Russell M., Eigenface-domain super-resolution for face recognition, IEEE Trans. Image Process. 12 (5) (2003) 597–606.

[36]

Zhangyang Wang, Shiyu Chang, Yingzhen Yang, Ding Liu, Thomas S Huang, Studying very low resolution recognition using deep networks, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4792–4800.

[37]

Xin Yu, Basura Fernando, Richard Hartley, Fatih Porikli, Super-resolving very low-resolution face images with supplementary attributes, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 908–917.

[38]

Maneet Singh, Shruti Nagpal, Richa Singh, Mayank Vatsa, Dual directed capsule network for very low resolution image recognition, in: International Conference on Computer Vision, 2019, pp. 340–349.

[39]

Zhao Xiaotong, Li Wei, Zhang Yifan, Feng Zhiyong, Residual super-resolution single shot network for low-resolution object detection, IEEE Access 6 (2018) 47780–47793.

[40]

Lu Qi, Jason Kuen, Jiuxiang Gu, Zhe Lin, Yi Wang, Yukang Chen, Yanwei Li, Jiaya Jia, Multi-scale aligned distillation for low-resolution detection, in: Conference on Computer Vision and Pattern Recognition, 2021, pp. 14443–14453.

[41]

Bautista Carlo Migel, Dy Clifford Austin, Mañalac Miguel Iñigo, Orbe Raphael Angelo, Cordel Macario, Convolutional neural network for vehicle detection in low resolution traffic videos, in: IEEE Region 10 Symposium, TENSYMP, IEEE, 2016, pp. 277–281.

[42]

Wang Xiao, Chen Jun, Liang Chao, Chen Chen, Wang Zheng, Hu Ruimin, Low-resolution pedestrian detection via a novel resolution-score discriminative surface, in: IEEE International Conference on Multimedia and Expo, ICME, IEEE, 2017, pp. 1123–1128.

[43]

Yin Cui, Yang Song, Chen Sun, Andrew Howard, Serge Belongie, Large scale fine-grained categorization and domain-specific transfer learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4109–4118.

[44]

Sermanet Pierre, Frome Andrea, Real Esteban, Attention for fine-grained categorization, 2014, arXiv preprint arXiv:1412.7054.

[45]

Sperazza Michael, Moore Johnnie N., Hendrix Marc S., High-resolution particle size analysis of naturally occurring very fine-grained sediment through laser diffractometry, J. Sediment. Res. 74 (5) (2004) 736–743.

[46]

Donahue Jeff, Jia Yangqing, Vinyals Oriol, Hoffman Judy, Zhang Ning, Tzeng Eric, Darrell Trevor, Decaf: A deep convolutional activation feature for generic visual recognition, in: International Conference on Machine Learning, PMLR, 2014, pp. 647–655.

[47]

Di Lin, Xiaoyong Shen, Cewu Lu, Jiaya Jia, Deep lac: Deep localization, alignment and classification for fine-grained recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1666–1674.

[48]

Shih Kevin J., Mallya Arun, Singh Saurabh, Hoiem Derek, Part localization using multi-proposal consensus for fine-grained categorization, 2015, arXiv preprint arXiv:1507.06332.

[49]

Zhang Ning, Donahue Jeff, Girshick Ross, Darrell Trevor, Part-based R-CNNs for fine-grained category detection, in: European Conference on Computer Vision, Springer, 2014, pp. 834–849.

[50]

Yaming Wang, Vlad I. Morariu, Larry S. Davis, Learning a discriminative filter bank within a CNN for fine-grained recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4148–4157.

[51]

Xu Kunran, Lai Rui, Gu Lin, Li Yishi, Multiresolution discriminative mixup network for fine-grained visual categorization, IEEE Trans. Neural Netw. Learn. Syst. (2021).

[52]

Cai Dingding, Chen Ke, Qian Yanlin, Kämäräinen Joni-Kristian, Convolutional low-resolution fine-grained classification, Pattern Recognit. Lett. 119 (2019) 166–171.

[53]

Peng Xingchao, Hoffman Judy, Stella X. Yu, Saenko Kate, Fine-to-coarse knowledge transfer for low-res image classification, in: International Conference on Image Processing, ICIP, IEEE, 2016, pp. 3683–3687.

[54]

Kim Sangwon, Nam Jaeyeal, Ko Byoung Chul, Vit-net: Interpretable vision transformers with neural tree decoder, in: International Conference on Machine Learning, PMLR, 2022, pp. 11162–11172.

[55]

Xu Qin, Wang Jiahui, Jiang Bo, Luo Bin, Fine-grained visual classification via internal ensemble learning transformer, IEEE Trans. Multimed. (2023).

[56]

Gou Jianping, Yu Baosheng, Maybank Stephen J, Tao Dacheng, Knowledge distillation: A survey, Int. J. Comput. Vis. 129 (6) (2021) 1789–1819.

Digital Library

[57]

Romero Adriana, Ballas Nicolas, Kahou Samira Ebrahimi, Chassang Antoine, Gatta Carlo, Bengio Yoshua, Fitnets: Hints for thin deep nets, 2014, arXiv preprint arXiv:1412.6550.

[58]

Huang Zehao, Wang Naiyan, Like what you like: Knowledge distill via neuron selectivity transfer, 2017, arXiv preprint arXiv:1707.01219.

[59]

Zhou Guorui, Fan Ying, Cui Runpeng, Bian Weijie, Zhu Xiaoqiang, Gai Kun, Rocket launching: A universal and efficient framework for training well-performing light net, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.

[60]

Chen Defang, Mei Jian-Ping, Zhang Yuan, Wang Can, Wang Zhe, Feng Yan, Chen Chun, Cross-layer distillation with semantic calibration, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 7028–7036.

[61]

Junho Yim, Donggyu Joo, Jihoon Bae, Junmo Kim, A gift from knowledge distillation: Fast optimization, network minimization and transfer learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4133–4141.

[62]

Seung Hyun Lee, Dae Ha Kim, Byung Cheol Song, Self-supervised knowledge distillation using singular value decomposition, in: European Conference on Computer Vision, ECCV, 2018, pp. 335–350.

[63]

Frederick Tung, Greg Mori, Similarity-preserving knowledge distillation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1365–1374.

[64]

Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho, Relational knowledge distillation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3967–3976.

[65]

Zheng Wenzhao, Huang Yuanhui, Zhang Borui, Zhou Jie, Lu Jiwen, Dynamic metric learning with cross-level concept distillation, in: European Conference on Computer Vision, Springer, 2022, pp. 197–213.

[66]

Huang Shaoli, Wang Xinchao, Tao Dacheng, Snapmix: Semantically proportional mixing for augmenting fine-grained data, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 1628–1636.

[67]

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo, Cutmix: Regularization strategy to train strong classifiers with localizable features, in: IEEE Conference on Computer Vision and Pattern Recognition, 2019.

[68]

Zhao Borui, Cui Quan, Song Renjie, Qiu Yiyu, Liang Jiajun, Decoupled knowledge distillation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11953–11962.

[69]

Tian Yonglong, Krishnan Dilip, Isola Phillip, Contrastive representation distillation, 2019, arXiv preprint arXiv:1910.10699.

[70]

Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, Jin Young Choi, A comprehensive overhaul of feature distillation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1921–1930.

[71]

Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia, Distilling knowledge via knowledge review, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5008–5017.

[72]

Zhang Tian, Chang Dongliang, Ma Zhanyu, Guo Jun, Progressive co-attention network for fine-grained visual classification, in: Visual Communications and Image Processing (VCIP), IEEE, 2021, pp. 1–5.

[73]

Du Ruoyi, Chang Dongliang, Bhunia Ayan Kumar, Xie Jiyang, Ma Zhanyu, Song Yi-Zhe, Guo Jun, Fine-grained visual classification via progressive multi-granularity training of jigsaw patches, in: European Conference on Computer Vision, Springer, 2020, pp. 153–168.

[74]

Zhuang Peiqin, Wang Yali, Qiao Yu, Learning attentive pairwise interaction for fine-grained classification, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 13130–13137.

Recommendations

Degradation model and attention guided distillation approach for low resolution face recognition
Abstract
Deep convolution neural networks (CNN) have shown their efficacy in face recognition tasks due to their ability to extract highly discriminant face representations from face images. On high-resolution benchmark datasets, outstanding ...
Highlights
- A highly accurate novel scheme for low-resolution face recognition is proposed
- Synthesizing low-resolution training images with real-world degradation effects.
- Distillation of informative high-resolution features from teacher to ...
Distillation and bound entanglement

Quantum entanglement has been known for over sixty years, however the full significance of it as a basic resource in quantum information theory is only being discovered. The fundamental problem is that the decoherence effect due to the environment ...
Discriminative information restoration and extraction for weakly supervised low-resolution fine-grained image recognition
Highlights
- To the best of our knowledge, we are the first to address the issue of weakly supervised low-resolution fine-grained image recognition in an end-to-end manner. By enhancing the network’s perception of discriminative features, the necessary ...
Abstract
The existing methods of fine-grained image recognition mainly devote to learning subtle yet discriminative features from the high-resolution input. However, their performance deteriorates significantly when they are used for low quality images ...

Comments

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 148, Issue C

Apr 2024

747 pages

ISSN:0031-3203

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 17 April 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents