research-article

Active learning in multi-label image classification with graph convolutional network embedding

Authors:

Ke QinAuthors Info & Claims

Volume 148, Issue C

Pages 56 - 65

https://doi.org/10.1016/j.future.2023.05.028

Published: 01 November 2023 Publication History

Abstract

Active learning has achieved considerable success in sample selection for deep learning models and has been widely used to address the issue of high-cost sample annotation. However, most of the existing active learning methods focus on single-label image classification and have limited use in multi-label scenarios. To address this issue and take advantage of label associations, we propose an active learning model based on the graph convolutional network (GCN) embedding and loss prediction network. Specifically, we construct a heterogeneous information network (HIN) that uses GCN embeddings to learn multiple label associations, as well as associations between images and labels. We also use a loss prediction network to predict target losses of unlabeled inputs. In addition, we propose a dynamic active coefficient to adjust the proportion of active learning gradually in the training process. Comprehensive multi-label image classification experiments with limited training labels are conducted on the MS-COCO, VOC 2007, and NUS-WIDE datasets. The comparison results demonstrate the superiority of our method compared with conventional methods in terms of both classification accuracy and convergence speed.

Highlights

•

We propose an active learning framework with a GCN embedding and loss prediction module.

•

We propose a dynamic to adjust the AL proportion for more effective sample selection.

•

The extensive multi-label experiment displays the efficacy of the proposed approach.

References

[1]

Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai, Exploit bounding box annotations for multi-label object recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 280–288.

[2]

Gao Bin-Bin, Zhou Hong-Yu, Learning to discover multi-class attentional regions for multi-label image recognition, IEEE Trans. Image Process. 30 (2021) 5920–5932.

[3]

Yu Niange, Hu Xiaolin, Song Binheng, Yang Jian, Zhang Jianwei, Topic-oriented image captioning based on order-embedding, IEEE Trans. Image Process. 28 (6) (2018) 2743–2754.

[4]

Quintanilla Erik, Rawat Yogesh, Sakryukin Andrey, Shah Mubarak, Kankanhalli Mohan, Adversarial learning for personalized tag recommendation, IEEE Trans. Multimed. 23 (2020) 1083–1094.

[5]

Qiang Li, Maoying Qiao, Wei Bian, Dacheng Tao, Conditional graphical lasso for multi-label image classification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2977–2986.

[6]

Wang Keze, Zhang Dongyu, Li Ya, Zhang Ruimao, Lin Liang, Cost-effective active learning for deep image classification, IEEE Trans. Circuits Syst. Video Technol. 27 (12) (2016) 2591–2600.

Digital Library

[7]

Yang Lin, Zhang Yizhe, Chen Jianxu, Zhang Siyuan, Chen Danny Z., Suggestive annotation: A deep active learning framework for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2017, pp. 399–407.

[8]

Buyu Liu, Vittorio Ferrari, Active learning for human pose estimation, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 4363–4372.

[9]

Wu Jian, Sheng Victor S., Zhang Jing, Li Hua, Dadakova Tetiana, Swisher Christine Leon, Cui Zhiming, Zhao Pengpeng, Multi-label active learning algorithms for image classification: Overview and future promise, ACM Comput. Surv. 53 (2) (2020) 1–35.

[10]

Wang Peng, Zhang Peng, Guo Li, Mining multi-label data streams using ensemble-based active learning, in: Proceedings of the 2012 SIAM International Conference on Data Mining, SIAM, 2012, pp. 1131–1140.

[11]

Tang Jinhui, Zha Zheng-Jun, Tao Dacheng, Chua Tat-Seng, Semantic-gap-oriented active learning for multilabel image annotation, IEEE Trans. Image Process. 21 (4) (2011) 2354–2360.

[12]

Xin Li, Yuhong Guo, Active Learning with Multi-Label SVM Classification, in: Twenty-Third International Joint Conference on Artificial Intelligence, 2013.

[13]

Wu Jian, Sheng Victor S., Zhang Jing, Zhao Pengpeng, Cui Zhiming, Multi-label active learning for image classification, in: 2014 IEEE International Conference on Image Processing, ICIP, IEEE, 2014, pp. 5227–5231.

[14]

Deepak Vasisht, Andreas Damianou, Manik Varma, Ashish Kapoor, Active learning for sparse bayesian multilabel classification, in: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014, pp. 472–481.

[15]

Sheng-Jun Huang, Songcan Chen, Zhi-Hua Zhou, Multi-label active learning: Query type matters, in: Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.

[16]

Ye Chen, Wu Jian, Sheng Victor S., Zhao Pengpeng, Cui Zhiming, Multi-label active learning with label correlation for image classification, in: 2015 IEEE International Conference on Image Processing, ICIP, IEEE, 2015, pp. 3437–3441.

[17]

Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, Yanwen Guo, Multi-label image recognition with graph convolutional networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5177–5186.

[18]

Reyes Oscar, Morell Carlos, Ventura Sebastián, Effective active learning strategy for multi-label learning, Neurocomputing 273 (2018) 494–508.

[19]

Li Weimin, Guo Chang, Liu Yanxia, Zhou Xiaokang, Jin Qun, Xin Mingjun, Rumor source localization in social networks based on infection potential energy, Inform. Sci. (2023).

[20]

Shi Chuan, Hu Binbin, Zhao Wayne Xin, Yu Philip S., Heterogeneous information network embedding for recommendation, IEEE Trans. Knowl. Data Eng. 31 (2) (2019) 357–370.

Digital Library

[21]

Yu Xiao, Li Weimin, Yang Bing, Li Xiaorong, Chen Jie, Fu Guohua, Deviation distance entropy: A method for quantifying the dynamic features of biomedical time series, Chaos Solitons Fractals 168 (2023).

[22]

Lin Tsung-Yi, Maire Michael, Belongie Serge, Hays James, Perona Pietro, Ramanan Deva, Dollár Piotr, Zitnick C. Lawrence, Microsoft coco: Common objects in context, in: European Conference on Computer Vision, Springer, 2014, pp. 740–755.

[23]

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng, Nus-wide: a real-world web image database from national university of singapore, in: Proceedings of the ACM International Conference on Image and Video Retrieval, 2009, pp. 1–9.

[24]

Gong Yunchao, Jia Yangqing, Leung Thomas, Toshev Alexander, Ioffe Sergey, Deep convolutional ranking for multilabel image annotation, 2013, arXiv preprint arXiv:1312.4894.

[25]

Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, Yu-Chiang Frank Wang, Order-free rnn with visual attention for multi-label classification, in: Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

[26]

Ben-Baruch Emanuel, Ridnik Tal, Zamir Nadav, Noy Asaf, Friedman Itamar, Protter Matan, Zelnik-Manor Lihi, Asymmetric loss for multi-label classification, 2020, arXiv preprint arXiv:2009.14119.

[27]

Qu Xiwen, Che Hao, Huang Jun, Xu Linchuan, Zheng Xiao, Multi-layered semantic representation network for multi-label image classification, 2021, arXiv preprint arXiv:2106.11596.

[28]

Li Xuchun, Wang Lei, Sung Eric, Multilabel SVM active learning for image classification, in: 2004 International Conference on Image Processing, Vol. 4, ICIP’04, IEEE, 2004, pp. 2207–2210.

[29]

Hu Wenya, Li Weimin, Zhou Xiaokang, Kawai Akira, Fueda Kaoru, Qian Quan, Wang Jianjia, Spatio-temporal graph convolutional networks via view fusion for trajectory data analytics, IEEE Trans. Intell. Transp. Syst. (2022).

[30]

Li Weimin, Ni Lin, Wang Jianjia, Wang Can, Collaborative representation learning for nodes and relations via heterogeneous graph neural network, Knowl.-Based Syst. 255 (2022).

Digital Library

[31]

Kipf Thomas N., Welling Max, Semi-supervised classification with graph convolutional networks, 2016, arXiv preprint arXiv:1609.02907.

[32]

Maas Andrew L., Hannun Awni Y., Ng Andrew Y., et al., Rectifier nonlinearities improve neural network acoustic models, in: Proc. Icml, Citeseer, 2013, p. 3.

[33]

Everingham Mark, Van Gool Luc, Williams Christopher K.I., Winn John, Zisserman Andrew, The pascal visual object classes (voc) challenge, Int. J. Comput. Vis. 88 (2) (2010) 303–338.

Digital Library

[34]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[35]

Van der Maaten Laurens, Hinton Geoffrey, Visualizing data using t-SNE, J. Mach. Learn. Res. 9 (11) (2008).

Cited By

Jin YGao RHe YZhu XWooldridge MDy JNatarajan S(2024)GLDLProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i11.29194(12965-12974)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i11.29194

Index Terms

Active learning in multi-label image classification with graph convolutional network embedding
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Weak Labeled Multi-Label Active Learning for Image Classification
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

In order to achieve better classification performance with even fewer labeled images, active learning is suitable for these situations. Several active learning methods have been proposed for multi-label image classification, but all of them assume that ...
Effective active learning strategy for multi-label learning

Data labelling is commonly an expensive process that requires expert handling. In multi-label data, data labelling is further complicated owing to the experts must label several times each example, as each example belongs to various categories. Active ...
Multi-label active learning through serial–parallel neural networks
Abstract
Multi-label active learning is an extension of supervised learning with high-dimensional label spaces and interactive scenarios. Its key issues include the exploitation of label correlations, handling of missing labels, and selection of query ...
Highlights
- We propose multi-label active learning through serial-parallel neural networks.
- Serial and parallel parts serve for feature extraction and label prediction.
- Pairwise outputs support missing label handling and label uncertainty ...

Comments

Information & Contributors

Information

Published In

cover image Future Generation Computer Systems

Future Generation Computer Systems Volume 148, Issue C

Nov 2023

637 pages

ISSN:0167-739X

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 November 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jin YGao RHe YZhu XWooldridge MDy JNatarajan S(2024)GLDLProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i11.29194(12965-12974)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i11.29194

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents