survey

Public Access

Multi-Label Active Learning Algorithms for Image Classification: Overview and Future Promise

Authors:

Victor S. Sheng,

Tetiana Dadakova,

Christine Leon Swisher,

Pengpeng ZhaoAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 53, Issue 2

Article No.: 28, Pages 1 - 35

https://doi.org/10.1145/3379504

Published: 20 March 2020 Publication History

All formats PDF

Abstract

Image classification is a key task in image understanding, and multi-label image classification has become a popular topic in recent years. However, the success of multi-label image classification is closely related to the way of constructing a training set. As active learning aims to construct an effective training set through iteratively selecting the most informative examples to query labels from annotators, it was introduced into multi-label image classification. Accordingly, multi-label active learning is becoming an important research direction. In this work, we first review existing multi-label active learning algorithms for image classification. These algorithms can be categorized into two top groups from two aspects respectively: sampling and annotation. The most important component of multi-label active learning is to design an effective sampling strategy that actively selects the examples with the highest informativeness from an unlabeled data pool, according to various information measures. Thus, different informativeness measures are emphasized in this survey. Furthermore, this work also makes a deep investigation on existing challenging issues and future promises in multi-label active learning with a focus on four core aspects: example dimension, label dimension, annotation, and application extension.

References

[1]

Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. 1993. Mining association rules between sets of items in large databases. In Proceedings of the ACM International Conference on Management of Data (SIGMOD’93). 207--216.

Digital Library

[2]

Erin L. Allwein, Robert E. Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A unifying approach for margin classifiers. In Proceedings of the 17th International Conference on Machine Learning (ICML’00). 9--16.

Digital Library

[3]

Massih R. Amini and Patrick Gallinari. 2005. Semi-supervised learning with an imperfect supervisor. Knowl. Inf. Syst. 13, 1 (2005), 1--42.

[4]

Dana Angluin. 1988. Queries and concept learning. Mach. Learn. 2, 4 (1988), 319--342.

[5]

Mustafa Bilgic. 2012. Combining active learning and dynamic dimensionality reduction. In Proceedings of the 2012 SIAM International Conference on Data Mining. SIAM, 696--707.

[6]

Hans Kristian Bø, Ole Solheim, Asgeir Store Jakola, Kjell-Arne Kvistad, Ingerid Reinertsen, and Erik Magnus Berntsen. 2017. Intra-rater variability in low-grade glioma segmentation. J. Neuro-oncol. 131, 2 (2017), 393--402.

[7]

Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning multi-label scene classification. Pattern Recogn. 37, 9 (2004), 1757--1771.

[8]

Selcuk Bucak, Serhat, Rong Jin, and Anil K. Jain. 2011. Multi-label learning with incomplete class assignments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 2801--2808.

[9]

Xiangyong Cao, Yang Chen, Qian Zhao, Deyu Meng, Yao Wang, Dong Wang, and Zongben Xu. 2016. Low-rank matrix factorization under general mixture noise distributions. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’16). 1493--1501.

Digital Library

[10]

Shayok Chakraborty, Vineeth Balasubramanian, and Sethuraman Panchanathan. 2011. Optimal batch selection for active learning in multi-label classification. In Proceedings of the 19th ACM International Conference on Multimedia. ACM, 1413--1416.

Digital Library

[11]

Beijing Chen, Huazhong Shu, Gouenou Coatrieux, Gang Chen, Xingming Sun, and Jean Louis Coatrieux. 2015. Color image analysis by quaternion-type moments. J. Math. Imag. Vis. 51, 1 (2015), 124--144.

Digital Library

[12]

Gang Chen, Yangqiu Song, Fei Wang, and Changshui Zhang. 2008. Semi-supervised multi-label learning by solving a sylvester equation. In Proceedings of the SIAM International Conference on Data Mining (SDM’08). 410--419.

[13]

Myung Jin Choi, Antonio Torralba, and Alan S. Willsky. 2012. A tree-based context model for object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2 (2012), 240--252.

Digital Library

[14]

David Cohn, Les Atlas, and Richard Ladner. 1994. Improving generalization with active learning. Mach. Learn. 15, 2 (1994), 201--221.

[15]

David. A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active learning with statistical models. J. Artif. Intell. Res. 4, 1 (1996), 705--712.

[16]

Ido Dagan and Sean P. Engelson. 1995. Committee-based sampling for training probabilistic classifiers. In Proceedings of the 12th International Conference on Machine Learning (ICML’95). 150--157.

[17]

Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider. 2009. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’09). 259--268.

Digital Library

[18]

Charles Elkan and Keith Noto. 2008. Learning classifiers from only positive and unlabeled data. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’08). 213--220.

Digital Library

[19]

Jonathan J. Entis, Priya Doerga, Lisa Feldman Barrett, and Bradford C. Dickerson. 2012. A reliable protocol for the manual segmentation of the human amygdala and its subregions using ultra-high resolution MRI. Neuroimage 60, 2 (2012), 1226--1235.

[20]

Andrea Esuli and Fabrizio Sebastiani. 2009. Active learning strategies for multi-label text classification. In Proceedings of the European Conference on Information Retrieval (ECIR’09). 102--113.

Digital Library

[21]

Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (2013), 249--283.

[22]

Atsushi Fujii, Takenobu Tokunaga, Kentaro Inui, and Hozumi Tanaka. 1998. Selective sampling for example-based word sense disambiguation. Comput. Linguist. 24, 4 (1998), 573--597.

Digital Library

[23]

Gabriel Pui Cheong Fung, Jeffrey X. Yu, Hongjun Lu, and Philip S. Yu. 2006. Text classification without negative examples revisit. IEEE Trans. Knowl. Data Eng. 18, 1 (2006), 6--20.

Digital Library

[24]

Nengneng Gao, Sheng-Jun Huang, and Songcan Chen. 2016. Multi-label active learning by model guided distribution matching. Front. Comput. Sci. 10, 5 (2016), 845--855.

Digital Library

[25]

Eva Gibaja and Sebastián Ventura. 2015. A tutorial on multilabel learning. ACM Comput. Surv. 47, 3 (2015), 52.

Digital Library

[26]

Robert J. Gillies, Paul E. Kinahan, and Hedvig Hricak. 2015. Radiomics: Images are more than pictures, they are data. Radiology 278, 2 (2015), 563--577.

[27]

Bin Gu and Victor S. Sheng. 2016. A robust regularization path algorithm for v-support vector classification. IEEE Trans. Neur. Netw. Learn. Syst. 1, 99 (2016), 1--8.

[28]

Bin Gu, Victor S. Sheng, and Shuo Li. 2015. Bi-parameter space partition for cost-sensitive SVM. In Proceedings of the International Conference on Artificial Intelligence (AAAI’15). 3532--3539.

[29]

Bin Gu, Xingming Sun, and Victor. S. Sheng. 2017. Structural minimax probability machine. IEEE Trans. Neur. Netw. Learn. Syst. 28, 7 (2017), 1646--1656.

[30]

Anqian Guo, Jian Wu, Victor S. Sheng, Pengpeng Zhao, and Zhiming Cui. 2017. Multi-label active learning with low-rank mapping for image classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’17). 259--264.

[31]

Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. ACM SIGKDD Explor. Newslett. 11, 1 (2009), 10--18.

Digital Library

[32]

Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.

[33]

Xian Sheng Hua and Guo Jun Qi. 2008. Online multi-label active annotation: Towards large-scale content-based video search. In Proceedings of the ACM International Conference on Multimedia (ACM MM’08). 141--150.

Digital Library

[34]

Shengjun Huang, Rong Jin, and Zhihua Zhou. 2010. Active learning by querying informative and representative examples. In Proceedings of the International Conference on Neural Information Processing Systems. 892--900.

[35]

Shengjun Huang, Rong Jin, and Zhihua Zhou. 2014. Active learning by querying informative and representative examples. IEEE Trans. Pattern Anal. Mach. Intell. 36, 10 (2014), 1936--49.

[36]

Sheng Jun Huang, Songcan Chen, and Zhihua Zhou. 2015. Multi-label active learning: Query type matters. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI’15), 946--952.

[37]

Yu Gang Jiang, Qi Dai, Jun Wang, Chong Wah Ngo, Xiangyang Xue, and Shih Fu Chang. 2012. Fast semantic diffusion for large-scale context-based image and video annotation. IEEE Trans. Image Process. 21, 6 (2012), 3080--3091.

Digital Library

[38]

Yang Jiao. 2015. Active Learning-Based Multi-Label Image Classification. Soochow University in China.

[39]

Yang Jiao, Pengpeng Zhao, Jian Wu, Yujie Shi, and Zhiming Cui. 2014. A multicriterion query-based batch mode active learning technique. In Foundations of Intelligent Systems. 669--680.

[40]

Yang Jiao, Pengpeng Zhao, Jian Wu, Xuefeng Xian, Haihui Xu, and Zhiming Cui. 2014. Active multi-label learning with optimal label subset selection. In Proceedings of the International Conference on Advanced Data Mining and Applications (ADMA’14). 523--534.

[41]

Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. 2009. Multi-class active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 2372--2379.

[42]

Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. 2007. Active learning with gaussian processes for object categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’07), Vol. 88. 1--8.

[43]

Aram Kawewong, Rapeeporn Pimpup, and Osamu Hasegawa. 2013. Incremental learning framework for indoor scene recognition. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI’13). 496--502.

[44]

Freda Kemp. 2003. Applied multiple regression/correlation analysis for the behavioral sciences. J. Roy. Stat. Soc. 52, 4 (2003), 691--691.

[45]

Ross D. King, Jem Rowland, Stephen G. Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, and Larisa N. Soldatova. 2009. The automation of science. Science 324, 5923 (2009), 85--89.

[46]

Ross D. King, Kenneth E. Whelan, Ffion M. Jones, Philip G. K. Reiser, Christopher H. Bryant, Stephen H. Muggleton, Douglas B. Kell, and Stephen G. Oliver. 2004. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427, 6971 (2004), 247--252.

[47]

V. Krishnamurthy. 2002. Algorithms for optimal scheduling and management of hidden Markov model sensors. IEEE Trans. Sign. Process. 50, 6 (2002), 1382--1397.

Digital Library

[48]

Kenneth Lang and Eric. B. Baum. 1992. Query learning can work poorly when a human oracle is used. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN’92).

[49]

Guang He Lee, Shao Wen Yang, and Shou De Lin. 2016. Toward implicit sample noise modeling: Deviation-driven matrix factorization. arXiv preprint arXiv:1610.09274 (2016).

[50]

David D. Lewis and Jason Catlett. 1994. Heterogenous uncertainty sampling for supervised learning. In Proceedings of the International Conference on Machine Learning (ICML’94). 148--156.

[51]

Jian Li, Xiaolong Li, Bin Yang, and Xingming Sun. 2015. Segmentation-based image copy-move forgery detection scheme. IEEE Trans. Inf. Forens. Secur. 10, 3 (2015), 507--518.

Digital Library

[52]

Xin Li and Yuhong Guo. 2013. Active learning with multi-label SVM classification. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’13). 1479--1485.

[53]

Xiaoli Li and Bing Liu. 2003. Learning to classify texts using positive and unlabeled data. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’03). 587--592.

[54]

Xuchun Li, Lei. Wang, and Eric. Sung. 2004. Multilabel SVM active learning for image classification. In Proceedings of the International Conference on Image Processing (ICIP’04), Vol. 4. 2207--2210.

[55]

Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I. Sánchez. 2017. A survey on deep learning in medical image analysis. Med. Image Anal. 42 (2017), 60--88.

[56]

B. Liu, Y. Dai, X. Li, and W. S. Lee. 2003. Building text classifiers using positive and unlabeled examples. In Proceedings of the IEEE International Conference on Data Mining (ICDM’03). 179--186.

[57]

Guangcan Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma. 2013. Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1 (2013), 171--184.

Digital Library

[58]

Deyu Meng and Fernando De La Torre. 2013. Robust matrix factorization with unknown noise. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13), 1337--1344.

Digital Library

[59]

Kaushik Mitra, Sameer Sheorey, and Rama Chellappa. 2010. Large-scale matrix factorization with missing data under additional constraints. Ophthal. Res. 40, 1 (2010), 35.

[60]

Gulisong Nasierding and Abbas Z. Kouzani. 2010. Empirical study of multi-label classification methods for image annotation and retrieval. In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA’10). 617--622.

[61]

Reports of International Data Corporation. 2010. Retrieved from http://www.idc.com/prodserv/prodserv.jsp.

[62]

Fredrik Olsson. 2009. A Literature Survey of Active Machine Learning in the Context of Natural Language Processing. Swedish Institute of Computer Science. Technical Report T2009:06.

[63]

Guojun Qi, Xiansheng Hua, Yong Rui, Jinhui Tang, and Hongjiang Zhang. 2008. Two-dimensional active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.

[64]

Guojun Qi, Xiansheng Hua, Yong Rui, Jinhui Tang, and Hongjiang Zhang. 2009. Two-dimensional multilabel active learning with an efficient online adaptation model for image classification. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (2009), 1880--1897.

Digital Library

[65]

Oscar Reyes, Abdulrahman H. Altalhi, and Sebastián Ventura. 2018. Statistical comparisons of active learning strategies over multiple datasets. Knowl.-Based Syst. 145 (2018), 274--288.

[66]

Oscar Reyes, Carlos Morell, and Sebastián Ventura. 2018. Effective active learning strategy for multi-label learning. Neurocomputing 273 (2018), 494--508.

Digital Library

[67]

Oscar Reyes, Eduardo Pérez, María Del Carmen Rodríguez-Hernández, Habib M. Fardoun, and Sebastián Ventura. 2016. JCLAL: A Java framework for active learning. J. Mach. Learn. Res. 17, 1 (2016), 3271--3275.

Digital Library

[68]

Oscar Reyes and Sebastián Ventura. 2018. Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Trans. Intell. Syst. Technol. 9, 4 (2018), 46.

Digital Library

[69]

Burr Settles. 2010. Active Learning Literature Survey. Computer Science Technical Report 1648, University of Wisconsin-Madison.

[70]

H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. 1992. Query by committee. In Proceedings of the The Annual Workshop on Computational Learning Theory (COLT’92). 287--294.

Digital Library

[71]

Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’08). 614--622.

[72]

Victor S. Sheng and Jing Zhang. 2019. Machine learning with crowdsourcing: A brief summary of the past research and future directions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’19), Vol. 33. 9837--9843.

[73]

Mohan Singh and Eoin Curran. 2008. Active learning for multi-label image annotation. In Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science (AICS’08). 173--182.

[74]

Asim Smailagic, Pedro Costa, Hae Young Noh, Devesh Walawalkar, Kartik Khandelwal, Adrian Galdran, Mostafa Mirshekari, Jonathon Fagert, Susu Xu, Pei Zhang, et al. 2018. MedAL: Accurate and robust deep active learning for medical image analysis. In Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA’18). IEEE, 481--488.

[75]

Maja Stikic, Kristof Van Laerhoven, and Bernt Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In Proceedings of the IEEE International Symposium on Wearable Computers. 81--88.

Digital Library

[76]

Michael D. Story and Marco Durante. 2018. Radiogenomics. Medical Physics 45, 11 (2018), e1111--e1122.

[77]

Yu Yin Sun, Yin Zhang, and Zhi Hua Zhou. 2010. Multi-label learning with weak label. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI’10). 593--598.

Digital Library

[78]

Jinhui Tang, Zhengun Zha, Dacheng Tao, and Tatseng Chua. 2012. Semantic-gap-oriented active learning for multilabel image annotation. IEEE Trans. Image Process. 21, 4 (2012), 2354--2360.

Digital Library

[79]

Simon Tong. 2001. Active Learning: Theory and Applications. Stanford University.

[80]

Simon Tong and Daphne Koller. 2001. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 11 (2001), 45--66.

Digital Library

[81]

Grigorios Tsoumakas, Ioannis Katakis, and David Taniar. 2007. Multi-label classification: An overview. Int. J. Data Warehous. Min. 3, 3 (2007), 1--13.

[82]

Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2008. Effective and efficient multilabel classification in domains with large number of labels. In Proceedings of the ECML/PKDD Workshop on Mining Multidimensional Data (MMD’08). 30--44.

[83]

Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2009. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook. Springer, 667--685.

[84]

Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, Jozef Vilcek, and Ioannis Vlahavas. 2011. MULAN: A Java library for multi-label learning. J. Mach. Learn. Res. 12, 7 (2011), 2411--2414.

Digital Library

[85]

Devis Tuia, Michele Volpi, Loris Copa, Mikhail Kanevski, and Jordi Munoz-Mari. 2011. A survey of active learning algorithms for supervised remote sensing image classification. IEEE J. Select. Top. Sign. Process. 5, 3 (2011), 606--617.

[86]

Gokhan Tur, Dilek Hakkani-Tur, and Robert E. Schapire. 2005. Combining active and semi-supervised learning for spoken language understanding. Speech Commun. 45, 2 (2005), 171--186.

[87]

M. Visser, D. M. J. Müller, R. J. M. van Duijn, M. Smits, N. Verburg, E. J. Hendriks, R. J. A. Nabuurs, J. C. J. Bot, R. S. Eijgelaar, M. Witte, et al. 2019. Inter-rater agreement in glioma segmentations on longitudinal MRI. NeuroImage: Clin. 22 (2019), 101727.

[88]

Meng Wang and Xiansheng Hua. 2011. Active learning in multimedia annotation and retrieval: A survey. ACM Trans. Intell. Syst. Technol. 2, 2 (2011), 1--21.

Digital Library

[89]

Zhihua Wei, Hanli Wang, and Rui Zhao. 2013. Semi-supervised multi-label image classification based on nearest neighbor editing. Neurocomputing 119 (2013), 462--468.

Digital Library

[90]

Xuezhi Wen, Ling Shao, Yu Xue, and Wei Fang. 2015. A rapid learning algorithm for vehicle classification. Inf. Sci. 295, 1 (2015), 395--406.

Digital Library

[91]

Jian Wu, Anqian Guo, Victor S. Sheng, Pengpeng Zhao, Zhiming Cui, and Hua Li. 2017. Adaptive low-rank multi-label active learning for image classification. In Proceedings of the ACM International Conference on Multimedia (ACM MM’17). 1336--1344.

Digital Library

[92]

Jian Wu, Chunfeng Lian, Su Ruan, Thomas R. Mazur, Sasa Mutic, Mark A. Anastasio, Perry W. Grigsby, Pierre Vera, and Hua Li. 2019. Treatment outcome prediction for cancer patients based on radiomics and belief function theory. IEEE Trans. Radiat. Plasma Med. Sci. 3, 2 (2019), 216--224.

[93]

Jian Wu, Su Ruan, Chunfeng Lian, Sasa Mutic, Mark A. Anastasio, and Hua Li. 2018. Active learning with noise modeling for medical image annotation. In Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI’18). IEEE, 298--301.

[94]

Jian Wu, Victor S. Sheng, Jing Zhang, Pengpeng Zhao, and Zhiming Cui. 2014. Multi-label active learning for image classification. In Proceedings of the IEEE International Conference on Image Processing (ICIP’14). 5227--5231.

[95]

Jian Wu, Chen Ye, Victor S. Sheng, Yufeng Yao, Pengpeng Zhao, and Zhiming Cui. 2015. Semi-automatic labeling with active learning for multi-label image classification. In Proceedings of the Pacific Rim Conference on Multimedia (PCM’15). 473--482.

[96]

Jian Wu, Chen Ye, Victor S. Sheng, Jing Zhang, Peng Peng Zhao, and Zhiming Cui. 2017. Active learning with label correlation exploration for multi-label image classification. IET Comput. Vis. 11, 7 (2017), 577--584.

[97]

Jian Wu, Shiquan Zhao, Victor S. Sheng, Jing Zhang, Chen Ye, Peng Peng Zhao, and Zhiming Cui. 2017. Weak labeled active learning with conditional label dependence for multi-label image classification. IEEE Trans. Multimedia 19, 6 (2017), 1156--1169.

Digital Library

[98]

Jian Wu, Shiquan Zhao, Victor S. Sheng, Pengpeng Zhao, and Zhiming Cui. 2016. Multi-label active learning for image classification with asymmetrical conditional dependence. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’16). 5227--5231.

[99]

Zhihua Xia, Xinhui Wang, Xingming Sun, Quansheng Liu, and Naixue Xiong. 2016. Steganalysis of LSB matching using differences between nonadjacent pixels. Multimedia Tools Appl. 75, 4 (2016), 1947--1962.

Digital Library

[100]

Xin Shun Xu, Yuan Jiang, Liang Peng, Xiangyang Xue, and Zhi Hua Zhou. 2011. Ensemble approach based on conditional random field for multi-label image and video annotation. In Proceedings of the ACM International Conference on Multimedia (ACM MM’11). 1377--1380.

Digital Library

[101]

Shuicheng Yan. 2012. Practical low-rank matrix approximation under robust L1-norm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 1410--1417.

[102]

Bishan Yang, Jian Tao Sun, Tengjiao Wang, and Zheng Chen. 2009. Effective multi-label active learning for text classification. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’09). 917--926.

Digital Library

[103]

Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. 2017. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’17). Springer, 399--407.

[104]

Shu Jun Yang, Yuan Jiang, and Zhi Hua Zhou. 2013. Multi-instance multi-label learning with weak label. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’13). 1862--1868.

[105]

Chen Ye, Jian Wu, Victor S. Sheng, and Pengpeng Zhao. 2015. Multi-label active learning with label correlation for image classification. In Proceedings of the IEEE International Conference on Image Processing (ICIP’15). 3437--3441.

Digital Library

[106]

Chen Ye, Jian Wu, Victor S. Sheng, Shiquan Zhao, Pengpeng Zhao, and Zhiming Cui. 2015. Multi-label active learning with chi-square statistics for image classification. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR’15). 583--586.

Digital Library

[107]

Hong-Jun Yoon, Arvind Ramanathan, Folami Alamudun, and Georgia Tourassi. 2018. Deep radiogenomics for predicting clinical phenotypes in invasive breast cancer. In Proceedings of the 14th International Workshop on Breast Imaging (IWBI’18), Vol. 10718. International Society for Optics and Photonics, 107181H.

[108]

Guoxian Yu, Guoji Zhang, Huzefa Rangwala, Carlotta Domeniconi, and Zhiwen Yu. 2012. Protein function prediction using weak-label learning. In Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine (ACM BCB’12). 202--209.

Digital Library

[109]

Hwanjo Yu. 2005. SVM selective sampling for ranking with application to data retrieval. In Proceedings of the 11th ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’05). 354--363.

Digital Library

[110]

Zheng Jun Zha, Xian Sheng Hua, Tao Mei, Jingdong Wang, Guo Jun Qi, and Zengfu Wang. 2008. Joint multi-label multi-instance learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.

[111]

Bang Zhang, Yang Wang, and Fang Chen. 2014. Multilabel image classification via high-order label correlation driven active learning. IEEE Trans. Image Process. 23, 3 (2014), 1430--1441.

Digital Library

[112]

Bang Zhang, Yang Wang, and Wei Wang. 2012. Batch mode active learning for multi-label image classification with informative label correlation mining. In Proceedings of the IEEE Workshop on the Applications of Computer Vision (WACV’12). 401--407.

Digital Library

[113]

Jing Zhang, Victor S. Sheng, and Jian Wu. 2019. Crowdsourced label aggregation using bilayer collaborative clustering. IEEE Trans. Neur. Netw. Learn. Syst. 30, 10 (2019), 3172--3185.

[114]

Xiaoyu Zhang, Jian Cheng, Changsheng Xu, Hanqing Lu, and Songde Ma. 2009. Multi-view multi-label active learning for image classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’09). 258--261.

[115]

Yi Zhang. 2010. Multi-task active learning with output constraints. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI’10). 1--6.

[116]

Shiquan Zhao, Jian Wu, Victor S. Sheng, Chen Ye, Pengpeng Zhao, and Zhiming Cui. 2015. Weak labeled multi-label active learning for image classification. In Proceedings of the ACM International Conference on Multimedia (ACM MM’15). 1127--1130.

Digital Library

[117]

Yuhui Zheng, Jeon Byeungwoo, Danhua Xu, Q. M. Jonathan Wu, and Zhang Hui. 2015. Image segmentation by generalized hierarchical fuzzy C-means algorithm. J. Intell. Fuzzy Syst. 28, 2 (2015), 961--973.

[118]

Zhi Hua Zhou. 2010. Semi-supervised learning by disagreement. Knowl. Inf. Syst. 24, 3 (2010), 415--439.

[119]

Zhi Hua Zhou, Yu Yin Sun, and Yu Feng Li. 2009. Multi-instance learning by treating instances as non-I.I.D. samples. In Proceedings of the Annual International Conference on Machine Learning (ICML’09). 1249--1256.

Digital Library

[120]

Zhi Hua Zhou and Min Ling Zhang. 2007. Multi-instance multi-label learning with application to scene classification. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS’07). 1609--1616.

[121]

Zhi Hua Zhou, Min Ling Zhang, Sheng Jun Huang, and Yu Feng Li. 2012. Multi-instance multi-label learning. Artif. Intell. 176, 1 (2012), 2291--2320.

Digital Library

[122]

Xiaojin Zhu. 2005. Semi-Supervised Learning Literature Survey. Computer Science Technical Report 1530, University of Wisconsin-Madison.

Cited By

Luo JGuo JZhu ZDu YYe Y(2024)Optimal feature extraction from multidimensional remote sensing data for orchard identification based on deep learning methodsJournal of Applied Remote Sensing10.1117/1.JRS.18.01451418:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JRS.18.014514
Goswami DChakraborty S(2024)Active Batch Sampling for Multi-label Classification with Binary User Feedback2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00252(2522-2531)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00252
Zhang WWang YYou ZLi YCao GYang ZCui B(2024)NC-ALG: Graph-Based Active Learning Under Noisy Crowd2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00210(2681-2694)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00210
Show More Cited By

Index Terms

Multi-Label Active Learning Algorithms for Image Classification: Overview and Future Promise
1. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Active learning

Recommendations

Effective multi-label active learning for text classification
KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

Labeling text data is quite time-consuming but essential for automatic text classification. Especially, manually creating multiple labels for each document may become impractical when a very large amount of data is needed for training multi-label text ...
Multi-Label Active Learning with Chi-Square Statistics for Image Classification
ICMR '15: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval

Active learning is to select the most informative examples to request their labels. Most previous studies in active learning for multi-label classification didn't pay enough attention on label correlations. This leads to a bad performance for ...
Weak Labeled Multi-Label Active Learning for Image Classification
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

In order to achieve better classification performance with even fewer labeled images, active learning is suitable for these situations. Several active learning methods have been proposed for multi-label image classification, but all of them assume that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 53, Issue 2

March 2021

848 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3388460

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 March 2020

Accepted: 01 December 2019

Revised: 01 August 2019

Received: 01 February 2017

Published in CSUR Volume 53, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Survey
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
4,887
Total Downloads

Downloads (Last 12 months)1,124
Downloads (Last 6 weeks)84

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Luo JGuo JZhu ZDu YYe Y(2024)Optimal feature extraction from multidimensional remote sensing data for orchard identification based on deep learning methodsJournal of Applied Remote Sensing10.1117/1.JRS.18.01451418:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JRS.18.014514
Goswami DChakraborty S(2024)Active Batch Sampling for Multi-label Classification with Binary User Feedback2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00252(2522-2531)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00252
Zhang WWang YYou ZLi YCao GYang ZCui B(2024)NC-ALG: Graph-Based Active Learning Under Noisy Crowd2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00210(2681-2694)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00210
Meinecke CGuéville EWrisley DJänicke S(2024)Is medieval distant viewing possible? : Extending and enriching annotation of legacy image collections using visual analyticsDigital Scholarship in the Humanities10.1093/llc/fqae02039:2(638-656)Online publication date: 24-Apr-2024
https://doi.org/10.1093/llc/fqae020
Sadeghi SNiu C(2024)Augmenting Human Decision-Making in K-12 Education: The Role of Artificial Intelligence in Assisting the Recruitment and Retention of Teachers of Color for Enhanced Diversity and InclusivityLeadership and Policy in Schools10.1080/15700763.2024.2358303(1-21)Online publication date: 27-May-2024
https://doi.org/10.1080/15700763.2024.2358303
Zhang YHuo WTang J(2024)Multi-label feature selection via latent representation learning and dynamic graph constraintsPattern Recognition10.1016/j.patcog.2024.110411151:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.patcog.2024.110411
Mahapatra DBozorgtabar BGe ZReyes M(2024)GANDALF: Graph-based transformer and Data Augmentation Active Learning Framework with interpretable features for multi-label chest Xray classificationMedical Image Analysis10.1016/j.media.2023.10307593(103075)Online publication date: Apr-2024
https://doi.org/10.1016/j.media.2023.103075
Li LXu J(2024)Multi-label category enhancement fusion distillation based on variational estimationKnowledge-Based Systems10.1016/j.knosys.2024.112092300(112092)Online publication date: Sep-2024
https://doi.org/10.1016/j.knosys.2024.112092
Rajan SSoremekun ELe Traon YChattopadhyay S(2024)Distribution-aware fairness test generationJournal of Systems and Software10.1016/j.jss.2024.112090215(112090)Online publication date: Sep-2024
https://doi.org/10.1016/j.jss.2024.112090
Kath HGouvêa TSonntag D(2024)Active Learning in Multi-label Classification of Bioacoustic DataKI 2024: Advances in Artificial Intelligence10.1007/978-3-031-70893-0_9(114-127)Online publication date: 30-Aug-2024
https://doi.org/10.1007/978-3-031-70893-0_9
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents