research-article

Open access

Cycle-Consistent Learning for Weakly Supervised Semantic Segmentation

Authors:

Stephen D.H. Yang,

Weijia LiAuthors Info & Claims

HCMA '22: Proceedings of the 3rd International Workshop on Human-Centric Multimedia Analysis

Pages 7 - 13

https://doi.org/10.1145/3552458.3556445

Published: 10 October 2022 Publication History

Abstract

We investigate a principle way to accomplish the weakly supervised semantic segmentation, only using scribbles as supervision. The key challenge of this task lies in how to accurately propagate the semantic labels from the annotated scribbles to those unlabeled regions so that accurate pseudo masks can be harvested to learn better segmentation models. To tackle this issue, we propose a simple, strong, and unified framework named Cycle-Consistent Learning (CCL) in this work. To be specific, our CCL first utilizes the given scribbles for training and makes a prediction for those unlabeled regions. Then, the predicted regions, in turn, serve as supervision for learning to predict the labeled scribbles. With such a cycle-consistent constraint, the accurate scribbles can reversely help ease those potential noises existing in the unlabeled regions, resulting in better pseudo masks. The training process of our CCL is looped until the network converges in an end-to-end way. We conduct extensive experiments on the popular PASCAL VOC benchmark and achieve a comparable result with the state-of-the-art method. The training mechanism of the CCL is straightforward and can be easily embedded into any future weakly supervised semantic segmentation approach.

References

[1]

Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. 2019. Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations. In CVPR. 2209--2218.

[2]

Jiwoon Ahn and Suha Kwak. 2018. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR. 4981--4990.

[3]

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. 2015. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. In ICLR.

[4]

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2018a. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, Vol. 40, 4 (2018), 834--848.

[5]

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017).

[6]

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018b. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV. 801--818.

[7]

Jifeng Dai, Kaiming He, and Jian Sun. 2015. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV.

Digital Library

[8]

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. IJCV, Vol. 88, 2 (2010), 303--338.

Digital Library

[9]

Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. 2019. Dual Attention Network for Scene Segmentation. In CVPR. 3146--3154.

[10]

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. 2011. Semantic contours from inverse detectors. (2011).

[11]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.

[12]

Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. 2017. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR.

[13]

Philipp Kr"ahenbühl and Vladlen Koltun. 2011. Efficient inference in fully connected crfs with gaussian edge potentials. In Advances in neural information processing systems. 109--117.

[14]

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. 2016. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR.

[15]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.

[16]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In CVPR.

[17]

George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille. 2015. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV. 1742--1750.

[18]

Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR.

[19]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI.

[20]

Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. 2019. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In CVPR. 3136--3145.

[21]

Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. 2018a. Normalized Cut Loss for Weakly-supervised CNN Segmentation. In CVPR.

[22]

Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. 2018b. On Regularized Losses for Weakly-supervised CNN Segmentation. In ECCV.

[23]

Bin Wang, Guojun Qi, Sheng Tang, Tianzhu Zhang, Yunchao Wei, Linghui Li, and Yongdong Zhang. 2019. Boundary perception guidance: a scribble-supervised semantic segmentation approach. In IJCAI. 3663--3669.

[24]

Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. 2020. Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation. In CVPR.

[25]

Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. 2017a. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In CVPR. 1568--1576.

[26]

Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Jiashi Feng, Yao Zhao, and Shuicheng Yan. 2017b. Stc: A simple to complex framework for weakly-supervised semantic segmentation. TPAMI, Vol. 39, 11 (2017), 2314--2320.

Digital Library

[27]

Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection. In ICCV.

[28]

Jingshan Xu, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yuge Huang, Pengcheng Shen, Shaoxin Li, and Jian Yang. 2021. Scribble-supervised semantic segmentation inference. In ICCV.

[29]

Hang Zhang, Kristin J. Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. 2018b. Context Encoding for Semantic Segmentation. In CVPR. 7151--7160.

[30]

Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. 2018a. Top-down neural attention by excitation backprop. IJCV, Vol. 126, 10 (2018), 1084--1102.

Digital Library

[31]

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. 2017. Pyramid scene parsing network. In CVPR. 2881--2890.

[32]

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In CVPR. 2921--2929.

[33]

Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. 2018. Weakly supervised instance segmentation using class peak response. In CVPR. 3791--3800.

[34]

Yi Zhu, Yanzhao Zhou, Huijuan Xu, Qixiang Ye, David Doermann, and Jianbin Jiao. 2019. Learning Instance Activation Maps for Weakly Supervised Instance Segmentation. In CVPR. 3116--3125.

Cited By

Wang YZhang JKan MShan SChen X(2023)BLPSeg: Balance the Label Preference in Scribble-Supervised Semantic SegmentationIEEE Transactions on Image Processing10.1109/TIP.2023.330134232(4921-4934)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3301342
Wu LZhong ZFang LHe XLiu QMa JChen H(2023)Sparsely Annotated Semantic Segmentation with Adaptive Gaussian Mixtures2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01483(15454-15464)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01483

Index Terms

Cycle-Consistent Learning for Weakly Supervised Semantic Segmentation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Image segmentation
2. Networks
  1. Network architectures
    1. Network design principles

Recommendations

Semi- and Weakly- Supervised Semantic Segmentation with Deep Convolutional Neural Networks
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Successful semantic segmentation methods typically rely on the training datasets containing a large number of pixel-wise labeled images. To alleviate the dependence on such a fully annotated training dataset, in this paper, we propose a semi- and weakly-...
Boosted MIML method for weakly-supervised image semantic segmentation

Weakly-supervised image semantic segmentation aims to segment images into semantically consistent regions with only image-level labels are available, and is of great significance for fine-grained image analysis, retrieval and other possible ...
Semantic Segmentation based on Stacked Discriminative Autoencoders and Context-Constrained Weakly Supervised Learning
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

In this paper, we focus on tacking the problem of weakly supervised semantic segmentation. The aim is to predict the class label of image regions under weakly supervised settings, where training images are only provided with image-level labels ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HCMA '22: Proceedings of the 3rd International Workshop on Human-Centric Multimedia Analysis

October 2022

106 pages

ISBN:9781450394925

DOI:10.1145/3552458

Program Chairs:
Dingwen Zhang
Northwestern Polytechnical University, Xi'an, China
,
Chaowei Fang
Xidian University, Xi'an, China
,
Wu Liu
JD AI Research, Beijing, China
,
Xinchen Liu
JD AI Research, Beijing, China
,
Jingkuan Song
University of Electronic Science & Technology of China, Chengdu, China
,
Hongyuan Zhu
Agency for Science, Technology, and Research (A*STAR), Singapore
,
Wenbing Huang
Tsinghua University, Beijing, China
,
John Smith
IBM Research, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

HCMA '22 Paper Acceptance Rate 12 of 21 submissions, 57%;

Overall Acceptance Rate 12 of 21 submissions, 57%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
281
Total Downloads

Downloads (Last 12 months)116
Downloads (Last 6 weeks)12

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YZhang JKan MShan SChen X(2023)BLPSeg: Balance the Label Preference in Scribble-Supervised Semantic SegmentationIEEE Transactions on Image Processing10.1109/TIP.2023.330134232(4921-4934)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3301342
Wu LZhong ZFang LHe XLiu QMa JChen H(2023)Sparsely Annotated Semantic Segmentation with Adaptive Gaussian Mixtures2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01483(15454-15464)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01483

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents