research-article

Semantic Guided Single Image Reflection Removal

Authors:

Feng LuAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 18, Issue 3s

Article No.: 151, Pages 1 - 23

https://doi.org/10.1145/3510821

Published: 01 November 2022 Publication History

Abstract

Reflection is common when we see through a glass window, which not only is a visual disturbance but also influences the performance of computer vision algorithms. Removing the reflection from a single image, however, is highly ill-posed since the color at each pixel needs to be separated into two values belonging to the clear background and the reflection, respectively. To solve this, existing methods use additional priors such as reflection layer smoothness, double reflection effect, and color consistency to distinguish the two layers. However, these low-level priors may not be consistently valid in real cases. In this paper, inspired by the fact that human beings can separate the two layers easily by recognizing the objects and understanding the scene, we propose to use the object semantic cue, which is high-level information, as the guidance to help reflection removal. Based on the data analysis, we develop a multi-task end-to-end deep learning method with a semantic guidance component, to solve reflection removal and semantic segmentation jointly. Extensive experiments on different datasets show significant performance gain when using high-level object-oriented information. We also demonstrate the application of our method to other computer vision tasks.

Supplementary Material

3510821.app (3510821.app.pdf)

Supplementary appendix

Download
17.60 MB

References

[1]

Amit Agrawal, Ramesh Raskar, Shree K. Nayar, and Yuanzhen Li. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. TOG 24, 3 (2005), 828–835.

Digital Library

[2]

Nikolaos Arvanitopoulos, Radhakrishna Achanta, and Sabine Susstrunk. 2017. Single image reflection suppression. In CVPR.

[3]

Anil S. Baslamisli, Thomas T. Groenestege, Partha Das, Hoang-An Le, Sezer Karaoglu, and Theo Gevers. 2018. Joint learning of intrinsic images and semantic segmentation. In ECCV.

[4]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In ECCV. Springer.

[5]

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV.

[6]

R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A MATLAB-like environment for machine learning. In NIPS Workshop.

[7]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In CVPR.

[8]

Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. 2020. Multi-scale boosted dehazing network with dense feature fusion. In CVPR.

[9]

M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. [n. d.]. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.

[10]

Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. 2017. A generic deep architecture for single image reflection removal and image smoothing. In ICCV.

[11]

Xiaojie Guo, Xiaochun Cao, and Yi Ma. 2014. Robust separation of reflection from multiple images. In CVPR.

[12]

Zhixiang Hao, Shadi You, Yu Li, and Feng Lu. 2019. Learning from synthetic photorealistic raindrop for single image raindrop removal. In ICCV Workshop.

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.

[14]

Yuchen Hong, Qian Zheng, Lingran Zhao, Xudong Jiang, Alex C. Kot, and Boxin Shi. 2021. Panoramic image reflection removal. In CVPR.

[15]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In CVPR.

[16]

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. In ECCV.

[17]

Sutskever Ilya, Martens James, Dahl George, and Hinton Geoffrey. 2013. On the importance of initialization and momentum in deep learning. In ICML.

[18]

Soomin Kim, Yuchi Huo, and Sung-Eui Yoon. 2020. Single image reflection removal with physically-based training images. In CVPR.

[19]

Suhong Kim, Hamed RahmaniKhezri, Seyed Mohammad Nourbakhsh, and Mohamed Hefeeda. 2020. Unsupervised single-image reflection separation using perceptual deep image priors. arXiv preprint arXiv:2009.00702 (2020).

[20]

N. Kong, Y. W. Tai, and J. S. Shin. 2014. A physically-based approach to reflection separation: From physical modeling to constrained optimization. TPAMI 36, 2 (2014), 209–221.

Digital Library

[21]

Chenyang Lei and Qifeng Chen. 2021. Robust reflection removal with reflection-free flash-only cues. In CVPR.

[22]

Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. 2020. Polarized reflection removal with perfect alignment in the wild. In CVPR.

[23]

Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. 2018. Benchmarking single-image dehazing and beyond. TIP (2018).

[24]

Chao Li, Yixiao Yang, Kun He, Stephen Lin, and John E. Hopcroft. 2020. Single image reflection removal through cascaded refinement. In CVPR.

[25]

Rui Li, Simeng Qiu, Guangming Zang, and Wolfgang Heidrich. 2020. Reflection separation via multi-bounce polarization state tracing. In ECCV. Springer.

[26]

Yu Li and Michael S. Brown. 2013. Exploiting reflection change for automatic reflection removal. In ICCV.

[27]

Yu Li and M. S. Brown. 2014. Single image layer separation using relative smoothness. In CVPR.

[28]

Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren, and Wangmeng Zuo. 2020. Two-Stage single image reflection removal with reflection-aware guidance. arXiv preprint arXiv:2012.00945 (2020).

[29]

Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, and Thomas S. Huang. 2020. Connecting image denoising and high-level vision tasks via deep learning. TIP (2020).

[30]

Yunfei Liu, Zhixiang Hao, Shadi You, Yu Li, and Feng Lu. 2019. PBRR: Physically Based Raindrop Rendering. https://liuyunfei.net/project/pbrr/.

[31]

Yunfei Liu and Feng Lu. 2020. Separate in latent space: Unsupervised single image layer separation. In AAAI.

[32]

Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV. Springer.

[33]

Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Learning to see through obstructions. In CVPR.

[34]

Daiqian Ma, Renjie Wan, Boxin Shi, Alex C. Kot, and Ling-Yu Duan. 2019. Learning to jointly generate and separate reflections. In ICCV.

[35]

MATLAB. 2010. version R2017b. The MathWorks Inc., Natick, Massachusetts.

[36]

Ajay Nandoriya, Mohamed Elgharib, Changil Kim, Mohamed Hefeeda, and Wojciech Matusik. 2017. Video reflection removal through spatio-temporal optimization. In ICCV.

[37]

Simon Niklaus, Xuaner Cecilia Zhang, Jonathan T. Barron, Neal Wadhwa, Rahul Garg, Feng Liu, and Tianfan Xue. 2021. Learned dual-view reflection removal. In WACV.

[38]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Workshop.

[39]

Abhijith Punnappurath and Michael S. Brown. 2019. Reflection removal using a dual-pixel sensor. In CVPR.

[40]

Yuhui Quan, Shijie Deng, Yixin Chen, and Hui Ji. 2019. Deep learning for seeing through window with raindrops. In ICCV.

[41]

Mohammad Saeed Rad, Behzad Bozorgtabar, Urs-Viktor Marti, Max Basler, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. SROBB: Targeted perceptual loss for single image super-resolution. In ICCV.

[42]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An incremental improvement. arXiv (2018).

[43]

Wenqi Ren, Jingang Zhang, Xiangyu Xu, Lin Ma, Xiaochun Cao, Gaofeng Meng, and Wei Liu. 2018. Deep video dehazing with semantic segmentation. TIP (2018).

[44]

Tushar Sandhan and Young Choi Jin. 2017. Anti-glare: Tightly constrained optimization for eyeglass reflection removal. In CVPR.

[45]

Y. Y. Schechner, N. Kiryati, and R. Basri. 1998. Separation of transparent layers using focus. In ICCV.

[46]

YiChang Shih, Dilip Krishnan, Fredo Durand, and William T. Freeman. 2015. Reflection removal using ghosting cues. In CVPR.

[47]

Christian Simon and In Kyu Park. 2015. Reflection removal for in-vehicle black box videos. In CVPR.

[48]

Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. TOG 31, 4 (2012), 1–10.

Digital Library

[49]

Renjie Wan, Boxin Shi, Ling Yu Duan, Ah Hwee Tan, and Alex C. Kot. 2017. Benchmarking single-image reflection removal algorithms. In ICCV.

[50]

Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, and Alex C. Kot. 2018. CRRN: Multi-scale guided concurrent reflection removal network. In CVPR.

[51]

Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yu Duan, Ah-Hwee Tan, and Alex C. Kot. 2019. CoRRN: Cooperative reflection removal network. TPAMI (2019).

[52]

Guoqing Wang, Changming Sun, and Arcot Sowmya. 2020. Cascaded attention guidance network for single rainy image restoration. TIP (2020).

[53]

Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang. 2019. Single image reflection removal exploiting misaligned training data and network enhancements. In CVPR.

[54]

Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, Guoqiang Han, and Shengfeng He. 2019. Single image reflection removal beyond linearity. In CVPR.

[55]

Sijia Wen, Yingqiang Zheng, and Feng Lu. 2021. Polarization guided specular reflection separation. TIP (2021).

[56]

Patrick Wieschollek, Orazio Gallo, Jinwei Gu, and Jan Kautz. 2018. Separating reflection and transmission images in the wild. In ECCV.

[57]

Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. TOG 34, 4 (2015), 1–11.

Digital Library

[58]

Jie Yang, Dong Gong, Lingqiao Liu, and Qinfeng Shi. 2018. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In ECCV.

[59]

Jiaolong Yang, Hongdong Li, Yuchao Dai, and Robby T. Tan. 2016. Robust optical flow estimation of double-layer images under transparency or reflection. In CVPR.

[60]

Yang Yang, Wenye Ma, Yin Zheng, Jian-Feng Cai, and Weiyu Xu. 2019. Fast single image reflection suppression via convex optimization. In CVPR.

[61]

Jae-Seong Yun and Jae-Young Sim. 2018. Reflection removal for large-scale 3D point clouds. In CVPR.

[62]

Xuaner Zhang, Ren Ng, and Qifeng Chen. 2018. Single image reflection separation with perceptual losses. In CVPR.

[63]

Yongqiang Zhao, Qunnie Peng, Jize Xue, and Seong G. Kong. 2015. Specular reflection removal using local structural similarity and chromaticity consistency. ICIP (2015).

[64]

Qian Zheng, Boxin Shi, Jinnan Chen, Xudong Jiang, Ling-Yu Duan, and Alex C. Kot. 2021. Single image reflection removal with absorption effect. In CVPR.

[65]

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In NeurIPS.

[66]

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ADE20K dataset. In CVPR.

[67]

Wang Zhou, Bovik Alan Conrad, Sheikh Hamid Rahim, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. TIP (2004).

Cited By

Peng BSun LLei JLiu BShen HLi WHuang Q(2024)Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366357020:8(1-19)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3663570
Huang QLi PHuang YShuang FCai Y(2024)Region-Focused Network for Dense CaptioningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364837020:6(1-20)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3648370
He WLi ZWang HXu TWang ZHuai BYuan NChen E(2024)Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic ElementsACM Transactions on Intelligent Systems and Technology10.1145/364509915:3(1-25)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3645099
Show More Cited By

Index Terms

Semantic Guided Single Image Reflection Removal
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

A Model-Guided Unfolding Network for Single Image Reflection Removal
MMAsia '21: Proceedings of the 3rd ACM International Conference on Multimedia in Asia

Removing undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image ...
Single Image Reflection Removal with Diffusion Model
ICCPR '23: Proceedings of the 2023 12th International Conference on Computing and Pattern Recognition

The removal of undesirable reflections from workpiece surface images captured under industrial conditions is a challenging task in image enhancement for the industry. While existing reflection removal methods have shown promising results in natural ...
Two-stage single image reflection removal with reflection-aware guidance
Abstract
Removing undesired reflection from an image captured through a glass surface is a very challenging problem with many practical applications. For improving reflection removal, cascaded deep models have been usually adopted to estimate the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 3s

October 2022

381 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3567476

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2022

Online AM: 18 February 2022

Accepted: 07 January 2022

Revised: 01 January 2022

Received: 01 July 2021

Published in TOMM Volume 18, Issue 3s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Natural Science Foundation of China (NSFC)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
516
Total Downloads

Downloads (Last 12 months)222
Downloads (Last 6 weeks)6

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Peng BSun LLei JLiu BShen HLi WHuang Q(2024)Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366357020:8(1-19)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3663570
Huang QLi PHuang YShuang FCai Y(2024)Region-Focused Network for Dense CaptioningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364837020:6(1-20)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3648370
He WLi ZWang HXu TWang ZHuai BYuan NChen E(2024)Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic ElementsACM Transactions on Intelligent Systems and Technology10.1145/364509915:3(1-25)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3645099
Qiu HLi HWu QShi HWang LMeng FXu L(2024)Learning Offset Probability Distribution for Accurate Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363721420:5(1-24)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3637214
Lv CZhang DGeng SWu ZHuang H(2024)Color Transfer for Images: A SurveyACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363515220:8(1-29)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3635152
Wu XFeng X(2024)Size Invariant Visual Cryptography Schemes With Evolving Threshold Access StructuresIEEE Transactions on Multimedia10.1109/TMM.2023.328257326(1488-1503)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3282573
Yang CWu XChung M(2023)Enhancement of Information Carrying and Decoding for Visual Cryptography with Error CorrectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361292720:1(1-24)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3612927
Zhang HLi PLiu XYang XAn L(2023)An Iterative Semi-supervised Approach with Pixel-wise Contrastive Loss for Road Extraction in Aerial ImagesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360637420:3(1-21)Online publication date: 10-Nov-2023
https://dl.acm.org/doi/10.1145/3606374
Liu BLei JPeng BYu CLi WLing N(2023)Novel View Synthesis from a Single Unposed Image via Unsupervised LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358746719:6(1-23)Online publication date: 31-May-2023
https://dl.acm.org/doi/10.1145/3587467
Xu YYang ZChen TLi KQing C(2023)Progressive Transformer Machine for Natural Character ReenactmentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355910719:2s(1-22)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3559107
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents