Article

From Shadow Segmentation to Shadow Removal

Authors:

Dimitris SamarasAuthors Info & Claims

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI

Pages 264 - 281

https://doi.org/10.1007/978-3-030-58621-8_16

Published: 23 August 2020 Publication History

Abstract

The requirement for paired shadow and shadow-free images limits the size and diversity of shadow removal datasets and hinders the possibility of training large-scale, robust shadow removal algorithms. We propose a shadow removal method that can be trained using only shadow and non-shadow patches cropped from the shadow images themselves. Our method is trained via an adversarial framework, following a physical model of shadow formation. Our central contribution is a set of physics-based constraints that enables this adversarial training. Our method achieves competitive shadow removal results compared to state-of-the-art methods that are trained with fully paired shadow and shadow-free images. The advantages of our training regime are even more pronounced in shadow removal for videos. Our method can be fine-tuned on a testing video with only the shadow masks generated by a pre-trained shadow detector and outperforms state-of-the-art methods on this challenging test. We illustrate the advantages of our method on our proposed video shadow removal dataset.

References

[1]

Arbel E and Hel-Or H Shadow removal using intensity surfaces and texture anchor points IEEE Trans. Pattern Anal. Mach. Intell. 2011 33 1202-1216

[2]

Choi, Y., Choi, M.J., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2017)

[3]

Chuang YY, Goldman DB, Curless B, Salesin DH, and Szeliski R Shadow matting and compositing ACM Trans. Graph. 2003 22 3 494-500 Special Issue of SIGGRAPH 2003 Proceeding

[4]

Dare P Shadow analysis in high-resolution satellite imagery of urban areas Photogram. Eng. Remote Sens. 2005 71 169-177

[5]

Ding, B., Long, C., Zhang, L., Xiao, C.: ARGAN: attentive recurrent generative adversarial network for shadow detection and removal. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10212–10221 (2019)

[6]

Drew, M.S.: Recovery of chromaticity image free from shadows via illumination invariance. In: IEEE Workshop on Color and Photometric Methods in Computer Vision, ICCV’03, pp. 32–39 (2003)

[7]

Finlayson, G., Drew, M.S.: 4-sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities. In: Proceedings of the International Conference on Computer Vision, vol. 2, pp. 473–480 (2001).

[8]

Finlayson G, Hordley S, Lu C, and Drew M On the removal of shadows from images IEEE Trans. Pattern Anal. Mach. Intell. 2006 28 1 59-68

[9]

Finlayson C, Hordley SD, and Drew MS Heyden A, Sparr G, Nielsen M, and Johansen P Removing shadows from images Computer Vision – ECCV 2002 2002 Berlin, Heidelberg Springer 823-836

[10]

Fredembach, C., Finlayson, G.D.: Hamiltonian path based shadow removal. In: BMVC (2005)

[11]

Gong, H., Cosker, D.: Interactive removal and ground truth for difficult shadow scenes. J. Opt. Soc. Am. A 33(9), 1798–1811 (2016). http://josaa.osa.org/abstract.cfm?URI=josaa-33-9-1798

[12]

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)

[13]

Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011)

[14]

Guo R, Dai Q, and Hoiem D Paired regions for shadow detection and removal IEEE Trans. Pattern Anal. Mach. Intell. 2012 35 12 2956-2967

[15]

Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-ShadowGAN: learning to remove shadows from unpaired data. In: ICCV (2019)

[16]

Hu, X., Wang, T., Fu, C.W., Jiang, Y., Wang, Q., Heng, P.A.: Revisiting shadow detection: a new benchmark dataset for complex world. arXiv:abs/1911.06998 (2019)

[17]

Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

[18]

KaewTrakulPong, P., Bowden, R.: An improved adaptive background mixture model for real- time tracking with shadow detection (2002)

[19]

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (2015)

[20]

Le, H., Goncalves, B., Samaras, D., Lynch, H.: Weakly labeling the antarctic: the penguin colony case. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)

[21]

Le, H., Nguyen, V., Yu, C.P., Samaras, D.: Geodesic distance histogram feature for video segmentation. In: ACCV (2016)

[22]

Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: Proceedings of the International Conference on Computer Vision (2019)

[23]

Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+D Net: training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (2018)

[24]

Le, H., Yu, C.P., Zelinsky, G., Samaras, D.: Co-localization with category-consistent features and geodesic distance propagation. In: ICCV 2017 Workshop on CEFRL: Compact and Efficient Feature Representation and Learning in Computer Vision (2017)

[25]

Li Y, Tang S, Zhang R, Zhang Y, Li J, and Yan S Asymmetric gan for unpaired image-to-image translation IEEE Trans. Image Process. 2019 28 5881-5896

[26]

Liu F and Gleicher M Forsyth D, Torr P, and Zisserman A Texture-consistent shadow removal Computer Vision – ECCV 2008 2008 Berlin, Heidelberg Springer 437-450

[27]

Liu, H., Gu, X., Samaras, D.: Wasserstein GAN with quadratic transport cost. In: The IEEE International Conference on Computer Vision (ICCV) (2019)

[28]

Liu, H., Xianfeng, G., Samaras, D.: A two-step computation of the exact GAN Wasserstein distance. In: International Conference on Machine Learning, pp. 3165–3174 (2018)

[29]

Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. arXiv:abs/1703.00848 (2017)

[30]

Mescheder, L., Nowozin, S., Geiger, A.: Which training methods for GANs do actually converge? In: International Conference on Machine Learning (2018)

[31]

Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: International Conference on Machine Learning (2018)

[32]

Müller, T., Erdnüeß, B.: Brightness correction and shadow removal for video change detection with UAVs. In: Defense + Commercial Sensing (2019)

[33]

Panagopoulos A, Wang C, Samaras D, and Paragios N Kutulakos KN Estimating shadows with the bright channel cue Trends and Topics in Computer Vision – ECCV 2010 2010 Berlin, Heidelberg Springer 1-12

[34]

Panagopoulos A, Wang C, Samaras D, and Paragios N Simultaneous cast shadows, illumination and geometry inference using hypergraphs IEEE Trans. Pattern Anal. Mach. Intell. 2013 35 2 437-449

[35]

Porter T and Duff T Compositing digital images Proc. ACM SIGGRAPH Conf. Comput. Graph. 1984 18 3 1-12

[36]

Prati A, Mikic I, Trivedi MM, and Cucchiara R Detecting moving shadows: algorithms and evaluation IEEE Trans. Pattern Anal. Mach. Intell. 2003 25 918-923

[37]

Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.H.: DeshadowNet: a multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

[38]

Shiting, W., Hong, Z.: Clustering-based shadow edge detection in a single color image. In: International Conference on Mechatronic Sciences, Electric Engineering and Computer, pp. 1038–1041 (2013).

[39]

Shor Y and Lischinski D The shadow meets the mask: pyramid-based shadow removal Comput. Graph. Forum 2008 27 2 577-586

[40]

Smith, A.R., Blinn, J.F.: Blue screen matting. In: Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (1996)

[41]

Su N, Zhang Y, Tian S, Yan Y, and Miao X Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2016 9 2568-2582

[42]

Thanh-Tung, H., Tran, T., Venkatesh, S.: Improving generalization and stability of generative adversarial networks. In: International Conference on Learning Representations (2019)

[43]

Vicente, T.F.Y., Hoai, M., Samaras, D.: Noisy label recovery for shadow detection in unfamiliar domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

[44]

Vicente TFY, Hoai M, and Samaras D Leave-one-out kernel optimization for shadow detection and removal IEEE Transactions on Pattern Analysis and Machine Intelligence 2018 40 3 682-695

[45]

Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Proceedings of the European Conference on Computer Vision (2016)

[46]

Vicente, T.F.Y., Samaras, D.: Single image shadow removal via neighbor-based region relighting. In: Proceedings of the European Conference on Computer Vision Workshops (2014)

[47]

Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

[48]

Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: CVPR (2020)

[49]

Wright, S.: Digital Compositing for Film and Video. Focal Press (2001)

[50]

Wu, Q., Zhang, W.,Vijay Kumar, B.V.K.: Strong shadow removal via patch-based shadow edge detection. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2177–2182 (2012)

[51]

Wu, T.P., Tang, C.K.: A Bayesian approach for shadow extraction from a single image. In: Tenth IEEE International Conference on Computer Vision (ICCV’05), vol. 1 1, pp. 480–487 (2005)

[52]

Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Natural shadow matting. ACM Trans. Graph. 26, 2 (2007). http://doi.acm.org/10.1145/1243980.1243982

[53]

Yang Q, Tan K, and Ahuja N Shadow removal using bilateral filtering IEEE Trans. Image Process. 2012 21 4361-4368

[54]

Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868–2876 (2017)

[55]

Zhang, L., Long, C., Zhang, X., Xiao, C.: RIS-GAN: explore residual and illumination with generative adversarial networks for shadow removal. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)

[56]

Zhang W, Zhao X, Morvan JM, and Chen L Improving shadow suppression for illumination robust face recognition IEEE Trans. Pattern Anal. Mach. Intell. 2019 41 611-624

[57]

Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.H.: Distraction-aware shadow detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5162–5171 (2019)

[58]

Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)

[59]

Zhu, L., et al.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (2018)

Cited By

Hsieh CDing J(2024)ADSP: Advanced Dataset for Shadow Processing, Enabling Visible Occluders via Synthesizing StrategyComputer Vision – ACCV 202410.1007/978-981-96-0917-8_19(329-347)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1007/978-981-96-0917-8_19
Howlader PLe HSamaras D(2024)Weighting Pseudo-labels via High-Activation Feature Index Similarity and Object Detection for Semi-supervised SegmentationComputer Vision – ECCV 202410.1007/978-3-031-73226-3_26(456-474)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73226-3_26
Winter DCohen MFruchter SPritch YRav-Acha AHoshen Y(2024)ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and InsertionComputer Vision – ECCV 202410.1007/978-3-031-72980-5_7(112-129)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72980-5_7
Show More Cited By

Index Terms

From Shadow Segmentation to Shadow Removal
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Computer graphics
    1. Image manipulation

Index terms have been assigned to the content through auto-classification.

Recommendations

Simple shadow removal using shadow depth map and illumination-invariant feature
Abstract
Shadows included in images provide useful information for visual scene analysis, but are also factors that negatively affect digital image analysis. Therefore, shadow detection and removal must be considered essential in the preprocessing of the ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...
A two-stage progressive shadow removal network
Abstract
Removing image shadows has been a challenging task in computer vision due to its diversity and complexity. Shadow removal techniques have been greatly enhanced by deep learning and shadow image datasets, but state-of-the-art methods generally ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI

Aug 2020

857 pages

ISBN:978-3-030-58620-1

DOI:10.1007/978-3-030-58621-8

Editors:
Andrea Vedaldi
University of Oxford, Oxford, UK
,
Horst Bischof
Graz University of Technology, Graz, Austria
,
Thomas Brox
University of Freiburg, Freiburg im Breisgau, Germany
,
Jan-Michael Frahm
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

© Springer Nature Switzerland AG 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 August 2020

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hsieh CDing J(2024)ADSP: Advanced Dataset for Shadow Processing, Enabling Visible Occluders via Synthesizing StrategyComputer Vision – ACCV 202410.1007/978-981-96-0917-8_19(329-347)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1007/978-981-96-0917-8_19
Howlader PLe HSamaras D(2024)Weighting Pseudo-labels via High-Activation Feature Index Similarity and Object Detection for Semi-supervised SegmentationComputer Vision – ECCV 202410.1007/978-3-031-73226-3_26(456-474)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73226-3_26
Winter DCohen MFruchter SPritch YRav-Acha AHoshen Y(2024)ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and InsertionComputer Vision – ECCV 202410.1007/978-3-031-72980-5_7(112-129)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72980-5_7
Vasluianu FSeizinger TWu ZRanjan RTimofte R(2024)Towards Image Ambient Lighting NormalizationComputer Vision – ECCV 202410.1007/978-3-031-72897-6_22(385-404)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72897-6_22
Yu JHe PPeng ZEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)FSR-Net: Deep Fourier Network for Shadow RemovalProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612359(2335-2343)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612359
Zhou PLu DLi CZhang JLiu LWang CMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Unsupervised Textured Terrain Generation via Differentiable RenderingProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548297(2654-2662)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548297
Zhu YFu XCao CWang XSun QZha ZMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Single Image Shadow Detection via Complementary MechanismProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547904(6717-6726)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547904
Jie LZhang H(2022)MGRLN-Net: Mask-Guided Residual Learning Network for Joint Single-Image Shadow Detection and RemovalComputer Vision – ACCV 202210.1007/978-3-031-26313-2_28(460-476)Online publication date: 4-Dec-2022
https://dl.acm.org/doi/10.1007/978-3-031-26313-2_28
Wan JYin HWu ZWu XLiu YWang S(2022)Style-Guided Shadow RemovalComputer Vision – ECCV 202210.1007/978-3-031-19800-7_21(361-378)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19800-7_21

View Options

View options

Figures

Tables

Media

View Table of Conten