Abstract
Shadow removal is a challenging problem due to its sensitivity to lighting and material conditions. In this paper, we propose a shading-aware shadow processing algorithm, which can automatically detect and remove complex shadows from a single color image. Our framework consists of two key steps. We firstly conduct a shadow-preserving filter upon the image which will effectively remove the image texture while preserving the shadow and shading information. Shadow regions are estimated by establishing a confidence map from the filtered image incorporating depth cue. We then develop a shading-aware optimization framework to remove shadows and recover shading in these regions. The extensive experimental results show that the proposed algorithm produces visually compelling results in a series of challenging images and it can handle complex shadows in both indoor and outdoor scenes. Quantitative and qualitative comparisons with current state-of-the-art methods strongly demonstrate the efficacy of our proposed approach.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Anat, L., Dani, L., Yair, W.: A closed-form solution to natural image matting. IEEE TPAMI 30(2), 228–242 (2007)
Arbel, E., Helor, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE TPAMI 33(6), 1202–1216 (2011)
Barron, J.T., Malik, J.: Intrinsic scene properties from a single rgb-d image. In: CVPR, pp. 17–24 (2013)
Chen, Q., Koltun, V.: A simple model for intrinsic image decomposition with depth cues. In: ICCV, pp. 241–248 (2013)
Cho, H., Lee, H., Kang, H., Lee, S.: Bilateral texture filtering. ACM TOG 33(4), 1–8 (2014)
Chuang, Y.Y., Goldman, D.B., Curless, B., Salesin, D.H., Szeliski, R.: Shadow matting and compositing. In: ACM SIGGRAPH, pp. 494–500 (2003)
Cucchiara, R., Grana, C., Piccardi, M., Prati, A., Sirotti, S.: Improving shadow suppression in moving object detection with HSV color information. In: Intelligent Transportation Systems, pp. 334–339 (2002)
Cun, X., Pun, C.M., Shi, C.: Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. AAAI (2020)
Ding, B., Long, C., Zhang, L., Xiao, C.: Argan: attentive recurrent generative adversarial network for shadow detection and removal. In: ICCV, pp. 10213–10222 (2019)
Eh, L., Jj, M.: Lightness and retinex theory. J. Opt. Soc. Am. 61(1), 1–11 (1971)
Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE TPAMI 28(1), 59–68 (2006)
Godard, C., Aodha, O.M., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: CVPR, pp. 6602–6611 (2017)
Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: ICCV, pp. 3828–3838 (2019)
Gryka, M., Terry, M., Brostow, G.J.: Learning to remove soft shadows. ACM TOG 34(5), 1–15 (2015)
Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR, pp. 2033–2040 (2011)
Hachama, M., Ghanem, B., Wonka, P.: Intrinsic scene decomposition from rgb-d images. In: ICCV, pp. 810–818 (2015)
Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. In: IEEE TPAMI, pp. 7454–7462 (2019)
Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-shadowgan: learning to remove shadows from unpaired data. In: ICCV, pp. 2472–2481 (2019)
Jeon, J., Cho, S., Tong, X., Lee, S.: Intrinsic image decomposition using structure-texture separation and surface normals. In: ECCV, pp. 218–233 (2014)
Karsch, K., Sunkavalli, K., Hadap, S., Carr, N., Jin, H., Fonte, R., Sittig, M., Forsyth, D.: Automatic scene inference for 3d object compositing. ACM TOG 33(3), 1–15 (2014)
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: Fourth International Conference on 3D Vision, pp. 239–248 (2016)
Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Detecting ground shadows in outdoor consumer photographs. Lect. Notes Comput. Sci. 6312, 322–335 (2010)
Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: ICCV, pp. 8578–8587 (2019)
Le, H., Vicente, Y., Tomas, F., Nguyen, V., Hoai, M., Samaras, D.: A+ D net: training a shadow detector with adversarial shadow attenuation. In: ECCV, pp. 662–678 (2018)
Li, Z., Snavely, N.: Learning intrinsic image decomposition from watching the world. In: CVPR, pp. 9039–9048 (2018)
Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: ECCV, pp. 437–450 (2008)
Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimation from a single image. In: CVPR, pp. 5162–5170 (2015)
Meka, A., Richardt, C., Theobalt, C.: Live intrinsic video. ACM TOG 35(4), 109 (2016)
Mikic, I., Cosman, P.C., Kogut, G.T., Trivedi, M.M.: Moving shadow and object detection in traffic scenes. In: IEEE TPAMI, vol. 1, pp. 321–324 (2000)
Mohan, A., Tumblin, J., Choudhury, P.: Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27(2), 23–31 (2007)
Nguyen, V., Vicente, T.F.Y., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: ICCV, pp. 4520–4528 (2017)
Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: ICCV, pp. 1693–1700 (2009)
Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.H.: Deshadownet: a multi-context embedding deep network for shadow removal. In: CVPR, pp. 2308–2316 (2017)
Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2002)
Sidorov, O.: Conditional GANS for multi-illuminant color constancy: revolution or yet another approach? In: CVPR (2019)
Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: ECCV, pp. 816–832 (2016)
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: CVPR, pp. 1788–1797 (2018)
Wei, J., Long, C., Zou, H., Xiao, C.: Shadow inpainting and removal using generative adversarial networks with slice convolutions. Comput. Graph. Forum 38, 381–392 (2019)
Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Natural shadow matting. ACM TOG 26(2), 8 (2007)
Xiao, C., Gan, J.: Fast image dehazing using guided joint bilateral filter. Vis. Comput. 28(6–8), 713–721 (2012)
Xiao, C., She, R., Xiao, D., Ma, K.L.: Fast shadow removal using adaptive multi-scale illumination transfer. Comput. Graph. Forum 32(8), 207–218 (2013)
Xiao, Y., Tsougenis, E., Tang, C.: Shadow removal from single rgb-d images. In: CVPR, pp. 3011–3018 (2014)
Xu, D., Ricci, E., Ouyang, W., Wang, X., Sebe, N.: Multi-scale continuous crfs as sequential deep networks for monocular depth estimation, pp. 161–169 (2017)
Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM TOG 31(6), 1–10 (2012)
Zhang, L., Long, C., Zhang, X., Xiao, C.: Ris-gan: Explore residual and illumination with generative adversarial networks for shadow removal. In: AAAI (2020)
Zhang, L., Yan, Q., Liu, Z., Zou, H., Xiao, C.: Illumination decomposition for photograph with multiple light sources. IEEE Trans. Image Process. 26(9), 4114–4127 (2017)
Zhang, L., Yan, Q., Zhu, Y., Zhang, X., Xiao, C.: Effective shadow removal via multi-scale image decomposition. Vis. Comput. 35(6–8), 1091–1104 (2019)
Zhang, L., Zhang, Q., Xiao, C.: Shadow remover: image shadow removal based on illumination recovering optimization. IEEE Trans. Image Process. 24(11), 4623–4636 (2015)
Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: CVPR, pp. 5167–5176 (2019)
Acknowledgements
Funding was provided by the Key Technological Innovation Projects of Hubei Province (Grant No. 2018AAA062), NSFC (Grant Nos. 61972298, 61672390, 61902286), National Key Research and Development Program of China (Grant No. 2017YFB1002600), China Postdoctoral Science Found (No. 2018M642933), and Wuhan University - Huawei GeoInformatics Innovation Lab.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Fan, X., Wu, W., Zhang, L. et al. Shading-aware shadow detection and removal from a single image. Vis Comput 36, 2175–2188 (2020). https://doi.org/10.1007/s00371-020-01916-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-020-01916-3