Interactive removal and ground truth for difficult shadow scenes

Han Gong; Darren Cosker

doi:10.1364/JOSAA.33.001798

Journal of the Optical Society of America A
Vol. 33,
Issue 9,
pp. 1798-1811
(2016)
•https://doi.org/10.1364/JOSAA.33.001798

Interactive removal and ground truth for difficult shadow scenes

Han Gong and Darren Cosker

Not Accessible

Your library or personal account may give you access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Han Gong and Darren Cosker, "Interactive removal and ground truth for difficult shadow scenes," J. Opt. Soc. Am. A 33, 1798-1811 (2016)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Abstract

A user-centric method for fast, interactive, robust, and high-quality shadow removal is presented. Our algorithm can perform detection and removal in a range of difficult cases, such as highly textured and colored shadows. To perform detection, an on-the-fly learning approach is adopted guided by two rough user inputs for the pixels of the shadow and the lit area. After detection, shadow removal is performed by registering the penumbra to a normalized frame, which allows us efficient estimation of nonuniform shadow illumination changes, resulting in accurate and robust removal. Another major contribution of this work is the first validated and multiscene category ground truth for shadow removal algorithms. This data set containing 186 images eliminates inconsistencies between shadow and shadow-free images and provides a range of different shadow types such as soft, textured, colored, and broken shadow. Using this data, the most thorough comparison of state-of-the-art shadow removal methods to date is performed, showing our proposed algorithm to outperform the state of the art across several measures and shadow categories. To complement our data set, an online shadow removal benchmark website is also presented to encourage future open comparisons in this challenging field of research.

Full Article | PDF Article

More Like This

Pixel-wise orthogonal decomposition for color illumination invariant and shadow-free image

Liangqiong Qu, Jiandong Tian, Zhi Han, and Yandong Tang
Opt. Express 23(3) 2220-2239 (2015)

Detection and removal of fence occlusions in an image using a video of the static/dynamic scene

Sankaraganesh Jonna, Krishna K. Nakka, Vrushali S. Khasare, Rajiv R. Sahay, and Mohan S. Kankanhalli
J. Opt. Soc. Am. A 33(10) 1917-1930 (2016)

Noise-bound method for detecting shadow-free scene changes in image sequences

Kenji Irie, Alan McKinnon, Keith Unsworth, and Ian Woodhead
J. Opt. Soc. Am. A 27(2) 167-173 (2010)

Previous Article Next Article

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (11)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (7)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (10)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Feature Comparison of Shadow Removal Methods^a

(a) Automatic Shadow Removal
	Preservation
	Illum.	Text.	Color Correction	Special Setup
Finlayson et al. [8]	No	No	No	None
Fredembach & Finlayson [9]	No	No	No	None
Yang et al. [10]	No	Yes	No	None
Guo et al. [2]	Yes	Yes	No	None
Drew [11]	No	Yes	No	Flash
Salamati et al. [12]	Yes	Yes	No	NIR/RGB
Khan et al. [3]	Yes	Yes	No	None
(b) User-Aided Shadow Removal
		Preservation
	User Input	Illum.	Text.	Color Correction
Wu et al. [13]	multiple strokes for each shadow	Yes	Yes	No
Liu & Gleicher [5]	precise boundary mark for each shadow	Yes	Yes	Yes
Shor & Lischinski [6]	single click for each shadow	Yes	No	No
Su & Chen [4]	rough boundary mark for each shadow	No	Yes	No
Arbel & Hel-Or [1]	multiple clicks for each shadow	Yes	Yes	Yes
Gong et al. [14]	single stroke for each shadow	Yes	Yes	Yes
Xiao et al. [15]	two types of scribbles for sampling shadow and lit intensities	Yes	Yes	Yes
Zhang et al. [16]	two types scribbles for sampling shadow and lit intensities	Yes	Yes	Yes

^a “Illumination preserving” refers to the ability to preserve the original illumination in the lit area. “Texture preserving” refers to the preservation of the correct surface texture under the penumbra after removal. “Color correction” refers to the ability to correct color artifacts caused by image post-processing after removal.

Parameter Learning Specification for the Optimization

ID	Description	Value Range	Initial Value	Type
$h_{1}$	Gaussian filter Kernel size (§3.A)	[2, 15]	5	Int.
$h_{2}$	medium filter Kernel size (§3.A)	[2, 15]	10	Int.
$h_{3}$	DBScan Radius (§3.B)	[0.01, 0.5]	0.2	Real
$h_{4}$	meanshift Radius (§3.B)	[0.01, 0.5]	0.06	Real
$h_{5}$	gradient advance scale (Alg. 1)	[2, 20]	10	Real
$h_{6}$	bilateral filter sampling spatial (Alg. 2)	[0.1, 0.3]	0.2	Real

Parameter Learning Results for Individual Error Measurement

	Texture			Softness			Brokeness			Colorfulness
ID	Weak	Medium	Strong	Weak	Medium	Strong	Weak	Medium	Strong	Weak	Medium	Strong	Other
$h_{1}$	13	14	15	13	12	15	14	15	3	9	14	14	14
$h_{2}$	3	7	12	3	15	15	7	3	10	12	15	15	6
$h_{3}$	0.0269	0.3644	0.4418	0.1124	0.0193	0.0304	0.1815	0.0319	0.2733	0.2317	0.0304	0.0352	0.2775
$h_{4}$	0.0680	0.0193	0.0149	0.1049	0.1359	0.017	0.0426	0.0291	0.0203	0.0330	0.3464	0.0382	0.0307
$h_{5}$	5.3380	6.1115	2.2695	11.1833	9.5072	18.2173	8.5195	8.2637	5.0299	19.6150	19.8436	10.4065	7.6485
$h_{6}$	0.2274	0.1582	0.1547	0.2496	0.2392	0.2199	0.2204	0.2367	0.1658	0.2457	0.1612	0.2228	0.2360

Shadow Removal Errors for Test Cases According to Four Attributes^a

	All Pixels Errors					Shadow Pixels Errors
Degree	Zhang et al. [16]	Guo et al. [2]	Gong et al. [14]	Su & Chen [4]	Ours	Zhang et al. [16]	Guo et al. [2]	Gong [14]	Su [4]	Ours
	Textureness
Weak	0.35 (0.17)	0.53 (0.50)	0.32 (0.19)	0.35 (0.24)	0.26 (0.16)	0.16 (0.20)	0.42 (0.57)	0.16 (0.18)	0.17 (0.34)	0.10 (0.09)
Medium	0.39 (0.25)	0.59 (1.09)	0.38 (0.33)	0.36 (0.16)	0.26 (0.11)	0.28 (0.25)	0.47 (1.15)	0.27 (0.35)	0.21 (0.27)	0.12 (0.09)
Strong	0.58 (0.38)	0.71 (0.60)	0.70 (0.42)	0.60 (0.41)	0.49 (0.40)	0.39 (0.50)	0.64 (1.03)	0.65 (0.55)	0.49 (0.60)	0.36 (0.44)
Mean	0.44 (0.27)	0.61 (0.73)	0.47 (0.31)	0.43 (0.27)	0.34 (0.22)	0.27 (0.38)	0.51 (0.92)	0.36 (0.36)	0.29 (0.40)	0.19 (0.21)
	Softness
Weak	0.37 (0.24)	0.52 (1.08)	0.33 (0.31)	0.33 (0.21)	0.23 (0.10)	0.24 (0.42)	0.39 (1.13)	0.21 (0.32)	0.18 (0.33)	0.10 (0.09)
Medium	0.40 (0.20)	0.70 (0.36)	0.44 (0.21)	0.42 (0.11)	0.34 (0.15)	0.25 (0.26)	0.64 (0.43)	0.29 (0.25)	0.24 (0.14)	0.15 (0.10)
Strong	0.69 (0.49)	1.09 (0.75)	0.76 (0.35)	0.72 (0.21)	0.60 (0.27)	0.49 (0.62)	1.01 (0.97)	0.71 (0.53)	0.69 (0.54)	0.40 (0.25)
Mean	0.48 (0.31)	0.77 (0.73)	0.51 (0.29)	0.49 (0.18)	0.39 (0.18)	0.33 (0.43)	0.68 (0.84)	0.40 (0.37)	0.37 (0.34)	0.22 (0.15)
	Brokenness
Weak	0.37 (0.23)	0.59 (0.98)	0.36 (0.29)	0.34 (0.15)	0.25 (0.13)	0.24 (0.40)	0.48 (1.04)	0.23 (0.31)	0.18 (0.24)	0.11 (0.09)
Medium	0.43 (0.22)	0.42 (0.29)	0.44 (0.25)	0.49 (0.38)	0.29 (0.14)	0.27 (0.27)	0.27 (0.35)	0.29 (0.28)	0.38 (0.58)	0.14 (0.11)
Strong	1.07 (0.47)	1.42 (1.06)	0.98 (0.31))	0.85 (0.25)	0.69 (0.30)	0.88 (0.72)	1.55 (1.84)	1.05 (0.50)	0.86 (0.65)	0.52 (0.32)
Mean	0.63 (0.31)	0.81 (0.78)	0.59 (0.29)	0.56 (0.26)	0.41 (0.19)	0.46 (0.46)	0.76 (1.08)	0.52 (0.36)	0.47 (0.49)	0.26 (0.17)
	Colorfulness
Weak	0.36 (0.18)	0.48 (0.64)	0.32 (0.18)	0.34 (0.18)	0.24 (0.11)	0.21 (0.24)	0.36 (0.78)	0.19 (0.20)	0.18 (0.23)	0.10 (0.08)
Medium	0.60 (0.50)	1.67 (2.29)	0.83 (0.67)	0.52 (0.24)	0.48 (0.18)	0.57 (1.06)	1.56 (2.07)	0.67 (0.73)	0.45 (0.65)	0.24 (0.14)
Strong	0.78 (0.57)	1.20 (0.99)	1.10 (0.68)	0.63 (0.49)	0.56 (0.31)	0.72 (1.00)	1.34 (2.33)	1.20 (1.18)	0.54 (0.84)	0.46 (0.48)
Mean	0.58 (0.41)	1.12 (1.31)	0.75 (0.51)	0.50 (0.30)	0.43 (0.20)	0.50 (0.77)	1.09 (1.73)	0.69 (0.70)	0.39 (0.57)	0.27 (0.23)
	Other
	0.35 (0.16)	0.38 (0.52)	0.27 (0.17)	0.26 (0.08)	0.19 (0.06)	0.16 (0.22)	0.25 (0.58)	0.14 (0.20)	0.09 (0.05)	0.06 (0.02)

^a The left and right sides of the table show the error scores where all pixels in the image are used, and just shadow area pixels, respectively. For each score of each attribute, the images with other predominant attributes (strong) are not used. Hence, test cases have a strong single bias towards one of the attributes. “Other” refers to a set of shadow cases showing no markedly predominant attributes. “Mean” refers to the average score for each category. Standard derivations are shown in brackets. In our ordering, the average error is compared before comparing the standard derivation. Method [2] is trained using a large shadow detection data set from [19]. The user input for method [16] is a combination of the simple input for our method and some additional strokes for accommodating the sensitive shadow detection of [16]. The best scores are made bold.

Time Comparison of Shadow Removal^a

Degree	Zhang et al. [16]		Guo et al. [2]		Gong et al. [14]		Su [4]		Ours
	MATLAB+C		MATLAB		MATLAB		C++		MATLAB
	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$
Textureness
Medium	6	40	0	157	2	35	22	44	4	7
Strong	4	38	0	74	2	81	20	44	4	10
Softness
Medium	6	35	0	159	4	267	18	44	4	13
Strong	7	33	0	23	9	31	109	43	3	21
Brokenness
Medium	6	35	0	61	8	88	37	43	4	14
Strong	8	35	0	158	10	241	61	51	8	54
Colorfulness
Medium	6	34	0	160	3	385	30	45	5	51
Strong	7	35	0	56	2	132	19	46	5	11
Other
	6	35	0	74	4	301	33	42	4	17
	6	34	0	80	2	114	28	42	4	13

T_{u}

T_{s}

refer to time (in seconds) for user interaction and time for system processing, respectively.

Table 1.

Feature Comparison of Shadow Removal Methods^a

(a) Automatic Shadow Removal
	Preservation
	Illum.	Text.	Color Correction	Special Setup
Finlayson et al. [8]	No	No	No	None
Fredembach & Finlayson [9]	No	No	No	None
Yang et al. [10]	No	Yes	No	None
Guo et al. [2]	Yes	Yes	No	None
Drew [11]	No	Yes	No	Flash
Salamati et al. [12]	Yes	Yes	No	NIR/RGB
Khan et al. [3]	Yes	Yes	No	None
(b) User-Aided Shadow Removal
		Preservation
	User Input	Illum.	Text.	Color Correction
Wu et al. [13]	multiple strokes for each shadow	Yes	Yes	No
Liu & Gleicher [5]	precise boundary mark for each shadow	Yes	Yes	Yes
Shor & Lischinski [6]	single click for each shadow	Yes	No	No
Su & Chen [4]	rough boundary mark for each shadow	No	Yes	No
Arbel & Hel-Or [1]	multiple clicks for each shadow	Yes	Yes	Yes
Gong et al. [14]	single stroke for each shadow	Yes	Yes	Yes
Xiao et al. [15]	two types of scribbles for sampling shadow and lit intensities	Yes	Yes	Yes
Zhang et al. [16]	two types scribbles for sampling shadow and lit intensities	Yes	Yes	Yes

Algorithm 1

Penumbra Sample End Point Selection

input: boundary point

(x_{b}, y_{b})

, fusion image

F

output: two ends

(p_{s}, p_{e})

of a sampling line

\tilde{F} \leftarrow \nabla F

;

p_{s} \leftarrow (x_{b}, y_{b})

;

p_{e} \leftarrow (x_{b}, y_{b})

;

L \leftarrow | \tilde{F} (x_{b}, y_{b}) |

;

Δ v \leftarrow \tilde{F} (x_{b}, y_{b}) / L

;

repeat

v_{s} \leftarrow \tilde{F} ([p_{s}])

;

v_{e} \leftarrow \tilde{F} ([p_{e}])

;

L_{s} \leftarrow v_{s} \cdot Δ v

;

L_{e} \leftarrow v_{e} \cdot Δ v

;

p_{s} \leftarrow p_{s} - Δ v

;

p_{e} \leftarrow p_{e} + Δ v

;

until

p_{s}

p_{e}

is not within the range of

F

h_{5} L_{s} > L

h_{5} L_{e} < L

;

Algorithm 2

Multi-Scale Color Correction

Input: shadow removed image

I^{r}

, reference lit pixels

P_{l}

, source shadow pixels

P_{u}

, all shadow pixels

P_{s}

output: color corrected image

I_{c}^{r a}

I^{r a} \leftarrow I^{r}

;

for

s = 1

to 3 do for each scale (e.g., Fig. 6)

I^{l} \leftarrow bfilter (I^{r a}, β / 2^{s + 1}, h_{6})

// apply bilateral filtering [e.g., Fig. 6(c2)]

I^{h} \leftarrow I^{r} - I^{l}

// get local intensity variation image [e.g., Fig. 6(c3)]

for

c = 1

to 3 do for each RGB channels

r_{σ} \leftarrow ς (I_{c}^{h} (P_{l})) / ς (I_{c}^{h} (P_{u}))

// get overall ratio of intensity variation

I_{c}^{r e} \leftarrow r_{σ} I^{h} (P_{s})

// get aligned intensity variation image [e.g., Fig. 6(c4)]

I_{c}^{r a} \leftarrow I^{r}

// copy intensities of lit pixels

I_{c}^{r a} (P_{s}) \leftarrow I_{c}^{l} (P_{s}) + I_{c}^{r e}

// add aligned intensity variation back [e.g., Fig. 6(c5)]

end

Table 2.

Parameter Learning Specification for the Optimization

ID	Description	Value Range	Initial Value	Type
$h_{1}$	Gaussian filter Kernel size (§3.A)	[2, 15]	5	Int.
$h_{2}$	medium filter Kernel size (§3.A)	[2, 15]	10	Int.
$h_{3}$	DBScan Radius (§3.B)	[0.01, 0.5]	0.2	Real
$h_{4}$	meanshift Radius (§3.B)	[0.01, 0.5]	0.06	Real
$h_{5}$	gradient advance scale (Alg. 1)	[2, 20]	10	Real
$h_{6}$	bilateral filter sampling spatial (Alg. 2)	[0.1, 0.3]	0.2	Real

Table 3.

Parameter Learning Results for Individual Error Measurement

	Texture			Softness			Brokeness			Colorfulness
ID	Weak	Medium	Strong	Weak	Medium	Strong	Weak	Medium	Strong	Weak	Medium	Strong	Other
$h_{1}$	13	14	15	13	12	15	14	15	3	9	14	14	14
$h_{2}$	3	7	12	3	15	15	7	3	10	12	15	15	6
$h_{3}$	0.0269	0.3644	0.4418	0.1124	0.0193	0.0304	0.1815	0.0319	0.2733	0.2317	0.0304	0.0352	0.2775
$h_{4}$	0.0680	0.0193	0.0149	0.1049	0.1359	0.017	0.0426	0.0291	0.0203	0.0330	0.3464	0.0382	0.0307
$h_{5}$	5.3380	6.1115	2.2695	11.1833	9.5072	18.2173	8.5195	8.2637	5.0299	19.6150	19.8436	10.4065	7.6485
$h_{6}$	0.2274	0.1582	0.1547	0.2496	0.2392	0.2199	0.2204	0.2367	0.1658	0.2457	0.1612	0.2228	0.2360

Table 4.

Shadow Removal Errors for Test Cases According to Four Attributes^a

	All Pixels Errors					Shadow Pixels Errors
Degree	Zhang et al. [16]	Guo et al. [2]	Gong et al. [14]	Su & Chen [4]	Ours	Zhang et al. [16]	Guo et al. [2]	Gong [14]	Su [4]	Ours
	Textureness
Weak	0.35 (0.17)	0.53 (0.50)	0.32 (0.19)	0.35 (0.24)	0.26 (0.16)	0.16 (0.20)	0.42 (0.57)	0.16 (0.18)	0.17 (0.34)	0.10 (0.09)
Medium	0.39 (0.25)	0.59 (1.09)	0.38 (0.33)	0.36 (0.16)	0.26 (0.11)	0.28 (0.25)	0.47 (1.15)	0.27 (0.35)	0.21 (0.27)	0.12 (0.09)
Strong	0.58 (0.38)	0.71 (0.60)	0.70 (0.42)	0.60 (0.41)	0.49 (0.40)	0.39 (0.50)	0.64 (1.03)	0.65 (0.55)	0.49 (0.60)	0.36 (0.44)
Mean	0.44 (0.27)	0.61 (0.73)	0.47 (0.31)	0.43 (0.27)	0.34 (0.22)	0.27 (0.38)	0.51 (0.92)	0.36 (0.36)	0.29 (0.40)	0.19 (0.21)
	Softness
Weak	0.37 (0.24)	0.52 (1.08)	0.33 (0.31)	0.33 (0.21)	0.23 (0.10)	0.24 (0.42)	0.39 (1.13)	0.21 (0.32)	0.18 (0.33)	0.10 (0.09)
Medium	0.40 (0.20)	0.70 (0.36)	0.44 (0.21)	0.42 (0.11)	0.34 (0.15)	0.25 (0.26)	0.64 (0.43)	0.29 (0.25)	0.24 (0.14)	0.15 (0.10)
Strong	0.69 (0.49)	1.09 (0.75)	0.76 (0.35)	0.72 (0.21)	0.60 (0.27)	0.49 (0.62)	1.01 (0.97)	0.71 (0.53)	0.69 (0.54)	0.40 (0.25)
Mean	0.48 (0.31)	0.77 (0.73)	0.51 (0.29)	0.49 (0.18)	0.39 (0.18)	0.33 (0.43)	0.68 (0.84)	0.40 (0.37)	0.37 (0.34)	0.22 (0.15)
	Brokenness
Weak	0.37 (0.23)	0.59 (0.98)	0.36 (0.29)	0.34 (0.15)	0.25 (0.13)	0.24 (0.40)	0.48 (1.04)	0.23 (0.31)	0.18 (0.24)	0.11 (0.09)
Medium	0.43 (0.22)	0.42 (0.29)	0.44 (0.25)	0.49 (0.38)	0.29 (0.14)	0.27 (0.27)	0.27 (0.35)	0.29 (0.28)	0.38 (0.58)	0.14 (0.11)
Strong	1.07 (0.47)	1.42 (1.06)	0.98 (0.31))	0.85 (0.25)	0.69 (0.30)	0.88 (0.72)	1.55 (1.84)	1.05 (0.50)	0.86 (0.65)	0.52 (0.32)
Mean	0.63 (0.31)	0.81 (0.78)	0.59 (0.29)	0.56 (0.26)	0.41 (0.19)	0.46 (0.46)	0.76 (1.08)	0.52 (0.36)	0.47 (0.49)	0.26 (0.17)
	Colorfulness
Weak	0.36 (0.18)	0.48 (0.64)	0.32 (0.18)	0.34 (0.18)	0.24 (0.11)	0.21 (0.24)	0.36 (0.78)	0.19 (0.20)	0.18 (0.23)	0.10 (0.08)
Medium	0.60 (0.50)	1.67 (2.29)	0.83 (0.67)	0.52 (0.24)	0.48 (0.18)	0.57 (1.06)	1.56 (2.07)	0.67 (0.73)	0.45 (0.65)	0.24 (0.14)
Strong	0.78 (0.57)	1.20 (0.99)	1.10 (0.68)	0.63 (0.49)	0.56 (0.31)	0.72 (1.00)	1.34 (2.33)	1.20 (1.18)	0.54 (0.84)	0.46 (0.48)
Mean	0.58 (0.41)	1.12 (1.31)	0.75 (0.51)	0.50 (0.30)	0.43 (0.20)	0.50 (0.77)	1.09 (1.73)	0.69 (0.70)	0.39 (0.57)	0.27 (0.23)
	Other
	0.35 (0.16)	0.38 (0.52)	0.27 (0.17)	0.26 (0.08)	0.19 (0.06)	0.16 (0.22)	0.25 (0.58)	0.14 (0.20)	0.09 (0.05)	0.06 (0.02)

Table 5.

Time Comparison of Shadow Removal^a

Degree	Zhang et al. [16]		Guo et al. [2]		Gong et al. [14]		Su [4]		Ours
	MATLAB+C		MATLAB		MATLAB		C++		MATLAB
	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$	$T_{u}$	$T_{s}$
Textureness
Medium	6	40	0	157	2	35	22	44	4	7
Strong	4	38	0	74	2	81	20	44	4	10
Softness
Medium	6	35	0	159	4	267	18	44	4	13
Strong	7	33	0	23	9	31	109	43	3	21
Brokenness
Medium	6	35	0	61	8	88	37	43	4	14
Strong	8	35	0	158	10	241	61	51	8	54
Colorfulness
Medium	6	34	0	160	3	385	30	45	5	51
Strong	7	35	0	56	2	132	19	46	5	11
Other
	6	35	0	74	4	301	33	42	4	17
	6	34	0	80	2	114	28	42	4	13

T_{u}

T_{s}

refer to time (in seconds) for user interaction and time for system processing, respectively.

Interactive removal and ground truth for difficult shadow scenes

Author Affiliations

Abstract

Cited By

Figures (11)

Tables (7)

Equations (10)

Journal of the Optical Society of America A