DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal
Using ViT Similarity

Abstract

Removing soft and self shadows that lack clear boundaries from a single image is still challenging. Self shadows are shadows that are cast on the object itself. Most existing methods rely on binary shadow masks, without considering the ambiguous boundaries of soft and self shadows. In this paper, we present DeS3, a method that removes hard, soft and self shadows based on adaptive attention and ViT similarity. Our novel ViT similarity loss utilizes features extracted from a pre-trained Vision Transformer. This loss helps guide the reverse sampling towards recovering scene structures. Our adaptive attention is able to differentiate shadow regions from the underlying objects, as well as shadow regions from the object casting the shadow. This capability enables DeS3 to better recover the structures of objects even when they are partially occluded by shadows. Different from existing methods that rely on constraints during the training phase, we incorporate the ViT similarity during the sampling stage. Our method outperforms state-of-the-art methods on the SRD, AISTD, LRSS, USR and UIUC datasets, removing hard, soft, and self shadows robustly. Specifically, our method outperforms the SOTA method by 16% of the RMSE of the whole image on the LRSS dataset.

Introduction

Shadows can be categorized into hard shadows, soft shadows, and self shadows (Salvador, Cavallaro, and Ebrahimi 2004; Huang and Chen 2009). When an object blocks beams of light, shadows are formed on surfaces or nearby objects, which are referred to as hard and soft shadows. Hard shadows have sharp boundaries, while soft shadows gradually transition from shadow to non-shadow regions without any distinct boundaries. Self shadows occur when a portion of an object obstructs beams of light, resulting in shadows being cast on the object itself. Removing all types of shadows is intractable due to ambiguities between shadow and non-shadow regions. Soft and self shadows are particularly challenging to identify and remove compared to hard shadows.

Figure 1: The results of SOTA supervised method (Wan et al. 2022) and weakly-supervised method (Liu et al. 2021b) in removing (a) self shadow, (b) soft shadow, and (c) hard shadow. Our DeS3 can preserve meaningful objects (duck, paper, bollard, etc.) during the reverse sampling, and achieve better shadow removal results.

To remove shadows, most of the methods use an off-the-shelf shadow detection method (Zhu et al. 2018) or user interaction (Gryka, Terry, and Brostow 2015) to obtain binary shadow masks. However, these binary shadow masks are problematic to acquire, particularly for soft and self shadows as shown in Fig. 3. Moreover, current shadow removal methods fail to handle self shadow images since obtaining the ground truth for outdoor self shadows is intractable.

The weakly-supervised methods (Le and Samaras 2020; Liu et al. 2021b) require binary shadow masks to distinguish shadow and shadow-free patches. These patches from the same images can reduce the domain gap of shadow and shadow-free domains. However, cropping patches is time-consuming (Liu et al. 2021b) and synthesizing shadows on the shadow-free regions cannot represent real-world shadow images. Moreover, these weakly-supervised methods require shadows to be homogeneous, which is often not the case, particularly for self shadows and soft shadows.

Existing unsupervised methods (Hu et al. 2019b; Liu et al. 2021a; Jin, Sharma, and Tan 2021) are GAN-based (Zhu et al. 2017), which require unpaired shadow and shadow-free images for training. They largely rely on the statistical similarity between images from the two domains (Le and Samaras 2020) (shadow and shadow-free domains). Unfortunately, once the two domains are statistically different, these methods produce hallucination/fake contents (Zhu et al. 2018) and suffer from unstable training.

Most SOTA methods rely on masks, but obtaining masks for self and soft shadows is intractable. Fixed attention in (Jin, Sharma, and Tan 2021) for self-shadows are unreliable. Hence, we propose an adaptive attention to progressively focus on self-shadows. For this reason, we employ the diffusion. Unlike other generative models, diffusion works progressively.

In this paper, we introduce DeS3, a diffusion-based method that removes hard, soft, and self shadows from a single image using adaptive attention and ViT similarity. “De” means “DeShadow”, “S3” means three types of shadows. Each step in the diffusion process brings about changes in our adaptive attention, as shown in Fig. 4, enabling DeS3 to effectively eliminate self and soft shadows that lack clear boundaries. Moreover, using the ViT similarity, our DeS3 can preserve object structures even when the object is partially occluded by self and soft shadows. To guide the reverse sampling in estimating the structural features, our basic idea is to inject the pretrained features from DINO-ViT (Caron et al. 2021). Specifically, since the keys’ self-similarity (Shechtman and Irani 2007) contains object structures (Tumanyan et al. 2022), they are utilized to compute the ViT similarity loss between the denoised samples and the input shadow condition. As shown in Fig. 1, our method can remove shadows and retain various objects (duck, paper, bollard, etc.).

In summary, we make the following contributions:

1. We introduce DeS3, the first shadow removal network, that performs shadow removal robustly on hard, soft and self shadows from a single image.

2. Our DeS3 does not require shadow masks from the dataset or a shadow detector. Through the adaptive attention in the progressive diffusion process, DeS3 learns to adjust to all types of shadow regions, particularly self-shadow regions.

3. To maintain object structure features, we integrate the ViT similarity loss into the reverse sampling. ViT’s features are independent from shadows and thus can extract structures more robustly.

Comprehensive experiments on the SRD, AISTD, LRSS, UIUC and USR datasets demonstrate that DeS3 outperforms the state-of-the-art methods, particularly on self and soft shadows. Our DeS3 outperforms ShadowDiffusion (Guo et al. 2023b) by 16% error reduction (from 3.49 to 3.01) in terms of RMSE on the overall areas of the LRSS dataset.

Related Work

Different image priors have been explored for single image shadow removal, e.g., modeling of illumination and color (Finlayson et al. 2005; Jin et al. 2023a), image regions (Guo, Dai, and Hoiem 2012; Vicente, Hoai, and Samaras 2017; Jin et al. 2022), image gradients (Gryka, Terry, and Brostow 2015; Jin et al. 2023b). However, traditional shadow removal methods may produce unsatisfactory results. Recently, supervised learning-based shadow removal methods (Chen et al. 2021; Fu et al. 2021; Liu et al. 2023a, b) have shown promising performance. DeshadowNet (Qu et al. 2017) removes shadows in an end-to-end manner. DSC (Hu et al. 2019a) captures global and context information from the direction-aware spatial attention module. SP+M-Net (Le and Samaras 2019) and SP+M+I-Net (Le and Samaras 2021) remove shadow using image decomposition. CANet (Chen et al. 2021) is a two-stage context-aware network. BMNet (Zhu et al. 2022a) removes shadows using invertible neural networks. ST-CGAN (Wang, Li, and Yang 2018) jointly detects and removes shadows. ARGAN (Ding et al. 2019) uses LSTM attention to detect shadows. DHAN (Cun, Pun, and Shi 2020) employs the SMGAN to generate shadow matting. RIS-GAN (Zhang et al. 2020) explores the relationship of the residual images. SG-ShadowNet (Wan et al. 2022) treats shadow removal as intra-image style transfer. These methods use Conditional GAN, StyleGAN (Karras, Laine, and Aila 2019) or Uformer (Guo et al. 2023a; Wang et al. 2022) as their architecture. However, supervised methods fail to handle self-shadows, as there are no ground truths (creating ones is intractable).

Figure 2: The architecture and the motivation of our DeS3. (1) The forward diffusion is shown in green. The reverse sampling starts from the noise map $\operatorname{\mathbf{x}}_{T}$ concatenated with the conditional shadow inputs $\operatorname{\tilde{\mathbf{x}}}$. Our DeS3 samples image $\mathbf{x}_{t}$ at each time step $t$. (2) We inject a classifier into the noise prediction network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},\mathbf{a}_{t},t)$. Adaptive attention $\mathbf{a}_{t}$ is progressively improved at each time step $t$. (3) To guide the reverse sampling to output the object structure features, we have the ViT similarity loss $\mathcal{L}_{\rm sim}$, extracted keys from the pre-trained DINO-ViT.

The weakly-supervised shadow removal methods (Le and Samaras 2020; Liu et al. 2021a) require shadow masks to distinguish shadow and non-shadow patches or to produce pseudo shadow pairs, making them unsuitable for handling soft or self-shadows. Unsupervised shadow removal methods (Hu et al. 2019b; Liu et al. 2021a; Jin, Sharma, and Tan 2021) need non-shadow reference images, which are hard to obtain if the input has self-shadows. They rely on CycleGAN (Zhu et al. 2017; Jin, Yang, and Tan 2022) and learn from unpaired data.

Denoising Diffusion Probabilistic Models (DDPM) (Ho, Jain, and Abbeel 2020) and Denoising Diffusion Implicit Models (DDIM) (Song, Meng, and Ermon 2021) have recently demonstrated promising generative ability (Li et al. 2023). Palette (Saharia et al. 2022a), Unit-DDPM (Sasaki, Willcocks, and Breckon 2021), RDDM (Liu et al. 2023c) were proposed for image-to-image translation. WeatherDiffusion (Özdenizci and Legenstein 2023) is a diffusion method presented for weather removal. However, they are not designed to remove shadows and lack object structures (Preechakul et al. 2022). Shadowdiffusion (Guo et al. 2023b) uses shadow degradation as a prior and unrolling diffusion to remove shadows. However, the degradation model’s accuracy depends on the mask, and hence this method may not work well for soft and self-shadow removal.

Proposed Method: DeS3

Figure 3: Shadows can be categorized into hard, soft, and self shadows. Left: Soft and self shadows. Right: Wrong binary masks, due to the ambiguous boundaries (the red boxes for the incorrect and blue boxes for mis-detected masks).

Fig. 2 shows our pipeline, comprising a forward diffusion (dashed line) and reverse sampling (solid line). The main goal of DeS3 is to remove hard, soft and self shadows from a single image using an end-to-end network (without masks).

Figure 4: Adaptive attention is refined during inference.

Figure 5: The results of adaptive attention that enables our reverse sampling to focus on hard, soft and self shadows.

(a) a

(b) b

(c) c

(d) d

(e) e

(f) f

Figure 6: d is DC (2021)’s result, and f is ours. We integrate adaptive attention with diffusion to tackle self-shadows.

Conditional DDIM

We use Denoising Diffusion Implicit Models (DDIM) (Song, Meng, and Ermon 2021) as our generative model. DDIM is trained on a clean image $\operatorname{\mathbf{x}}_{0}\sim q(\operatorname{\mathbf{x}}_{0})$ via a forward diffusion process $q(\operatorname{\mathbf{x}}_{t}|\operatorname{\mathbf{x}}_{t-1})$ that sequentially adds the Gaussian noise at every time steps $t$,  i.e. , $q(\operatorname{\mathbf{x}}_{t}|\operatorname{\mathbf{x}}_{t-1})=\mathcal{N}(\operatorname{\mathbf{x}}_{t};\sqrt{1-\beta_{t}}\operatorname{\mathbf{x}}_{t-1},\beta_{t}\operatorname{\mathbf{I}})$, where $\{\beta\}_{t=0}^{T}$ is a variance schedule. The forward diffusion of length T expressed as:

$$q(\operatorname{\mathbf{x}}_{1:T}|\operatorname{\mathbf{x}}_{0})=\prod_{t=1}^{T}q(\operatorname{\mathbf{x}}_{t}|\operatorname{\mathbf{x}}_{t-1}).$$ (1)

Diffusion learns to reverse the process in Eq.(1) with the Gaussian transitions, i.e. , $p_{\theta}(\operatorname{\mathbf{x}}_{t-1}|\operatorname{\mathbf{x}}_{t})=\mathcal{N}(\operatorname{\mathbf{x}}_{t-1};\bm{\mu}_{\theta}(\operatorname{\mathbf{x}}_{t},t),\mathbf{\Sigma}_{\theta}(\operatorname{\mathbf{x}}_{t},t))$, at each time step.

The reverse denoising process is parameterized by a trainable network (e.g., U-Net), which estimates the mean $\bm{\mu}_{\theta}(\operatorname{\mathbf{x}}_{t},t)$ and variance $\mathbf{\Sigma}_{\theta}(\operatorname{\mathbf{x}}_{t},t)$ with parameter $\theta$. This reverse process starts from a standard normal distribution $p(\operatorname{\mathbf{x}}_{T})=\mathcal{N}(\operatorname{\mathbf{x}}_{T};\mathbf{0},\operatorname{\mathbf{I}})$ and follows: $p_{\theta}(\operatorname{\mathbf{x}}_{0:T})=p(\operatorname{\mathbf{x}}_{T})\prod_{t=1}^{T}p_{\theta}(\operatorname{\mathbf{x}}_{t-1}|\operatorname{\mathbf{x}}_{t}).$ The noise prediction network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},t)$ can be optimized by this objective function: $L_{\theta}(\text{DM})=\mathbb{E}_{\operatorname{\mathbf{x}}_{0},\bm{\epsilon},t}\Big{[}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},t)\|^{2}\Big{]}.$ After the optimization, we can sample from the learned parameterized Gaussian transitions $p_{\theta}(\operatorname{\mathbf{x}}_{t-1}|\operatorname{\mathbf{x}}_{t})$ by: ${\mathbf{x}}_{t-1}=\bm{\mu}_{\theta}({\mathbf{x}}_{t},t)+\mathbf{\Sigma}_{\theta}^{1/2}({\mathbf{x}}_{t},t)\bm{\epsilon},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \bm{\epsilon}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}).$ Our DeS3 use conditional DDIM to output deterministic and consistent samples.

We reverse process $p_{\theta}(\operatorname{\mathbf{x}}_{0:T}|\operatorname{\tilde{\mathbf{x}}})$, without changing the diffusion process $q(\operatorname{\mathbf{x}}_{1:T}|\operatorname{\mathbf{x}}_{0})$ in Eq.(1). We replace the noise prediction network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},t)$ with $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t)$, where $\operatorname{\tilde{\mathbf{x}}}$ denotes the shadow image. Here, we use the conditional DDIM (Song, Meng, and Ermon 2021) and input shadow images $\operatorname{\tilde{\mathbf{x}}}$ and shadow-free $\operatorname{\mathbf{x}}_{0}$ pairs $(\operatorname{\tilde{\mathbf{x}}},\operatorname{\mathbf{x}}_{0})$ to output shadow-free image. We concatenate $\operatorname{\tilde{\mathbf{x}}}$ and $\mathbf{x}_{t}$ channel-wisely, and input them to the deterministic reverse process. Therefore, we treat shadow removal as a reverse process with the loss function $L_{\theta}(\text{CDM})$ and parameterize a noise prediction network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t)$:

$$L_{\theta}(\text{CDM}) =\mathbb{E}_{\operatorname{\mathbf{x}}_{0},\bm{\epsilon},t}\Big{[}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t)\|^{2}\Big{]},$$ (2)

$$\hat{\operatorname{\mathbf{x}}}_{0}(\operatorname{\mathbf{x}}_{t}) =\frac{1}{\sqrt{{\bar{\alpha}_{t}}}}\left(\mathbf{x}_{t}-\sqrt{1-{\bar{\alpha}_{t}}}\cdot\mathbf{\bm{missing}}{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t)\right),$$

where $\hat{\operatorname{\mathbf{x}}}_{0}(\operatorname{\mathbf{x}}_{t})$ refers to the estimated clean image from the sample $\operatorname{\mathbf{x}}_{t}$. For brevity, we short $\hat{\operatorname{\mathbf{x}}}_{0}(\operatorname{\mathbf{x}}_{t})$ as $\operatorname{\mathbf{x}}$.

The shadow removal starts from the noise map, $\operatorname{\mathbf{x}}_{T}\sim\operatorname{\mathcal{N}}(\mathbf{0},\operatorname{\mathbf{I}})$, with the condition shadow input $\operatorname{\tilde{\mathbf{x}}}$, and applies the diffusion towards the target non-shadow image $\operatorname{\mathbf{x}}_{0}$. The sampling from the reverse process $\operatorname{\mathbf{x}}_{t-1}\sim p_{\theta}(\operatorname{\mathbf{x}}_{t-1}|\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}})$ employs: $\operatorname{\mathbf{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\left(\frac{\operatorname{\mathbf{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\cdot\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t)}{\sqrt{\bar{\alpha}_{t}}}\right)+\sqrt{1-\bar{\alpha}_{t-1}}\cdot\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},t).$

Adaptive Classifier-driven Attention

Our novelty lies in adaptive attention, which can highlight the regions of all three types of shadows, particularly self-shadows. Figs. 6e to 6f show our results using adaptive attention, and additional results can be found in Fig. 5. Our adaptive attention is progressively refined throughout the reverse process, as illustrated in Fig. 4. In contrast, (Jin, Sharma, and Tan 2021) relies solely on fixed CAMs, which tend to struggle with handling self-shadows, as demonstrated in Figs. 6c to 6d.

Since a plain diffusion model is not aware of the shadow regions, we design an adaptive attention to allow the reverse sampling to focus on the shadow regions. Our soft attention map differs from existing binary masks shown in Fig. 3, which are hard to obtain for self and soft shadows. We fuse the attention into diffusion by injecting the classifier in the U-Net. Specifically, by labeling the clear image $\operatorname{\mathbf{x}}_{0}$ as 0, and the shadow condition $\operatorname{\tilde{\mathbf{x}}}$ as 1, and using binary classification with Class Activation Map (CAM) (Zhou et al. 2016). Given an image $x\in\{\operatorname{\tilde{\mathbf{x}}},\operatorname{\mathbf{x}}_{0}\}$, $L_{\text{cam}}=-(\mathbb{E}_{x\sim\operatorname{\tilde{\mathbf{x}}}}[log(C(x))]+\mathbb{E}_{x\sim\operatorname{\mathbf{x}}_{0}}[log(1-C(x))])$, where $C$ is the classifier. By leveraging the information from the classifier, the CAM attention is learned from the data and can focus on hard, soft, and self-shadow regions, as shown in Fig. 5.

Unlike the fixed attention in (Jin, Sharma, and Tan 2021), our attention is adaptive and progressively refines throughout the diffusion process. Besides the CAM attention, we employ a residual map refinement. By utilizing shadow and shadow-free pairs during training, we can employ a residual map to refine classifier-driven attention progressively. That is to say, the noise estimation network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},\mathbf{a}_{t},t)$ has two tasks: estimating the noise and progressively refining the attention $\mathbf{a}_{t}$. We add one Sigmoid Conv layer after the last layer of the noise estimation network, and obtain the residual map $\mathbf{m}_{\text{res}}$ by computing the difference map between shadow and shadow-free pair. This difference map is then used to guide the refinement of $\mathbf{a}_{t}$ via the following loss:

$$L_{\text{att}} =\mathbb{E}_{t\sim[1,T]}\Big{[}\|\mathbf{a}_{t}-\mathbf{m}_{\text{res}}\|^{2}\Big{]},$$ (3)

where $\mathbf{m}_{\text{res}}=\sigma(\operatorname{\mathbf{x}}_{0}-\operatorname{\tilde{\mathbf{x}}})$, $\sigma(\cdot)$ is the Sigmoid Conv layer.

Figure 7: We visualize the deep DINO-ViT features. These object structure ViT features help DeS3 preserve the object/scene structures (e.g., fire hydrant in self shadow example, duck, bag, bear in soft shadow examples).

Figure 8: In our reverse sampling process, we use the loss function to guide the output to follow the ViT similarity. When $\mathcal{L}_{\text{s}im}$ is high, continue the reverse sampling till 25 steps. When $\mathcal{L}_{\text{s}im}$ is low, the reverse sampling can be stopped early.

Method	Train	RMSE $\downarrow$			PSNR $\uparrow$			SSIM $\uparrow$
		S	NS	ALL	S	NS	ALL	S	NS	ALL
DHAN (2020)	P+M	8.94	4.80	5.67	33.67	34.79	30.51	0.978	0.979	0.949
Auto (2021)	P+M	8.56	5.75	6.51	32.26	31.87	28.40	0.966	0.945	0.893
EM (2022b)	P+M	10.00	6.04	7.20	29.44	26.67	24.16	0.937	0.879	0.779
BM (2022a)	P+M	6.61	3.61	4.46	35.05	36.02	31.69	0.981	0.982	0.956
SG (2022)	P+M	7.53	3.08	4.33	33.88	36.43	31.38	0.981	0.987	0.959
DC (2021)	UP	7.70	3.65	4.66	34.00	35.53	31.53	0.975	0.981	0.955
SR3 (2022b)	P	-	-	-	35.44	34.35	31.29	0.980	0.970	0.946
W.Dif. (2023)	P	-	-	-	33.38	31.15	28.45	0.981	0.972	0.951
DSC (2019a)	P	8.81	4.41	5.71	30.65	31.94	27.76	0.960	0.965	0.903
Ours	P	5.88	2.83	3.72	37.45	38.12	34.11	0.984	0.988	0.968

Table 1: Quantitative results on the SRD dataset. S, NS and ALL represent shadow, non-shadow and entire regions. M shows ground truth shadow masks are also used in training. “P” and “UP” stand for “Paired” and “Unpaired”.

Figure 9: Self shadow removal results on UIUC dataset. (a) Input images. (b) Our results. (c)$\sim$(g) Results of the state-of-the-art methods. Note that our method does not need additional masks in training and testing.

Object Structures in Reverse Sampling

Unlike the VGG loss (Jin, Sharma, and Tan 2021) used in training, we propose to use a DINO-ViT loss as a stopping criterion in the reverse sampling (inference) stage. DINO-ViT demonstrates greater robustness in preserving object structures across all types of shadows when compared to VGG, as shown in Fig. 13. Moreover, in contrast to the losses in (Jin, Sharma, and Tan 2021), our DINO-ViT loss is exclusively employed during inference, serving as a stopping criterion, as depicted in Fig. 8. If the total loss $L_{t}$ is smaller than that in the subsequent diffusion step, we halt the reverse process earlier.

In contrast to the existing shadow removal methods that use the supervised losses (requiring ground truth shadow-free) in training, we use the self-tuned losses in the reverse sampling (inference). Preserving object/scene structures is critical for shadow removal (Qu et al. 2017). Therefore, we introduce the ViT similarity loss to guide the reverse sampling. To preserve the object structures, our key idea is to exploit a self-similarity prior (Shechtman and Irani 2007). For the soft shadow example in Fig. 7, the left and right parts of the bag contain different lighting conditions and appearances, but they do share similar structures (i.e., self-similarity). A pre-trained DINO-ViT (Caron et al. 2021) can provide better object structure features, compared to the CNN backbone, transformer-based backbones shown in (Karmali et al. 2022). Motivated by this, we use a pre-trained DINO-ViT as our feature extractor, enabling us to capture deep features (Amir et al. 2021), and providing useful feature supervision to our DeS3.

DINO-ViT Keys

We extract keys (deep features) from the pre-trained DINO-ViT. In the multi-head self-attention layer, the keys contain the object structure information (Tumanyan et al. 2022). In Fig. 7, we show the Principal Component Analysis (PCA) visualization of the keys’ self-similarity and demonstrate the top three components as RGB at the deepest layer of DINO-ViT. As one can observe, the feature representations capture the object structures, helping our DeS3 preserve and recover meaningful features.

We propose the ViT similarity loss between the intermediate keys of the input image and the estimated denoised output image during the reverse sampling given a noise estimation network $\bm{\epsilon}_{\theta}(\operatorname{\mathbf{x}}_{t},\operatorname{\tilde{\mathbf{x}}},\mathbf{a}_{t},t)$ and ViT model:

$$\mathcal{L}_{\text{s}im}(\mathbf{\operatorname{\tilde{\mathbf{x}}}},\mathbf{\operatorname{\mathbf{x}}})=\left\|S^{l}(\mathbf{\operatorname{\tilde{\mathbf{x}}}})-S^{l}(\mathbf{\operatorname{\mathbf{x}}})\right\|_{F},$$ (4)

where $S$ is the self-similarity descriptor, $S^{l}(x)\in\mathbb{R}^{(n)\!\times\!(n)}$, with $n\times n$ dimension, where $n$ is the number of patches. In DINO-ViT, the input image is split into $n$ non-overlapping patches. $l=11$ is the deepest ViT layer. $\left\|\cdot\right\|_{F}$ is the Frobenius norm. The self-similarity descriptor is defined as:

$$S^{l}(\mathbf{\operatorname{\mathbf{x}}})_{ij}=\text{cos}(k^{l}_{i}(\mathbf{\operatorname{\mathbf{x}}}),k^{l}_{j}(\mathbf{\operatorname{\mathbf{x}}}))=1-\frac{k^{l}_{i}(\mathbf{\operatorname{\mathbf{x}}})\cdot k^{l}_{j}(\mathbf{\operatorname{\mathbf{x}}})}{\left\|k^{l}_{i}(\mathbf{\operatorname{\mathbf{x}}})\right\|\cdot\left\|k^{l}_{j}(\mathbf{\operatorname{\mathbf{x}}})\right\|},$$

where $\text{cos}(\cdot)$ is the cosine similarity between spatial keys $k_{i}$ and $k_{j}$.

Aspects	Ours	S.Diff.	SG	G2R	SP+M+I	DC	SP+M
1.DeS$\uparrow$	8.6$\pm$1.3	5.8$\pm$3.9	6.2$\pm$2.7	5.6$\pm$3.8	6.0$\pm$4.0	5.4$\pm$1.8	5.6$\pm$2.3
2.Real$\uparrow$	9.0$\pm$0.9	6.9$\pm$2.8	6.9$\pm$2.0	7.2$\pm$3.2	7.0$\pm$1.5	6.0$\pm$0.5	6.6$\pm$1.8

Table 2: User study on the self shadow removal of the UIUC dataset (Guo, Dai, and Hoiem 2011), our method obtained the highest mean (the max score is 10), showing our method is effective in shadow removal (DeS) and visually realistic (Real). The best results are in bold, the second best are in underline.

Experiments

Figure 10: Hard shadow removal results on the USR (top), AISTD (second) and SRD (bottom four rows) datasets.

Figure 11: Shadow removal results on the soft shadow dataset LRSS (top three) and SRD (bottom three rows) datasets. Note that our method does not need additional masks in training and testing.

Implementation

To ensure fair comparisons, all the baselines, including ours are trained and tested on the same datasets. We trained our DeS3 on each dataset and tested on the corresponding dataset, e.g., for SRD (Qu et al. 2017), we used 2680 SRD and 408 SRD images for training and testing, respectively. We use 1000 steps for training and 25 steps for inference, noise schedule $\beta_{t}$ linear from 0.0001 to 0.02. $\alpha$ and $\beta$ are empirically set to 0.5 in training. $\mathcal{L}_{\text{t}}(\operatorname{\tilde{\mathbf{x}}},\operatorname{\mathbf{x}})=\lambda_{\rm sim}\mathcal{L}_{\rm sim}$, which is the total reverse sampling loss, where $\lambda_{\text{sim}}=2$, we balance the self-tuned ViT loss to select the good sampling output.

Shadow Removal on Hard Shadows

The SRD dataset consists of 2680 training and 408 testing pairs of shadow and shadow-free images without shadow masks. It contains hard shadows and soft shadows. For the baseline methods that require shadow masks, we additionally use the processing results of DHAN (Cun, Pun, and Shi 2020) for SRD shadow masks.

Fig. 10 (bottom four rows) and Table 1 show the hard shadow removal results on the SRD dataset. Fig. 11 (bottom three rows) show the soft shadow removal results on the SRD dataset, which show our DeS3 outperforms the SOTA methods. The best values for each metric are highlighted in bold. Moreover, DeS3 achieves the highest PSNR and SSIM values in shadow, non-shadow and overall areas. PSNR and SSIM are widely used in image generation tasks (2023c; 2023b; 2023a). Fig. 10 (second row) show results on the AISTD dataset, which demonstrate the superiority of our DeS3 compared to baseline results. The USR dataset (Hu et al. 2019b) is an unpaired dataset, including hard, soft, and self shadows. We evaluate on USR test shadow images and show results in Fig. 10 (first row).

Method	Guo	Gryka	DHAN	SP+M	MS.GAN	DC	Guo (at)	S.Diff.	Ours
RMSE$\downarrow$	6.02	4.38	7.92	7.48	7.13	3.48	5.87	3.49	3.01
PSNR$\uparrow$	27.88	29.25	25.57	23.93	25.12	31.01	28.02	28.86	33.95
Train	P+M	P+M+S	P+M+S	P+M	UP	UP	P	P+M	P

Table 3: Results on the LRSS (soft shadow) dataset. M and S represent ground truth shadow masks and synthetic data are used in training. “P” and “UP” stand for “Paired” and “Unpaired”. Our DeS3 does not need shadow masks.

Figure 12: Ablation studies on the ViT similarity loss.

Figure 13: Ablation study on ViT vs VGG, the output of ViT is better than VGG.

Shadow Removal on Soft and Self Shadows

The LRSS (Gryka, Terry, and Brostow 2015) is a soft shadow dataset with 134 shadow images; we followed (Jin, Sharma, and Tan 2021; Gryka, Terry, and Brostow 2015), using the same 34 LRSS images with their corresponding shadow-free images for evaluation shown in Table 3. Our DeS3 achieves the highest PSNR and lowest RMSE.

Fig. 11 (top three rows) show results on the LRSS dataset. Besides the SOTA baselines, we compared our method with the traditional hard and soft shadow removal method (Guo, Dai, and Hoiem 2012). Since the majority of the SOTA baselines need the shadow mask for evaluation, we use the shadow mask provided by LRSS dataset (Gryka, Terry, and Brostow 2015). However, the provided binary shadow mask does not consider the ambiguous boundaries of soft and self shadows, thus leading to remaining shadows in the ambiguous boundaries.

UCF (Zhu et al. 2010) and UIUC dataset (Guo, Dai, and Hoiem 2011), contain 245 and 108 images. UIUC provides 30 images where shadows are caused by objects in the scene, which are used for self shadow evaluation, the results are shown in Fig. 9. For the baseline methods that require shadow masks, we additionally use the detection results of BDRAR (Zhu et al. 2018).

Method	S $\downarrow$	NS $\downarrow$	Overall$\downarrow$
w/o Adaptive Classifier Atten	6.35	3.02	4.46
w/o $\mathcal{L}_{\text{s}im}$	6.21	2.99	3.94
Ours (complete model)	5.88	2.83	3.72

Table 4: Ablation experiments of our method using the SRD dataset. The numbers represent RMSE, lower is better.

Ablation Studies

Fig. 12 and Table. 4 show the effectiveness of the ViT similarity loss used in DeS3. From Fig. 12 (second row), the ViT similarity loss help in maintaining the structures of the mountain, owing to the features (the self-similarity keys) of the pre-trained DINO-ViT. We compare our method with two diffusion baselines: WeatherDiffusion (Özdenizci and Legenstein 2023) and SR3 (Saharia et al. 2022b) shown in Tab.1. More results are in the supplementary material.

Conclusion

We have proposed DeS3, the first shadow removal network that performs shadow removal robustly on hard, soft and self shadows. Unlike existing methods, our DeS3 does not rely on masks during training and testing. To guide our reverse sampling process to preserve object/scene structure information, we propose the self-tuned ViT similarity loss. Unlike existing methods, our DeS3 performs shadow removal robustly on outdoor self shadows, which is intractable to have ground truths.