Neutral Editing Framework for Diffusion-based Video Editing

Textual neutralization $(g\circ f)_{\mathcal{T}}$ at the top of Figure 3 is an application of the neutralization to the target prompt, which obtains neutral prompt $\mathcal{T}_{n}$ from target prompt $\mathcal{T}$ and video $\mathcal{V}$. To implement the factor identification $f_{\mathcal{T}}$ and the semantic disentanglement $g_{\mathcal{T}}$ in the textual neutralization, we first define the editing factors in the target prompt as ‘words’ contributing to editing. Thus, the $f_{\mathcal{T}}$ aims to localize these words in the target prompt $\mathcal{T}$. Since the target prompt is a description of desired modifications in the current video, the prompt and the video exhibit semantic misalignment due to the words associated with the modifications (i.e. editing factors). Intrigued by this observation, we measure all words in $\mathcal{T}$ based on their cosine similarities with $\mathcal{V}$ to localize the words exhibiting low similarities as the editing factors. Therefore, we define a textual factor identification as $f_{\mathcal{T}}(\mathcal{T},\mathcal{V})$ to produce scores about localization of editing factor in the target prompt based on their similarity scores as given below:

where the $\mathbf{w}=\psi_{T}(\mathcal{T})\in\mathbb{R}^{M\times d}$ is $d$-dimensional normalized word features of target prompt. The $M$ is the number of words and $\psi_{T}$ is the CLIP [25] text encoder. The $\mathbf{v}=\psi_{I}(\mathcal{V})\in\mathbb{R}^{L\times d}$ is $d$-dimensional video features with frame length $L$, where $\psi_{I}$ is the CLIP image encoder. $\textrm{mean}(\cdot)$ is a mean-pooling along frame axis. As editing factors have low similarity scores with video, we inverse the scores by subtracting them from one, finally producing the textual editing factor score denoted as $\mathbf{z}_{\mathcal{T}}=f_{\mathcal{T}}(\mathcal{T},\mathcal{V})$. This score $\mathbf{z}_{\mathcal{T}}$ is utilized for the following textual semantic disentanglement.

The textual semantic disentanglement $g_{\mathcal{T}}$ aims to build a neutral prompt $\mathcal{T}_{n}$ from a target prompt $\mathcal{T}$ by semantically reducing editing factors in the $\mathcal{T}$. To perform this, $g_{\mathcal{T}}$ employs the editing factor score $\mathbf{z}_{\mathcal{T}}$ to identify the editing factors and reduce their meaning by disentangling them. To be specific, we present two technical contributions about the disentangling methods: (1) factor swapping and (2) factor deforming. The factor swapping is to swap the identified editing factors with other words. To the scores above a specific value $s$ (e.g. 0.7) on $\mathbf{z}_{\mathcal{T}}$ (e.g. [0.1, 0.9, $\cdots$, 0.2]), their corresponding words are decided as editing factors, denoting them as $W_{\mathbf{z}_{\mathcal{T}}>s}$ in word space, where $\mathbf{z}_{\mathcal{T}}>s$ is the indices of words in target prompt $\mathcal{T}$ of a higher score than $s$. Thus, the $W_{\mathbf{z}_{\mathcal{T}}>s}$ are swapped with other word tokens, where to mitigate a semantic intervention by swapping tokens, we define a dummy token as $<$DMY$>$ for the swapping. As a result, the textual semantic disentanglement with factor swapping ultimately produces a neutral prompt $g_{\mathcal{T}}(z_{\mathcal{T}})=\mathcal{T}_{n}=[W_{1},\cdots,W_{i},\cdots,W_{M}]$ including $W_{\mathbf{z}_{\mathcal{T}}>s}=$ $<$DMY$>$, where $W_{i}$ denotes corresponding $i$-th word in the target prompt $\mathcal{T}$. Although the $\mathcal{T}_{n}$ with factor swapping maintains a distinct difference³³3Experimental studies in Appendix F provide optimal region of the $s$. with the $\mathcal{T}$ by the editing factors $W_{\mathbf{z}_{\mathcal{T}}>s}$, it relies on a heuristic manner in selecting the editing factor and is difficult to distinguish the difference among the factors. To this end, we further devised a feature-level disentanglement referred to as factor deforming. To be specific, it first disentangles the target prompt features $\mathbf{w}\in\mathbb{R}^{M\times d}$ into a format of linear combination using factor score $\mathbf{z}_{\mathcal{T}}\in\mathbb{R}^{M}$ as $\mathbf{w}=\mathbf{z}_{\mathcal{T}}\circ\mathbf{w}+(1-\mathbf{z}_{\mathcal{T}})\circ\mathbf{w}$, where $\circ$⁴⁴4Here, $\circ$ is different from composite operation in $f\circ g$ is element-wise multiplication with broadcasting. After that, we deform the features attended by $\mathbf{z}_{\mathcal{T}}$ using deformable operation $h(\cdot)$ as:

This format selectively deforms the text features concerning the editing factor while keeping disparities among features of editing factors. We apply feature down-scaling for the deforming⁵⁵5Appendix F also provides other factor deforming methods as $h(\mathbf{w})=\alpha\times\mathbf{w}$ with scaler $0\leq\alpha<1$. For $\alpha=0$, all deformed features become identical, working similarly to the factor swapping. Finally, we define a neutral prompt features $g_{\mathcal{T}}(\mathbf{z}_{\mathcal{T}})=\mathbf{w}_{n}\in\mathbb{R}^{M\times d}$, which is utilized in tuning process of NeuEdit framework.

Visual neutralization $(g\circ f)_{\mathcal{V}}$ at the bottom of Figure 3 is another application of neutralization to the input video, which produces neutral video $\mathcal{V}_{n}$ from target prompt $\mathcal{T}$ and video $\mathcal{V}$. Neutral video improves the effectiveness of editing by reducing the influence of the original content in a region to be edited. To construct a factor identification $f_{\mathcal{V}}$ and semantic disentanglement $g_{\mathcal{V}}$ in the visual neutralization, we also define the editing factors in the video as ‘pixels’ contributing to editing. Thus the $f_{\mathcal{V}}$ aims to localize these pixels in the input video $\mathcal{V}$. As editing models’ multi-modal interaction modules (i.e. cross-attention)⁶⁶6Video editing is based on pre-trained knowledge (e.g. Stable Diffusion [27]) with multi-modal attention to provide required modifications. contain information about the interactions between texts and frames, we employ this information to build the $f_{\mathcal{V}}$. We first embed each $i$-th frame of video into patch-wise features as $\mathbf{p}^{i}\in\mathbb{R}^{(W_{p}\times H_{p})\times d}$ and perform cross attention with a target prompt features $\mathbf{w}\in\mathbb{R}^{M\times d}$ to get cross attention maps as $\mathbf{m}^{i}\in\mathbb{R}^{(W_{p}\times H_{p})\times M}$, where ($W_{p}\times H_{p}$) is the number patches in the frame and $M$ is the number of words in the target prompt. Among the attention maps $\mathbf{m}^{i}$, we highlight the maps related to the editing using textual editing factor score $\mathbf{z}_{\mathcal{T}}\in\mathbb{R}^{M\times 1}$ as (Please see also illustration in the bottom of Figure 3 for clear understanding.):

where $\mathbf{z}_{\mathcal{V}}^{i}$ is the $i$-th frame visual editing factor score. After restoring $\mathbf{z}_{\mathcal{V}}^{i}$ up to the original frame scale ($W\times H$) and aggregating all frames, we finally define the visual editing identification as $f_{\mathcal{V}}(\mathcal{T},\mathcal{V})=\mathbf{z}_{\mathcal{V}}\in\mathbb{R}^{L\times(W\times H)}$, where $L$ is the number of video frames. The visual factor score $\mathbf{z}_{\mathcal{V}}$ is utilized for the following visual semantic disentanglement.

The visual semantic disentanglement $g_{\mathcal{V}}$ aims to build neutral video $\mathcal{V}_{n}$ from input video $\mathcal{V}$ by reducing the meaning of editing factors in the video. To this, based on visual editing factor score $\mathbf{z}_{\mathcal{V}}$, the $g_{\mathcal{V}}$ identifies factors contributing to editing at a pixel level and semantically reduces them. Similar to textual semantic disentanglement $g_{\mathcal{T}}$, we apply a factor deforming by separating video pixels into two groups of pixels and deforming a group related to editing as below:

where we applied Gaussian blurring to deform the video concerned about editing factors as $h(\mathcal{V})=\mathcal{V}*G$ with Gaussian kernel $G(x,y)=\frac{1}{2\pi\sigma^{2}}e^{-(x^{2}+y^{2})/2\sigma^{2}}$. Therefore the visual semantic disentanglement is summarized as $g_{\mathcal{V}}(\mathbf{z}_{\mathcal{V}})=\mathcal{V}_{n}$. The $\mathcal{V}_{n}$ is used for editing instead of the $\mathcal{V}$, alleviating restrictions imposed by the original content and facilitating dynamic variations within the editing areas.

For textual factor identification $f_{\mathcal{T}}$, CLIP model (ViT-L/14) [25] is used for text and image features. The attention map for visual neutralization is based on the cross attention module of video diffusion model [38] (i.e. Stable Diffusion v1.5). The experimental settings are $W=H=512,W_{p}=H_{p}=16$ on NVIDIA A100 GPU.

We validate videos on DAVIS [22] and LOVEU-TGVE [39], which are video editing challenge dataset⁸⁸8https://sites.google.com/view/loveucvpr23/track4 comprising 32 to 128 frames of each. NeuEdit framework is validated about non-rigid/rigid editing on recent editing systems including Tune-A-video, Video-P2P, FateZero [23] on their public codes.

Figure 5 shows the qualitative results of recent editing systems [38, 19] with the proposed neutral editing framework. (See also qualitative results in Appendix G). To validate the qualitative effectiveness of neutralization, we perform case studies in terms of two types of editing: (a) non-rigid editing and (b) rigid editing. In the case of non-rigid editing, current editing systems’ results are not aligned with the target prompt, generating original input videos or incorrectly synthesizing original content (e.g. trees) and required variations (e.g. wings). However, these models with the NeuEdit framework demonstrate effective non-rigid editing on various targets including human and object. It is also notable that motion editing about thumbs-up is conditionally performed according to the visibility of the skier’s hand, this is because the visual neutralization is sensibly applied to visible editing factors (i.e. hand). We provide further analysis of this in Section 5.4. In the case of rigid editing (i.e. top: style transfer, bottom: object overlay), the current models and the models with NeuEdit provide qualitatively proper modifications. But, in detail, only the models under NeuEdit framework maintain a finer fidelity to the unedited region (i.e. yellow box) in the video. It is considered that feature commonality between neutral prompt and target prompt, excluding the editing factor, improves selective fidelity in the model. At the bottom, we also edited a man in kite-surfing to resemble Spider-Man. Interestingly, the model with NeuEdit also modifies the action of catching a kite as catching it with a spider web. We consider the neutral video shown in the pink box contributes to effective editing, mitigating the restriction by the original content in the area to be edited.

Table 1 presents evaluations of the non-rigid/rigid editing on videos of DAVIS and TGVE¹⁰¹⁰10Appendix C provides further results on UCF101[35] of recent editing systems with the NeuEdit about four assessments (i.e. alignment, fidelity, consistency, human evaluation). The effectiveness of NeuEdit is confirmed in all models. Especially in non-rigid editing, textual alignment is significantly improved. Fidelity evaluates the preservation of unedited areas in video, such that we measure fidelity after masking identical regions related to editing. The fidelity is effectively enhanced in the tuning-based models (i.e. TAV, Video-P2P) than tuning-free model (i.e. FateZero), which tells that neutral prompt contributes to improving fidelity.