flatten: optical FLow-guided ATTENtion for consistent text-to-video editing

Latent diffusion models operate in the latent space with an auto-encoder and demonstrate superior performance in text-to-image generation. In the forward process, Gaussian noise is added to the latent input $\bm{z}_{0}$. The density of $\bm{z}_{t}$ given $\bm{z}_{t-1}$ can be formulated as:

$$\centering q(\bm{z}_{t}|\bm{z}_{t-1})=\mathcal{N}(\bm{z}_{t};\sqrt{1-\beta_{t}}\bm{z}_{t-1},\beta_{t}\bm{\text{I}}),\@add@centering$$

(1)

where $\beta_{t}$ is the variance schedule for the timestep $t$. The number of timesteps used to train the diffusion model is denoted by $T$. The backward process uses a trained U-Net $\epsilon_{\theta}$ for denoising:

$$\centering p_{\theta}(\bm{z}_{t-1}|\bm{z}_{t})=\mathcal{N}(\bm{z}_{t-1};\mu_{\theta}(\bm{z}_{t},\bm{\tau},t),\Sigma_{\theta}(\bm{z}_{t},\bm{\tau},t)),\@add@centering$$

(2)

where $\bm{\tau}$ indicates the textual prompt. $\mu_{\theta}$ and $\Sigma_{\theta}$ are computed by the denoising model $\epsilon_{\theta}$.

DDIM can convert a random noise to a deterministic $\bm{z}_{0}$ during sampling (Song et al., 2020a; Dhariwal & Nichol, 2021). Based on the assumption that the ODE process can be reversed in the small-step limit, the deterministic DDIM inversion can be formulated as:

$$\centering\bm{z}_{t+1}=\sqrt{\frac{\alpha_{t+1}}{\alpha_{t}}}\bm{z}_{t}+\sqrt{\alpha_{t+1}}\left(\sqrt{\frac{1}{\alpha_{t+1}-1}}-\sqrt{\frac{1}{\alpha_{t}}-1}\right)\epsilon_{\theta}(\bm{z}_{t}),\@add@centering$$

(3)

where $\alpha_{t}$ denotes $\prod^{t}_{i=1}(1-\beta_{i})$. DDIM inversion is employed to invert the input $\bm{z}_{0}$ into $\bm{z}_{T}$, which can be used for reconstruction and further editing tasks.

The original U-Net architecture employed in an image-based diffusion model comprises a stack of 2D convolutional residual blocks, spatial attention blocks, and cross-attention blocks that incorporate textual prompt embeddings. To adapt the T2I model to the T2V editing task, we inflate the convolutional residual blocks and the spatial attention blocks. Similar to previous works (Ho et al., 2022b; Wu et al., 2022), the $3\times 3$ convolution kernels in the convolutional residual blocks are converted to $1\times 3\times 3$ kernels by adding a pseudo temporal channel. In addition, the spatial attention is replaced with a dense spatio-temporal attention paradigm. In contrast to the spatial self-attention strategy applied to the patches in a single frame, we adopt all patch embeddings across the entire video as the queries ($\bm{Q}$), keys ($\bm{K}$), and values ($\bm{V}$). This dense spatio-temporal attention can provide a comprehensive perspective throughout the video. Note that the parameters of the linear projection layers and the feed-forward networks in the new dense spatio-temporal attention blocks are inherited from those in the original spatial attention blocks.

To further improve the visual consistency of the output frames, we integrate our proposed flow-guided attention in the extended U-Net blocks. We combine FLATTEN with dense spatio-temporal attention since both attention mechanisms are designed to aggregate visual context. Given the latent video features, we first perform dense spatio-temporal attention. Specific linear projection layers are employed to convert the patch embeddings of the latent features into the queries, keys, and values, respectively. The results of dense spatio-temporal attention are denoted as $\bm{H}$. To avoid introducing newly trainable parameters and preserve the feature distribution, we do not apply new linear transformations to recompute the queries, keys, and values. We directly use $\bm{H}$ as the input of flow-guided attention. Note that no positional encoding is introduced. When a patch embedding serves as a query, the corresponding keys and the values for FLATTEN are gathered from the output of dense spatio-temporal attention $\bm{H}$ based on the patch trajectories sampled from optical flow. More details are demonstrated in Section 3.3. After performing flow-guided attention, the output is forwarded to the feed-forward network from the dense spatio-temporal attention block. We activate FLATTEN not only during DDIM sampling but also when performing DDIM inversion since using FLATTEN in DDIM inversion allows a more efficient inversion by introducing additional temporal dependencies. More details are discussed in Appendix A.

We also implement the feature injection following the image editing method (Tumanyan et al., 2023). For efficiency, we do not reconstruct the source video but inject the features from DDIM inversion during sampling. With these adaptations, our framework establishes and enhances the connections between frames, thus contributing to high-quality and highly consistent edited videos.

Given two consecutive RGB frames from the source video, we use RAFT (Teed & Deng, 2020) to estimate optical flow. The optical flow between two frames denotes a dense pixel displacement field $(f_{x},f_{y})$. The coordinates of each pixel $(x_{k},y_{k})$ in the $k$-th frame can be projected to its corresponding coordinates in the ($k+1$)-th frame based on the displacement field. The new coordinates in the ($k+1$)-th frame can be formulated as:

$$\centering(x_{k+1},\;y_{k+1})=(x_{k}+f_{x}(x_{k},y_{k}),\;y_{k}+f_{y}(x_{k},y_{k})).\@add@centering$$

(4)

In order to implicitly use optical flow to guide the attention modules, we downsample the displacement fields of all frame pairs to the resolution of the latent space.

We sample the patch trajectories in the latent space based on the downsampled fields $(\hat{f}_{x},\hat{f}_{y})$. We start iterating from the patches on the first frame. For a patch with coordinates $(x_{0},y_{0})$ on the first frame, its coordinates on all subsequent frames can be derived from the displacement field. The coordinates are linked, and the trajectory sequence can be presented as:

$$\centering traj=\{(x_{0},y_{0}),(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{K},y_{K})\},\@add@centering$$

(5)

where $K$ denotes the frame number of the source video. For a latent space with the size $H\times W$, there is ideally a trajectory set denoted as $\{traj_{1},traj_{2},...,traj_{N}\}$, where $N=HW$. However, certain patches disappear over time, and new patches appear in the video. For each new patch that appears in the video, a new trajectory is created. As a result, the size of the trajectory set $N$ is generally larger than $HW$. To simplify the implementation of flow-guided attention, when an occlusion happens, we randomly select a trajectory to continue sampling and stop the other conflicting trajectories. This strategy ensures that each patch in the video is uniquely assigned to a single trajectory, and there is no case where a patch is on multiple trajectories.

Flow-guided attention is performed on the sampled patch trajectories. The overview of FLATTEN is illustrated in Figure 4. We gather the embeddings of the patches on the same trajectory from the latent feature $\bm{z}$. The patch embeddings on a trajectory $traj$ can be presented as:

$$\centering\bm{z}_{traj}=\{\bm{z}(x_{0},y_{0}),\bm{z}(x_{1},y_{1}),\bm{z}(x_{2},y_{2}),\cdots,\bm{z}(x_{K},y_{K})\},\@add@centering$$

(6)

where $\bm{z}(x_{k},y_{k})$ indicates the patch embedding at the coordinates $(x_{k},y_{k})$ in the $k$-th frame. We perform multi-head attention with the patch embeddings on the same trajectory. For a query $\bm{z}(x_{k},y_{k})$, the corresponding keys and values are the other patch embeddings on the same trajectory $traj$. No additional position encoding is introduced. Our flow-guided attention can be formulated as follows:

	$\displaystyle\centering\bm{Q}\@add@centering$	$\displaystyle=\bm{z}(x_{k},y_{k}),$		(7)
	$\displaystyle\bm{K}=\bm{V}$	$\displaystyle=\bm{z}_{traj}-\{\bm{z}(x_{k},y_{k})\},$		(8)
	$\displaystyle\text{Attn}(\bm{Q},\bm{K},\bm{V})$	$\displaystyle=\text{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d}})\bm{V},\vspace{-2mm}$		(9)

where $\sqrt{d}$ is a scaling factor. The latent features $\bm{z}$ are updated by flow-guided attention to eliminate the negative effects from feature aggregation of irrelevant patches in dense spatio-temporal attention. Importantly, we ensure that each patch embedding on the latent feature is uniquely assigned to a single trajectory during patch trajectory sampling. This assignment resolves conflicts and allows for a comprehensive update of all patch embeddings.

We utilize optical flow to connect the patches in different frames and sample the patch trajectories. Our flow-guided attention facilitates the information exchange between patches on the same trajectory, thus improving visual consistency in video editing. We integrate FLATTEN into our framework and implement text-to-video editing without any additional training. Furthermore, FLATTEN can also be easily integrated into any diffusion-based T2V editing method, as shown in Section 4.4.

We evaluate our text-to-video editing framework with 53 videos sourced from LOVEU-TGVE^***https://sites.google.com/view/loveucvpr23/track4. 16 of these videos are from DAVIS (Perazzi et al., 2016), and we denote this subset as TGVE-D. The other 37 videos are from Videvo, which are denoted as TGVE-V. The resolution of the videos is re-scaled to $512\times 512$. Each video consists of 32 frames labeled with a ground-truth caption and 4 creative textual prompts for editing.

As per standard (Wu et al., 2022; Qi et al., 2023; Ceylan et al., 2023; Geyer et al., 2023), we use the following automatic evaluation metrics: For textual alignment, we use CLIP (Radford et al., 2021) to measure the average cosine similarity between the edited frames and the textual prompt, denoted as CLIP-T. To evaluate visual consistency, we adopt the flow warping error $\text{E}_{warp}$ (Lai et al., 2018), which warps the edited video frames according to the estimated optical flow of the source video and measures the pixel-level difference. Using these metrics independently cannot comprehensively represent editing performance. For instance, $\text{E}_{warp}$ reports 0 errors when the edited video is exactly the source video. Therefore, we propose $S_{edit}$ as our main evaluation metric, which combines CLIP-T and $\text{E}_{warp}$ as a unified score. Specifically, the editing score is calculated as $S_{edit}$ = CLIP-T/$\text{E}_{warp}$. Following the previous work (Wu et al., 2022), we also adopt CLIP-F and PickScore, which computes the average cosine similarity between all frames in a video and the estimated alignment with human preferences, respectively. For brevity, the numbers of CLIP-F/CLIP-T/$\text{E}_{warp}$ shown in this paper are scaled up by 100/100/1000.

We inflate a pre-trained text-to-image diffusion model and integrate FLATTEN into the U-Net to implement T2V editing without any training or fine-tuning. To estimate the optical flow of the source videos, we utilize RAFT (Teed & Deng, 2020). We find that applying flow-guided attention in DDIM inversion can also improve latent noise estimation by introducing additional temporal dependencies. Therefore, we use flow-guided attention both in DDIM sampling and inversion. More details are shown in Appendix A. We implement 100 timesteps for DDIM inversion and 50 timesteps for DDIM sampling. Following the image editing method (Tumanyan et al., 2023), the diffusion features are saved during DDIM inversion and are further injected during sampling. To efficiently perform the dense spatio-temporal attention in the modified U-Net, we use xFormers (Lefaudeux et al., 2022), which can reduce GPU memory consumption.