Photoswap:
Personalized Subject Swapping in Images

Abstract

In an era where images and visual content dominate our digital landscape, the ability to manipulate and personalize these images has become a necessity. Envision seamlessly substituting a tabby cat lounging on a sunlit window sill in a photograph with your own playful puppy, all while preserving the original charm and composition of the image. We present Photoswap, a novel approach that enables this immersive image editing experience through personalized subject swapping in existing images. Photoswap first learns the visual concept of the subject from reference images and then swaps it into the target image using pre-trained diffusion models in a training-free manner. We establish that a well-conceptualized visual subject can be seamlessly transferred to any image with appropriate self-attention and cross-attention manipulation, maintaining the pose of the swapped subject and the overall coherence of the image. Comprehensive experiments underscore the efficacy and controllability of Photoswap in personalized subject swapping. Furthermore, Photoswap significantly outperforms baseline methods in human ratings across subject swapping, background preservation, and overall quality, revealing its vast application potential, from entertainment to professional editing.

Figure 1: Photoswap can effortlessly replace the subject in a source image, which could be either synthetic (first two rows) or real (bottom row), with a personalized subject specified in reference images, while preserving the original subject pose and the composition of the source image.

1 Introduction

Imagine a digital world where the boundaries of reality and creativity blur, where a photograph of a tabby cat lounging in a sunlit window sill can effortlessly be transformed to feature your playful puppy in the same pose. Or envision yourself as a part of a famous movie scene, replaced seamlessly with the original character while preserving the very essence and composition of the scene. Can we achieve this level of personalized image editing, not just with expert-level photo manipulation skills, but in an automated, user-friendly manner? This question lies at the heart of personalized subject swapping, the challenging task of replacing the subject in an image with a user-specified subject, while maintaining the integrity of the original pose and composition. It opens up a plethora of applications in areas such as entertainment, advertising, and professional editing.

Personalized subject swapping is a complex undertaking that comes with its own set of challenges. The task requires a profound comprehension of the visual concept inherent to both the original subject and the replacement subject. Simultaneously, it demands the seamless integration of the new subject into the existing image. One of the critical objectives in subject swapping is to preserve the similar pose of the replacement subject. It is crucial that the swapped subject seamlessly fits into the original pose and scene, creating a natural and harmonious visual composition. This necessitates careful consideration of factors such as lighting conditions, perspective, and overall aesthetic coherence. By effectively blending the replacement subject with these elements, the final image maintains a sense of continuity and authenticity.

Existing image editing methods fall short in addressing these challenges. Many of these techniques are restricted to global editing and lack the finesse needed to seamlessly integrate new subjects into existing images. For example, for most text-to-image (T2I) models, a slightly prompt change could lead to a totally different image. Recent works [27, 25, 6, 3, 46] allow user to control the generation with an additional input such as user brush, semantic layout, or sketches. However, it is still challenging to guide the generation process to follow users’ intent on the generation of object shape, texture, and identity. Other approaches [16, 41, 26] have explored the potential of using text prompts to edit image content in the context of synthetic image generation. Despite showing promise, these methods are not yet fully equipped to handle the intricate task of swapping subjects in existing images with user-specified subjects.

Therefore, we present Photoswap, a novel framework that leverages pre-trained diffusion models for personalized subject swapping in images. In our approach, the diffusion model learns to represent the concept of the subject ($O_{t}$). Then the representative attention map and attention output saved in the source image generation process will be transferred into the generation process of the target image to generate the new subject while keeping non-subject pixels unchanged. Our extensive experiments and evaluations demonstrate the effectiveness of Photoswap. Not only does our method enable the seamless swapping of subjects in images, but it also maintains the pose of the swapped subject and the overall coherence of the image. Remarkably, Photoswap outperforms baseline methods by a large margin in human evaluations of subject identity preservation, background preservation, and overall quality of the swapping (e.g., 50.8% vs. 28.0% in terms of overall quality). The contributions of this work are as follows: 1) We present a new framework for personalized subject swapping in images. 2) We propose a training-free attention swapping method that governs the editing process. 3) The efficacy of our proposed framework is demonstrated through extensive experiments including human evaluation.

2 Related Work

2.1 Text-to-Image Generation

In the early stages of text-based image generation, Generative Adversarial Networks (GANs) [14, 2, 19] were widely used due to their exceptional ability to produce high-quality images. These models aimed to align textual descriptions with synthesized images through multi-modal vision-language learning, achieving impressive results on specific domains (e.g., bird, chair and human face). When combined with CLIP [32], a large pre-trained model that learns visual-textual representations from millions of caption-image pairs, GAN models [7] have demonstrated promising outcomes in cross-domain text-to-image (T2I) generation. Recently, T2I generation has seen remarkable progress with auto-regressive [29, 9, 9] and diffusion models [27, 15, 30, 36], offering diverse outcomes and can synthesize high-quality images closely aligned with textual descriptions in arbitrary domains.

Rather than focusing on T2I generation tasks without any constraints, subject-driven T2I generation [28, 4, 35] requires the model to identify the specific object from a set of visual examples and synthesize novel scenes incorporating them based on the input text prompts. Building upon modern diffusion techniques, recent approaches such as DreamBooth [35] and Textual Inversion [12, 13, 21, 26] learn to invert special tokens from a given set of images. By combining these tokens with text prompts, they generate personalized unseen images. To improve data efficiency, retrieval augmentation techniques [39, 1, 5] leverages external knowledge bases to overcome limitations posed by rare entities, resulting in visually relevant appearances and enhanced personalization. In our work, we aim to tackle personalized subject swapping, not only preserving the identity of subjects in reference images, but also maintaining the context of the source image.

2.2 Text-guided Image Editing

Text-guided image editing manipulates an existing image based on the input textual instructions, while preserving certain aspects or characteristics of the original image. Early works based on GAN models [19] only limits to a certain object domain. Diffusion-based methods [46, 27, 10] break this barrier and support text-guided image editing. Though these methods generate stunning results, many of them suffer from conducting local editing, and additional manual masks [25, 45, 25] are required to constrain the editing regions, which is often tedious to draw. By employing cross-attention [16] or spatial characteristics [41], the local editing can be achieved but struggles with non-rigid transformations (e.g., changing pose) and retaining the original image layout structure. While Imagic [20] addresses the need for non-rigid transformations by fine-tuning a pre-trained diffusion model to capture image-specific appearances, it requires test-time finetuning, which is not time-efficient for deployment. Moreover, relying solely on text as input lacks precise control. In contrast, we propose a novel training-free attention swapping scheme that enables precise personalization based on reference images, without the need for time-consuming finetuning.

2.3 Exemplar-guided Image Editing

Exemplar-guided image editing covers a broad range of applications, and most of the works [42, 17, 49] can be categorized as exemplar-based image translation tasks, conditioning on various information, such as stylized images [24, 8, 48], layouts [44, 22, 18], skeletons [22], sketches/edges [38]. With the convenience of stylized images, image style transfer [23, 47] receives extensive attentions, replying on methods to build a dense correspondence between input and reference images, but it cannot deal with local editing. To achieve local editing with non-rigid transformation, conditions like bounding boxes and skeletons are introduced, but require drawing efforts from users, which sometimes are hard to obtain. A recent work [43] poses exemplar-guided image editing task as an inpainting task with the mask and transfers the semantic content from the reference image to the source one, with the context intact. Different from these works, we propose a more user-friendly scenario by conducting personalized subject swapping with only reference images and obtain high-quality editing results.

3 Preliminary

Diffusion models are a type of generative model that operates probabilistically. In this process, an image is created by gradually eliminating noise from the target that is characterized by Gaussian noise. In the context of text-to-image generation, a diffusion model typically involves a process where an initial random image is gradually refined step by step, with each step guided by a learned model, until it becomes a realistic image. The changes to the image spread out and affect many pixels over time. Given an initial random noise ${\boldsymbol{z}}_{T}\!\sim\!\mathcal{N}(0,\mathbf{I})$, the diffusion model gradually denoise ${\boldsymbol{z}}_{t}$, which gives ${\boldsymbol{z}}_{t-1}$.

Diffusion models are probabilistic generative models that learn to generate images by simulating a random process called a diffusion process. In the image generation process, the diffusion model gradually predicts the noise at the current diffusion step and denoises to get the final image. In this study, we utilize a pre-trained text-to-image diffusion model, Stable Diffusion [33], which encodes the image into latent space and gradually denoises the latent variable to generate a new image. Stable Diffusion is based on a U-Net architecture [34], which generates latent variable ${\boldsymbol{z}}_{t-1}$ conditioned on a given text prompt $P$ and the latent variable ${\boldsymbol{z}}_{t}$ from the previous step $t$:

$${\boldsymbol{z}}_{t-1}={\boldsymbol{\epsilon_{\theta}}}({\boldsymbol{z}}_{t},P,t)$$ (1)

The U-Net consists of layers that include repetition of self-attention and cross-attention blocks. This study focuses on manipulating self-attention and cross-attention to achieve the task of personalized subject swapping.

4 The Photoswap Method

Figure 2: The Photoswap framework. Given several images of a new concept, the diffusion model first learns the concept and converts it into a token. The upper part is the generation process of the source image, while the bottom part is the generation process of target image. The initial noise feature $z^{t}_{T}$ is copied from $z^{s}_{T}$ of the source. The attention output and attention map in the source image generation process would be transferred to the target image generation process. The final feature $z^{t}_{0}$ is decoded to output the target image. Refer to Section 4 for more details.

Providing a few reference images of a personalized target subject $O_{t}$, Photoswap can seamlessly swap it with another subject $O_{s}$ in a given source image $I_{s}$. The Photoswap pipeline is illustrated in Figure 2. To learn the visual concept of the target subject $O_{t}$, we fine-tune a diffusion model with reference images and do object inversion to represent $O_{t}$ using special token *. Then, to substitute the subject in the source image, we first obtain the noise $z_{T}$ that can be used to re-construct the source image $I_{s}$. Next, through the U-Net, we obtain the needed feature map and attention output in the self-attention and cross-attention layers, including $M$, $A$, and ${\boldsymbol{\phi}}$ (which we will introduce in Sec. 4.2). Finally, during the target image generation process that is conditioned on the noise $z_{T}$ and the target text prompt $P_{t}$, in the first $\lambda$ steps, those intermediate variables ($M$, $A$, and ${\boldsymbol{\phi}}$) would be replaced with corresponding ones obtained during the the source image generation process. In the last ($T-\lambda$) steps, no attention swapping is needed and we can continue the denoising process as usual to obtain the final resulting image. Sec. 4.1 discusses the visual concept learning technique we used, and Sec. 4.2 details the training-free attention swapping method for controllable subject swapping.

4.1 Visual Concept Learning

Subject swapping requires a thorough understanding of the subject’s identity and specific characteristics. This knowledge enables the creation of accurate representations that align with the source subject. The subject’s identity influences the composition and perspective of the image, including its shape, proportions, and textures, which affect the overall arrangement of elements. However, existing diffusion models lack information about the target subject ($O_{t}$) in their weights because the training data for text-to-image generation models does not include personalized subjects. To overcome this limitation and generate visually consistent variations of subjects from a given reference set, we need to personalize text-to-image diffusion models accurately. Recent advancements have introduced various methods, such as fine-tuning the diffusion model with distinct tokens associated with specific subjects, to achieve this “personalization”[11, 35, 21]. In our experiments, we primarily utilize DreamBooth[35] as a visual concept learning method. It’s worth noting that alternative concept learning methods can also be effectively employed with our framework.

Figure 3: SVD visualization of self-attention maps. Each image’s attention map is resized to 64x64 at every layer, and we calculate the average map across all layers for all diffusion time steps. Most significant components are extracted with SVD and visualized. Remarkably, the visualized results demonstrate a strong correlation with the layout of the generated image. The top two rows are visualization about synthetic images while the bottom two rows are about real images.

Figure 4: Self-attention map visualization across diffusion time steps. This representation reveals that the layout of the generated image is intrinsically embedded in the self-attention map from the initial steps. Consequently, to assert control over the layout, it is imperative to commence the attention swap at the earliest stages of the process.

4.2 Controllable Subject Swapping via Training-free Attention Swapping

Subject swapping poses intriguing challenges, requiring the maintenance of the source image’s spatial layout and geometry while integrating a new subject concept within the same pose. This necessitates preserving the critical features in the source latent variable, which encapsulates the source image information, and leveraging the influence of the target image text prompt $P_{t}$, which carries the concept token, to inject the new subject into the image.

The central role of the attention layer in orchestrating the generated image’s layout has been well-established in prior works [16, 3, 41]. To keep non-subject pixels intact, we orchestrate the generation of the target image $I_{t}$ by transferring vital variables to the target image generation process. Here, we explore how distinct intermediate variables within the attention layer can contribute to a controllable generation in the context of subject swapping.

Within the source image generation process, we denote the cross-attention map as ${\boldsymbol{A}}^{s}_{i}$, the self-attention map as ${\boldsymbol{M}}^{s}_{i}$, the cross-attention output as $\psi^{s}_{i}$, and the self-attention output as $\phi^{s}_{i}$. The corresponding variables in the target image generation process are denoted as ${\boldsymbol{A}}^{t}_{i}$, ${\boldsymbol{M}}^{t}_{i}$, $\psi^{t}_{i}$, $\phi^{t}_{i}$, where $i$ represents the current diffusion step.

In the self-attention block, the latent feature $z_{i}$ is projected into queries ${\boldsymbol{q_{i}}}$, keys ${\boldsymbol{k_{i}}}$, and values ${\boldsymbol{v_{i}}}$. We obtain the self-attention block’s output ${\boldsymbol{\phi}}_{i}$ using the following equation:

$${\boldsymbol{\phi}}_{i}={\boldsymbol{M}}_{i}{\boldsymbol{v}}_{i}\quad\text{where}\quad{\boldsymbol{M}}_{i}=\text{Softmax}\left({\boldsymbol{q}}_{i}{{\boldsymbol{k}}_{i}}^{T}\right)$$ (2)

where ${\boldsymbol{M}}_{i}$ is the self-attention map, and $\phi_{i}$ is the feature output from the self-attention layer. The cross-attention block’s output ${\boldsymbol{\psi}}_{i}$ is:

$${\boldsymbol{\psi}}_{i}={\boldsymbol{A}}_{i}{\boldsymbol{v}}_{i}\quad\text{where}\quad{\boldsymbol{A}}_{i}=\text{Softmax}\left({\boldsymbol{q}}_{i}{{\boldsymbol{k}}_{i}}^{T}\right)$$ (3)

where $\boldsymbol{A}_{i}$ is the cross-attention map. In both self-attention and cross-attention, the attention map $\boldsymbol{M}_{i}$ and $\boldsymbol{A}_{i}$ are correlated to the similarity between $q_{i}$ and $k_{i}$, acting as weights that dictate the combination of information in $v_{i}$. In this work, the manipulation of the diffusion model focus on self-attention and cross-attention within U-Net, specifically, swapping $\boldsymbol{\phi}$, $\boldsymbol{M}$, and $\boldsymbol{A}$, while keeping $\psi$ unchanged.

Algorithm 1 The Photoswap Algorithm

Inputs: source image $I_{s}$, reference images $O_{t}$, source image text prompt $P_{s}$, target image text prompt $P_{t}$, diffusion model $\theta$

$\theta^{*}\leftarrow\theta,O_{t}$ $\triangleright$ Finetune diffusion model to include the new concept

$\boldsymbol{z}_{T}^{s}\leftarrow DDIMInversion(ImageEncoder(I_{s}),P_{s})$$\triangleright$ Using DDIM to guarantee re-construction

${\boldsymbol{z}}^{t}_{T}\leftarrow\boldsymbol{z}_{T}^{s}$ $\triangleright$ Using the same starting noise

for $i=T,T-1,...,1$ do

    $\epsilon^{s},\boldsymbol{\phi}^{s}_{i},\boldsymbol{M}^{s}_{i},\boldsymbol{A}^{s}_{i}\leftarrow{\boldsymbol{\epsilon}}_{\theta^{*}}\!(\boldsymbol{z}^{s}_{i},P_{s},i)$ $\triangleright$ Denoise to get the attention output and map for source image

    $\boldsymbol{\phi}_{i}^{t},\boldsymbol{M}_{i}^{t},\boldsymbol{A}^{t}_{i}\leftarrow{\boldsymbol{\epsilon}}_{\theta^{*}}\!\!\left(\boldsymbol{z}_{i}^{t},P_{t},i\right)$ $\triangleright$ Denoise to get the attention output and map for target image

    $\boldsymbol{\phi}_{i}^{*},\boldsymbol{M}_{i}^{*},\boldsymbol{A}^{*}_{i}\leftarrow\text{SWAP}(\boldsymbol{\phi}_{i}^{s},\boldsymbol{M}_{i}^{s},\boldsymbol{A}^{s}_{i},\boldsymbol{\phi}_{i}^{t},\boldsymbol{M}_{i}^{t},\boldsymbol{A}^{t}_{i},i)$

    $\epsilon^{*}\leftarrow{\boldsymbol{\epsilon}}_{\theta^{*}}\!\!\left(\boldsymbol{z}_{i}^{t},P_{t},i,\boldsymbol{\phi}_{i}^{*},\boldsymbol{M}_{i}^{*},\boldsymbol{A}^{*}_{i}\right)$ $\triangleright$ Denoise the updated attention map and output

    $\boldsymbol{z}_{i-1}^{s}\leftarrow DDIMSampler(\boldsymbol{z}_{i}^{s},\epsilon^{s})$ $\triangleright$ Sample next latent variable for source image

    $\boldsymbol{z}_{i-1}^{t}\leftarrow DDIMSampler(\boldsymbol{z}_{i}^{t},\epsilon^{*})$ $\triangleright$ Sample next latent variable for source image

end for

$I_{t}=ImageDecoder(\boldsymbol{z}_{0}^{t})$

return $I_{t}$

function SWAP($\boldsymbol{\phi}^{s},\boldsymbol{M}^{s},\boldsymbol{A}^{s},\boldsymbol{\phi}^{t},\boldsymbol{M}^{t},\boldsymbol{A}^{t},i$)

    $\boldsymbol{\phi}^{*}\leftarrow(i<\lambda_{\phi})?\boldsymbol{\phi}^{s}:\boldsymbol{\phi}^{t}$ $\triangleright$ Control self-attention feature swap

    $\boldsymbol{M}^{*}\leftarrow(i<\lambda_{M})?\boldsymbol{M}^{s}:\boldsymbol{M}^{t}$ $\triangleright$ Control self-attention Map swap

    $\boldsymbol{A}^{*}\leftarrow(i<\lambda_{A})?\boldsymbol{A}^{s}:\boldsymbol{A}^{t}$ $\triangleright$ Control cross-attention map swap

return $\boldsymbol{\phi}^{*},\boldsymbol{M}^{*},\boldsymbol{A}^{*}$

end function

Self-attention map $\boldsymbol{M}$, as it calculates the similarity within spatial features after linear projection, plays a pivotal role in governing spatial content during the generation process. As visualized in Figure 3, we capture $\boldsymbol{M}$ during the image generation and highlight the leading components via Singular Value Decomposition (SVD). This visualization reveals a high correlation between $\boldsymbol{M}$ and the geometry and content of the generated image. Further, when visualizing the full steps of the diffusion process (Figure 4), we discern that the layout information is mirrored in the self-attention from the initial steps. This insight underscores the necessity of initiating the swap early on to prevent the emergence of a new, inherent layout.

Cross-attention map ${\boldsymbol{A}}$ is determined by both latent variable and text prompt, as in Equation 3, and ${\boldsymbol{A}}^{s}_{i}v$ can be viewed as a weighted sum of the information from a text prompt. Copying ${\boldsymbol{A}}^{s}_{i}$ to ${\boldsymbol{A}}^{t}_{i}$ during the target image generation process improves the layout alignment between the source image and the target image.

Self-attention output $\phi$, derived from the self-attention layer, encapsulates rich content information from the source image, independent of direct computation with textual features. Hence, replacing $\boldsymbol{\phi_{i}^{t}}$ with $\boldsymbol{\phi_{i}^{s}}$ enhances the preservation of context and composition from the original image. Our observations indicate that $\boldsymbol{\phi}$ exerts a more profound impact on the image layout than the cross-attention map $\boldsymbol{A}$.

Cross-attention output $\psi$, emanating from the cross-attention layer, embodies the visual concept of the target subject. It is vital to note that substituting cross-attention output $\psi_{i}^{s}$ with $\psi_{i}^{t}$ would obliterate all information from the target text prompt $P_{t}$, as illustrated in Equation 3. Given that $k_{i}^{t}$ and $v_{i}^{t}$ are projections of target prompt embeddings, we retain $\psi_{i}^{s}$ unchanged to safeguard the target subject’s identity.

Algorithm 1 provides the pseudo code of our full Photoswap algorithm.

Figure 5: Photoswap results across various object and image domains, demonstrating its wide applicability. From everyday objects to cartoon, the diversity in subject swapping tasks has showcased the versatility and robustness of our framework across different contexts.

5 Experiments

5.1 Imlementation Details

For the implementation of subject swapping on real images, we require an additional process that utilizes an image inversion method, specifically the DDIM inversion [40], to transform the image into initial noise. This inversion method relies on a reversed sequence of sampling to achieve the desired inversion. However, there exist inherent challenges when this inversion process is applied in text-guided synthesis within a classifier-free guidance setting. Notably, the inversion can potentially amplify the accumulated error, which could ultimately lead to subpar reconstruction outcomes. To fortify the robustness of the DDIM inversion and to mitigate this issue, we further optimize the null text embedding, as detailed in Mokady et al. [26]. The incorporation of this optimization technique bolsters the effectiveness and reliability of the inversion process, consequently allowing for a more precise reconstruction. Without further notice, the DDIM inversion in this paper is enhanced by null text embedding optimization.

During inference, we utilize the DDIM sampling method with 50 denoising steps and classifier-free guidance of 7.5. The default step $\lambda_{A}$ for cross-attention map replacement is 20. The default step $\lambda_{M}$ for self-attention map replacement is 25, while the default step for self-attention feature $\lambda_{\phi}$ replacement is 10. Note that the replacement steps may change to some specific checkpoint. As mentioned in Section 4, the target prompt $P_{t}$ is just source prompt $P_{s}$ with the object token being replaced with the new concept token. For concept learning, we mainly utilize DreamBooth [35] to finetune a stable diffusion 2.1 to learn the new concept from 3 5 images. The learning rate is set to 1e-6. We use Adawm optimizer with 800 hundred training step. We finetune both the U-net and text encoder. The DreamBooth training takes around 10 minutes on a machine with 8 A100 GPU cards.

5.2 Personalized Subject Swapping Results

Figure  5 showcases the effectiveness of our Photoswap technique for subject swapping. Our approach excels at preserving crucial aspects such as spatial layout, geometry, and the pose of the original subject while seamlessly introducing a reference subject into the target image. Remarkably, even in cartoon images, our method ensures that the background remains intact during the subject change process. A notable example is the "cat" image, where our technique successfully retains all the intricate details from the source image, including the distinctive "Whiskers." This demonstrates our framework’s ability to accurately capture and preserve fine-grained information during subject swapping.

(a) Multi-subject swap.

(b) Occluded subject swap.

Figure 6: Photoswap results on multi-subject and occluded subject scenarios. The results show that Photoswap can disentangle and replace multiple subjects at once. Also, Photoswap can identify the target object while avoiding influencing the non-subject pixels.

We further demonstrate the versatility of Photoswap by showcasing its effectiveness in multiple subject swap and occluded object swap scenarios. As depicted in Figure LABEL:fig:real_three_faces (a), we present a source image featuring two sunglasses, which are successfully replaced with reference glass while preserving the original layout of the sunglasses. Similarly, in Figure LABEL:fig:real_three_faces (b), we observe a source image with a dog partially occluded by a suit. The resulting swapped dog wears a suit that closely matches the occluded region. These examples serve to highlight the robustness of our proposed Photoswap method in handling various real-world cases, thereby enabling users to explore a broader range of editing possibilities.

5.3 Comparison with Baseline Methods

Personalized object swap is a new task and there is no existing benchmark. However, we could modify the existing attention manipulation based methods. More specifically, we used the same concept learning method DreamBooth to finetune the same stable diffusion checkpoint to inject the new concept. To fairly compare with our results, we modified existing prompt-based editing method P2P [16] , an editing method based diffusion models. Note that origin P2P only works on a pair of synthetic images, in our setting we use same concept learning dreambooth an fix the seed to allow concept swapping. On the other hand, PnP [41] could also be implmented in similar setting, however we found PnP usually can not lead to satisfactory object swapping and may lead to a huge difference between the source image and the generated image. We suspect that it is because PnP is designed for image translation so it does not initiate the attention manipulation step from the beginning step. The qualitative comparision between Photoswap and P2P+dreambooth is shown in Figure  7. We observe that P2P with DreamBooth could achieve achieve basic object swap, but it still suffers from background mismatching issue.

Figure 7: Qualitative comparison between P2P+DreamBooth and Photoswap. We can observe that P2P+dreambooth is capable of achieving subject swapping. However, it faces challenges in preserving both the background and the reference subject accurately, while for Photoswap, it is robust to handle various cases.

	Photoswap	P2P+DreamBooth	Tie
Subject Swapping	46.8%	25.6%	27.6%
Background Preservation	40.7%	32.7%	26.6%
Overall Quality	50.8%	28.0%	21.2%

Table 1: Human evaluation between Photoswap and P2P. Note that both Photoswap and P2P leverage same concept learning method DreamBooth. The results indicates, for most of the cases above 70%, the proposed Photoswap is better or on par with P2P.

Human Evaluation.

We conduct a human evaluation to study the editing quality by (1) Which result better swaps the subject as the reference and keeps its identity; (2) Which result better preserves the background; (3) Which result has better overall subject-driven swapping. We randomly sample 99 examples and adopt Amazon MTurk to compare between two results. To avoid potential bias, we hire 3 Turkers for each sample. Table 1 demonstrates the comparison between our Photoswap and P2P. Firstly, more turkers (over 46%) denote that our Photoswap better swaps the subject yet keeps its identity at the same time. Moreover, we can also preserve the background in the source image (41% vs. 33%), which is another crucial goal of this editing. In summary, Photoswap precisely performs subject swapping and preserves the remaining part from the input, leading to an overall superiority (50%) to P2P.

5.4 Controlling Subject Identity

The effectiveness of the proposed mutual self-attention is demonstrated through both synthetic image synthesis and real image editing. Additionally, we perform an analysis of the control strategy with varying values of $M$ during the denoising process. Figure  8 provides insights into this analysis. It is observed that when applying self-attention control with a large swapping step $\lambda_{M}$ for $M$, the synthesized image closely resembles the source image in terms of both style and identity. In this scenario, all contents from the source image are preserved, while the subject style learned from the reference subject is disregarded. As the value of $M$ decreases, the synthesized image maintains the subject from the reference image while retaining the layout and pose of the contents from the source image. This gradual transition in the control strategy allows for a balance between subject style transfer and preservation of the original image’s contents.

Figure 8: Ablation results of $M$ in Photoswap. With $M$ value increase, the generated one is more similar to the style and identity of source image and dissimilar to the reference subject, vice versa.

5.5 Attention Swapping Step Analysis

In this section, we visualize the effect of the influence of swapping steps of different components. As discussed in the main paper, self-attention output $\phi$, and self-attention map $M$, derived from the self-attention layer, encompasses comprehensive content information from the source image, without relying on direct computation with textual features. Previous works such as Hertz et al. [16] did not explore the usage of $\phi$ and $M$ in the object-level image editing process.

Figure 9: Results at different swapping steps. With consistent steps, swapping the self-attention output provides superior control over the layout, including the subject’s gestures and the background details. However, excessive swapping could affect the subject’s identity, as the new concept introduced through the text prompt might be overshadowed by the swapping of the attention output or attention map. This effect is more clear when swapping the self-attention output $\lambda_{\phi}$. Furthermore, we observed that replacing the attention map for an extensive number of steps can result in an image with significant noise, possibly due to a compatibility issue between the attention map and the $v$ vector.

Figure 9 provides a visual representation of the effect of incrementally increasing the swapping step for one $\lambda$ hyperparameter while maintaining the other two at zero. Although all of them can be utilized for subject swapping, they demonstrate varying levels of layout control. At the same swapping step, the self-attention output $\phi$ offers more robust layout control, facilitating better alignment of gestures and preservation of background context. In contrast, the self-attention map $\boldsymbol{M}$ and cross-attention map $\boldsymbol{A}$ demonstrate similar capabilities in controlling the layout.

However, extensive swapping can affect the subject’s identity, as the novel concept introduced via the text prompt might be eclipsed by the swapping of the attention output or attention map. This effect becomes particularly evident when swapping the self-attention output. This analysis further informs the determination of the default $\lambda_{\phi}$, $\lambda_{M}$, and $\lambda_{A}$ values. While the cross-attention map $\boldsymbol{A}$ facilitates more fine-grained generation control, given its incorporation of information from textual tokens, we discovered that $\boldsymbol{\phi}$ offers stronger holistic generation control, bolstering the overall output’s quality and integrity.

5.6 Results of Other Concept Learning Methods

Our mainly use DreamBooth as the concept learning method in the experiments, primarily due to its superior capabilities in learning subject identities [35]. However, our method is not strictly dependent on any specific concept learning method. In fact, other concept learning methods could be effectively employed to introduce the concept of the target subject.

Figure 10: Results of Text Inversion [12] as the concept learning module. It can successfully capture key subject features, but its performance drops when representing complex structures such as human faces.

To illustrate this, we present the results of Photoswap when applying Text Inversion [12]. We train the model using 8 A100 GPUs with a batch size of 4, a learning rate of 5e-4, and set the training steps to 1000. Results in Figure 10 indicate that Text Inversion also proves to be an effective concept learning method, as it successfully captures key features of the target object. Nevertheless, we observe that Text Inversion performance is notably underwhelming when applied to human faces. We postulate that this is because Text Inversion focuses on learning a new embedding for the novel concept, rather than finetuning the entire model. Consequently, the capacity to express the new concept becomes inherently limited, resulting in its less than optimal performance in certain areas.

5.7 Ethics Exploration

Figure 11: Results on real human face images across different races. Evidently, the skin colors are also successfully transferred when swapping a white person with a black person, and vice versa.

Like many AI technologies, text-to-image diffusion models can potentially exhibit biases reflective of those inherent in the training data [37, 31]. Given that these models are trained on vast text and image datasets, they might inadvertently learn and perpetuate biases, such as stereotypes and prejudices, found within this data. For instance, should the training data contain skewed representations or descriptions of specific demographic groups, the model may produce biased images in response to related prompts.

However, Photoswap has been designed to mitigate bias within the generation process of a text-to-image diffusion model. It achieves this by directly substituting the depicted subject with the intended target. In Figure  11, we present our evaluation of face swapping across various skin tones. It is crucial to observe that when there is a significant disparity between the source and reference images, the swapping results tend to homogenize the skin color. As a result, we advocate for the use of Photoswap on subjects of similar racial backgrounds to achieve more satisfactory and authentic outcomes. Despite these potential disparities, the model ensures the preservation of most of the target subject’s specific facial features, reinforcing the credibility and accuracy of the final image.

5.8 Failure Cases

Figure 12: Failure cases. The model sometimes struggles to accurately reconstruct hand details and complex background information such as formula on a whiteboard.

Here we highlight two common failure cases. First, the model struggles to accurately reproduce hands. When the subject includes hands and fingers, the swapping results often fail to precisely mirror the original hand gestures or the number of fingers. This issue could be an inherited challenge from Stable Diffusion. Moreover, Photoswap can encounter difficulties when the image comprises complex information. As illustrated in the lower row of Figure 12, Photoswap fails to reconstruct the complicated formula on a whiteboard. Therefore, while Photoswap exhibits strong performance across various scenarios, it’s crucial to acknowledge these limitations when considering its application in real-world scenarios involving intricate hand gestures or complex abstract information.

6 Conclusion

This paper introduces Photoswap, a novel framework designed for personalized subject swapping in images. To facilitate seamless subject photo swapping, we propose leveraging self-attention control by exchanging intermediate variables within the attention layer between the source image and reference images. Despite its simplicity, our extensive experimentation and evaluations provide compelling evidence for the effectiveness of Photoswap. Our framework offers a robust and intuitive solution for subject swapping, enabling users to effortlessly manipulate images according to their preferences. In the future, we plan to further advance the method to address those common failure issues to enhance the overall performance and versatility of personalized subject swapping.