This repository contains the official implementation for the paper "InsertDiffusion: Identity Preserving Visualization of Objects through a Training-Free Diffusion Architecture".

The implementation supports two different running modes:

1. image mode - allows to "render" a preexisting image (arbitrary, as long as the background is white alternatively the object can be extracted from the original background by providing an object description) into an arbitrary background
2. point mode - (not central to our research) allows to create bike outlines from a point cloud and then insert the "rendered" bikes into an arbitrary background

In addition, the implementation offers the option to colorize the preexisting or generated images. When not using already colored images this drastically improves realism.

Further, a new environment can either be created from scratch (background replacement mode) or a preexisting image can be used and adapted.

All features are available from the main.py CLI and can also be integrated into your own projects. To use the CLI or the code an environment with torch, Pillow, pandas, transformers, and diffusers is required.

To insert an image into a newly created background, execute:

python main.py --image "<path to your image>" --background_prompt "<your prompt>"

So for example:

python main.py --image "./test_images/bikes/166.png" --background_prompt "a bicycle centered in frame in munich, 8k, red gemini, 50mm, f2.8, cinematic"

Instead of providing a prompt directly you can use the auto-prompting features by providing the necessary information. For bikes you need to provide the desired location, the index of the bike and a path to the datasheet. For cars you will have to provide the desired new location, the car manufacturer and car type. For products only the new location and a product type is required.

For further options and insertion into a given background consult the full CLI documentation below.

Instead of using the CLI, you may integrate the procedure into your code with this python function (again make sure to have all dependencies installed and copy the utils module to your project):

from utils import get_mask_from_image, sd_inpainting, sd_img2img, paste_image

def insert_diffusion(image: Image, mask_threshold: int, prompt: str, negative_prompt: str, img2img_model: str, inpainting_model: str, img2img_strength: float, inpainting_steps: int, inpainting_guidance: float, img2img_guidance: float, background_image: Image=None, composition_strength: float=1) -> Image:
    mask = get_mask_from_image(image, mask_threshold)
    
    if background_image is not None:
        image = paste_image(image, background_image)

    inpainted = sd_inpainting(inpainting_model, image, mask, prompt, negative_prompt, inpainting_guidance, inpainting_steps, inpainting_strength=composition_strength)
    result = sd_img2img(img2img_model, inpainted, prompt, negative_prompt, img2img_strength, img2img_guidance)
    return result

We implemented an additional method to generate Biked images from point clouds and then insert them into a background. This method does not work well, however, and is, therefore, not central to our research. The generation from point clouds was implementation by Ioan-Daniel Craciun and is based upon a DDPM/DDIM implemented and trained by Jiajae Fan.

To create images from point clouds remove the --image argument from your CLI call. In point mode you have to provide a datasheet path via --datasheet_path. Thus, the minimal CLI command becomes:

python main.py --point --datasheet_path "<path to datasheet>" --background_prompt "<your prompt>"

You may also use auto-prompting in point mode.

To integrate the generation of bikes from point clouds into your project copy the utils folder and use the following function:

from utils import get_mask_and_background
from utils import inpaint as bike_inpainting

def bike_diffusion(parameter_csv_path: str, device: torch.device, ckpt_id: str='29000', mask_dilation: int=5, mask_fill_holes: bool=True, bike_idx: int=0, wheel_design_type: int=0, width: int=256, height: int=256):
    assert wheel_design_type == 0 or wheel_design_type == 1
    mask, background = get_mask_and_background(parameter_csv_path, bike_idx, wheel_design_type, width, height)
    bike_img = bike_inpainting(background, mask, device, 50, ckpt_id, mask_dilation, mask_fill_holes)
    return bike_img.convert('RGB')

The image returned by this function is ready to be used for colorization or insertion.

If you are using an uncolored image add --colorize to one of the previous CLI calls. In addition, you need to provide a colorization prompt via --colorization_prompt or use auto-prompting by providing a color via --color.

An example CLI call could be:

python main.py --image "./test_images/bike_outline/168.png" --colorize  --datasheet_path "./csv/df_parameters_final.csv" --place "beach at sunset" --color "purple" --bike_idx 5"

Alternatively, to integrate colorization into your project, copy the utils module and use:

from utils import sd_colorization

def colorization(image: Image, colorization_model: str, upscaling_model: str, colorization_prompt: str, colorization_negative_prompt: str, fill_holes: bool, dilation: int, strength: float, prompt_guidance: float):
    colorized = sd_colorization(colorization_model, upscaling_model, image, colorization_prompt, negative_prompt=colorization_negative_prompt, fill_holes=fill_holes, dilation_iterations=dilation, colorization_strength=strength, prompt_guidance=prompt_guidance)
    return colorized

Note: CLI is still under construction and might be subject to change.

Note: Where applicable default values represent the parameters determined in our experiments. But different values might be optimal depending on the specific use case.

An additional script interactive.py provides an implementation to generate images interactively in a Human-in-the-Loop fashion. This means, at each step five options are generated in parallel and the user is asked to choose the best option. Then, only the chosen image is used for the next step.

No additional CLI arguments are accepted by the script the user will be prompted to make all decisions.

To run in interactive mode execute:

python interactive.py

The evaluation code used or our paper can be found under ./evaluation. Note that quantitative metrics were computed using the following CLI command:

python evaluate.py --exp_name "<experiment name>" --gen_file_path "<path to generated images>" --ref_file_path "<path to reference files>" --masks_path "<only used for composition, path to masks>"

Human-Evaluation metrics and inferential statistics were computed using the provided notebook.

If find our work helpful and want to use it for your research or project, please cite the paper as follows:

@misc{2407.10592,
Author = {Phillip Mueller and Jannik Wiese and Ioan Craciun and Lars Mikelsons},
Title = {InsertDiffusion: Identity Preserving Visualization of Objects through a Training-Free Diffusion Architecture},
Year = {2024},
Eprint = {arXiv:2407.10592},
}