Stable Diffusion is a large text to image diffusion model trained on billions of images. Image diffusion models learn to denoise images to generate output images. Stable Diffusion uses latent images encoded from training data as input.

As of 2024, diffusion models are mainly used for computer vision tasks, including image denoising, inpainting, super-resolution, image generation, and video generation. These typically involves training a neural network to sequentially denoise images blurred with Gaussian noise.

Semantic Guidance for Diffusion Models was proposed in SEGA: Instructing Text-to-Image Models using Semantic Guidance and provides strong semantic control over image generation. Small changes to the text prompt usually result in entirely different output images.

The text-to-video generation diffusion model consists of three sub-networks: text feature extraction model, text feature-to-video latent space diffusion model, and video latent space to video visual space model. The overall model parameters are about 1.7 billion.