Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control. Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. Such capabilities are difficult to learn solely from task-specific data.

A diffusion model consists of three major components: the forward process, the reverse process, and the sampling procedure. The goal of diffusion models is to learn a diffusion process that generates a probability distribution for a given dataset from which we can then sample new images.

The text-to-video generation diffusion model consists of three sub-networks: text feature extraction model, text feature-to-video latent space diffusion model, and video latent space to video visual space model. The overall model parameters are about 1.7 billion.

Image-to-image diffusion models are conditional diffusion models of the form p(y∣x), where both x and y are images, e.g., x is a grayscale image and y is a color image. These models have been applied to image super-resolution [Nichol and Dhariwal 2021; Saharia et al.