High-Resolution Image Synthesis with Latent Diffusion Models

🤖 Plain-English Summary

By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve advanced synthesis results on image data and beyond. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner.

🔑 Key Findings

• Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining.
• However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations.
• To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders.

🔗 Related Resources