Scalable Diffusion Models with Transformers

🤖 Plain-English Summary

We explore a new class of diffusion models based on the transformer architecture. We find that DiTs with higher Gflops—through increased transformer depth/width or increased number of input tokens—consistently have lower FID.

🔑 Key Findings

• We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches.
• We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops.
• We find that DiTs with higher Gflops—through increased transformer depth/width or increased number of input tokens—consistently have lower FID.

🔗 Related Resources