Scaling Vision Transformers

🤖 Plain-English Summary

Attention-based neural networks such as the Vision Transformer (ViT) have recently attained advanced results on many computer vision benchmarks. As a result, we successfully train a ViT model with two billion parameters, which attains a new advanced on ImageNet of 90.45% top-1 accuracy.

🔑 Key Findings

• Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively.
• While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale.
• To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute.

🔗 Related Resources