BEiT: BERT Pre-Training of Image Transformers

🤖 Plain-English Summary

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).

🔑 Key Findings

• Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers.
• Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens).
• We first "tokenize" the original image into visual tokens.

🔗 Related Resources