BEiT: BERT Pre-Training of Image Transformers

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).

⚡ This is an original paraphrased summary — not copied from the abstract. Full paper available at the source link below.

• 1 Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers.
• 2 Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens).
• 3 We first "tokenize" the original image into visual tokens.

This research advances how AI systems learn, reason, and solve problems — with direct implications for automation and scientific discovery.

This summary is based on publicly available metadata and abstract. For the full research paper, visit the original source: