Emerging Properties in Self-Supervised Vision Transformers

🤖 Plain-English Summary

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels.

🔑 Key Findings

• Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets.
• Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT.
• Our study also underlines the importance of momentum encoder , multi-crop training , and the use of small patches with ViTs.

🔗 Related Resources