An Empirical Study of Training Self-Supervised Vision Transformers

🤖 Plain-English Summary

This paper does not describe a novel method. We discuss the currently positive evidence as well as challenges and open questions.

🔑 Key Findings

• Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT).
• While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging.
• In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT.

🔗 Related Resources