ConViT: improving vision transformers with soft convolutional inductive biases*

🤖 Plain-English Summary

Abstract Convolutional architectures have proven to be extremely successful for vision tasks. We conclude by presenting various ablations to better understand the success of the ConViT.

🔑 Key Findings

• Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling.
• Vision transformers rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification.
• However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks.

🔗 Related Resources