CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

🤖 Plain-English Summary

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters.

🔑 Key Findings

• Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification.
• To this end, we propose a dual-branch transformer to com-bine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features.
• Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other.

🔗 Related Resources