CMT: Convolutional Neural Networks Meet Vision Transformers

🤖 Plain-English Summary

Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively.

🔑 Key Findings

• However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs).
• In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models.
• We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to extract local information.

🔗 Related Resources