MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

🤖 Plain-English Summary

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. Without bells-and-whistles, MViTv2 has advanced performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 AP <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">box</sup> on COCO object detection as well as 86.1% on Kinetics-400 video classification.

🔑 Key Findings

• We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections.
• We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work.
• We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute.

🔗 Related Resources