EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

🤖 Plain-English Summary

Vision transformers have shown great success due to their high model capabilities. Compared to the recent efficient model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while running <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5.8\times/3.7\times$</tex> faster on the GPU/CPU, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$7.4\times faster$</tex> when converted to ONNX format.

🔑 Key Findings

• However, their remarkable performance is accompanied by heavy computation costs, which makes them unsuitable for real-time applications.
• In this paper, we propose a family of high-speed vision transformers named Efficient ViT.
• We find that the speed of existing transformer models is commonly bounded by memory inefficient operations, especially the tensor reshaping and element-wise functions in MHSA.

🔗 Related Resources