MetaFormer is Actually What You Need for Vision

🤖 Plain-English Summary

Transformers have shown great potential in computer vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules.

🔑 Key Findings

• A common belief is their attention-based token mixer module contributes most to their competence.
• However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well.
• Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance.

🔗 Related Resources