Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

🤖 Plain-English Summary

Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the advanced, while enjoying faster inference speed.

🔑 Key Findings

• Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens.
• Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions.
• In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning.

🔗 Related Resources