MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

🤖 Plain-English Summary

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR.

🔑 Key Findings

• However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes.
• This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text.
• In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question.

🔗 Related Resources