BiFormer: Vision Transformer with Bi-Level Routing Attention

🤖 Plain-English Summary

As the core building block of vision transformers, attention is a powerful tool to capture long-range dependency. Empirical results across several computer vision tasks such as image classification, object detection, and semantic segmentation verify the effectiveness of our design.

🔑 Key Findings

• However, such power comes at a cost: it incurs a huge computation burden and heavy memory footprint as pairwise token interaction across all spatial locations is computed.
• A series of works attempt to alleviate this problem by introducing handcrafted and content-agnostic sparsity into attention, such as restricting the attention operation to be inside local windows, axial stripes, or dilated windows.
• In contrast to these approaches, we propose a novel dynamic sparse attention via bi-level routing to enable a more flexible allocation of computations with content awareness.

🔗 Related Resources