TrackFormer: Multi-Object Tracking with Transformers

🤖 Plain-English Summary

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatio-temporal trajectories. TrackFormer introduces a new tracking-by-attention paradigm and while simple in its design is able to achieve advanced performance on the task of multi-object tracking (MOT17) and segmentation (MOTS20).

🔑 Key Findings

• We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end trainable MOT approach based on an encoder-decoder Transformer architecture.
• Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence.
• The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the conceptually new and identity preserving track queries.

🔗 Related Resources