  We propose an online tracking algorithm that performs the object detection
and data association under a common framework, capable of linking objects after
a long time span. This is realized by preserving a large spatio-temporal memory
to store the identity embeddings of the tracked objects, and by adaptively
referencing and aggregating useful information from the memory as needed. Our
model, called MeMOT, consists of three main modules that are all
Transformer-based: 1) Hypothesis Generation that produce object proposals in
the current video frame; 2) Memory Encoding that extracts the core information
from the memory for each tracked object; and 3) Memory Decoding that solves the
object detection and data association tasks simultaneously for multi-object
tracking. When evaluated on widely adopted MOT benchmark datasets, MeMOT
observes very competitive performance.

Computer Vision and Pattern Recognition