Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi
tT:t+TandMj
t0T:t0+T, that
are centered around frames tandt0respectively. They overlap in time ifjtt0j<2T+ 1. Let us denote their overlapping time interval as \t;t0. Given two overlapping in-stance tracks, we can check if they match by comparing
their predicted instance masks in the overlapping frames.
If the masks of the two instance tracks align well, then they
are likely to encode the same object instance. Otherwise,
each track represents a different object instance. We compute a matching score mi;j
t;t0between two clip-level instance
tracks using our previously deﬁned soft IoU metric as:
mi;j
t;t0=1
j\t;t0jX
~t2\t;t0sIoU (Mi
tT:t+T(~t); Mj
t0T:t0+T(~t))(4)
Video-Level Instance ID Assignment. We denote withY
the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi
tT:t+T
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani