of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi
tT:t+T
andMj
t0T:t0+T. The score qi
t(y)is effectively an average
of the soft IoU computed between instance track Mi
tT:t+T,
and all its overlapping instance tracks that have been assigned video-level instance ID y.
Letq= maxy2Yqi
t(y)be the maximum score obtained
by considering all possible video-level instance IDs y2Y.
Ifqis greater than a certain threshold, the current instance track Mi
tT:t+Tis assigned the video-level instance
IDy= arg maxy2Yqi
t(y). Otherwise, the clip-level track
does not match any of the current video-level instances. In
such case we create a new video-level instance ID and assign it to the the clip-level track while also expanding the
setY, i.e.,yi
t=jYj+ 1andY=Y[fjYj + 1g.
Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani