introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular
Mask R-CNN [16] to video by adding a branch that propagates frame-level instance masks from each video frame
to other frames within a temporal neighborhood (which we
refer to as a clip). This allows our method to compute cliplevel instance tracks centered at each individual frame of the
video. These densely estimated clip-level tracks are then
aggregated to form accurate and coherent object instance
sequences for the entire video, regardless of its length. This
renders our approach capable of handling challenging cases
of occlusions, disocclusions, and motion blur. Our method
achieves the best reported accuracy on the YouTube-VIS
dataset [42], outperforming the ICCV 2019 challenge winner [28] despite being much simpler and using signiﬁcantly
less labeled data (1000x fewer images and 10x fewer bounding boxes). In Table 1, we compare our approach vs these
prior methods in terms of accuracy and other characteristics.
2. Related Work
Instance Segmentation in Images. Compared to instance
segmentation in images [11, 16, 44, 1, 2, 22, 26], the problem considered in this paper requires not only to segment
object instances in individual frames, but also to determine
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani