Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This
is the task considered in this work. There are only a few
video instance segmentation methods we can compare our
approach to. The MaskTrack R-CNN [42] presents a uniﬁed model for video instance segmentation. It augments the
original Mask R-CNN [16] with a tracking branch that establishes associations among object instances segmented in
separate frames. Furthermore, we include the ICCV 2019
video instance segmentation challenge winner [28] in our
comparison. This approach divides video instance segmentation into four separate subproblems: classiﬁcation, detection, segmentation, and tracking. A separate model (or
an ensemble of models) is used to solve each of these subproblems, and these solutions are then combined to produce
video instance segmentation results. For brevity, from now
on we refer to it as EnsembleVIS to indicate that it is an ensemble approach designed for video instance segmentation.
Our MaskProp framework provides advantages over
both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani