drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from
MaskTrack R-CNN, the accuracy drops to 36:9mAP. Note
that all of these baselines are implemented using the same
backbone and detection networks.
worse in mAP compared to our model. Lastly, we note that
additionally pre-training MaskProp on the OpenImages [23]
dataset as was done in [28] further boosts our performance
to50:0mAP.
5.2. Ablation Experiments
Mask Propagation Branch. To investigate the effectiveness of our mask propagation branch, we compare our
method with a FlowNet2 [20] propagation baseline. For
this baseline, we use exactly the same setup as for our
MaskProp, except that instance masks are propagated using the optical ﬂow predicted by a FlowNet2 network [20]
rather than our proposed mask propagation scheme. For a
more complete comparison, we also include the MaskTrack
R-CNNfrom Table 2, which uses the originally proposed
tracking branch [42], but is implemented using the same
backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani