using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction
In this paper, we tackle the recently introduced video instance segmentation problem [42]. This task requires segmenting all instances of a predeﬁned set of object classes
in each frame, classifying them, and linking individual instances over the entire sequence.
In recent years, convolutional networks have obtained remarkable results in still-image object detection [16, 33, 14,
15], and segmentation [27, 45, 8, 4]. However, extending
these models to video instance segmentation is challenging. In order to localize objects precisely, these methods
have to operate at very large spatial resolution. As a result,
detectors based on the popular ResNet-101 or ResNet-152
backbones [17] can rarely ﬁt more than one image per GPU
during training. In the context of video instance segmentation this is problematic because tracking objects over time
requires analyzing multiple video frames simultaneously.
To address this issue, one could reduce the spatial resFigure 1: In this paper, we tackle the problem of video instance segmentation, which requires classifying, segmenting, and tracking object instances in a given video sequence.
Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani