object instances in individual frames, but also to determine
instance correspondences across multiple frames. We leverage the Mask R-CNN model [16] for still-image instance
segmentation and adapt it to track object instances in video.
Object Detection in Video. Object detection in video requires classifying and localizing objects in every frame of
a given video. Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames. However, these systems
are typically not designed for tracking object instances. In
contrast, our mask propagation produces clip-level instancesegmentations rather than frame-level bounding boxes.
Video Object Segmentation. The task of video object
segmentation requires segmenting foreground objects in a
class-agnostic fashion [43, 21, 36, 38], often by leveraging
ground truth masks available for the ﬁrst frame during inference [6, 32, 10, 19, 37]. Instead, video instance segmentation requires ﬁnding all instances of a predeﬁned set of
object classes in each frame, classifying them and linking
them over the entire sequence.
Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani