Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.
olution of the input and ﬁt more video frames in a GPU.
However, doing so typically leads to a signiﬁcant drop in
segmentation or detection performance. Alternatively, one
could perform high-resolution instance segmentation on individual frames and then link segmentations temporally in
a separate post-processing stage. However, performing instance segmentation and tracking in two disjoint steps often produces suboptimal results, because these two tasks
are closely intertwined. The key challenge then becomes
designing a uniﬁed model that can track objects in video
while maintaining strong detection accuracy.
Currently, the best method for video instance segmentation is the ICCV 2019 challenge winner [28]. It tackles
video instance segmentation by dividing it into four problems: 1) detection, 2) classiﬁcation, 3) segmentation, and
4) tracking. These four problems are solved independently
using several off-the-shelf components and their respective
solutions are combined and adapted to the video instance
segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani