3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW
an input video consisting of LRGB frames of spatial size
HW. The aim of our system is to segment and temporally
link all object instances that are visible for at least one frame
inVand that belong to a predeﬁned set of categories C=
f1;:::;Kg. To achieve this goal, our model outputs a videolevel instance mask track Mi2RLHWwith a category
labelci2f1;:::;Kgand a conﬁdence score si2[0;1]for
each object instance idetected in the video.
Evaluation Metric. Video instance segmentation is evaluated according to the metrics of average precision (AP) and
average recall (AR). Unlike in the image domain, these metrics are evaluated over the video sequence. Thus, to evaluate spatiotemporal consistency of the predicted mask sequences, the video Intersection over Union (IoU) between
a predicted object instance iand a ground truth object instancejis computed as:
IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani