Mi
tT:t+T2 R(2T+1)1HWfor each individual clip
VtT:t+T2R(2T+1)3HWof length (2T+ 1) in the
video, i.e., for t= 1;2;:::;L (clips at the beginning and
the end of the video will include fewer frames).
We want to use clips that are long enough to allow us
to jointly solve instance segmentation and tracking while
handling challenging cases of occlusion and motion blur.
At the same time, the clip should be short enough to allow
us to ﬁt it at high spatial resolution in the memory of a GPU.
The resulting clip-level instance masks Mi
tT:t+Tproduced densely for all overlapping clips t= 1;:::;L are
then aggregated to produce video-level instance masks Mi.
Our approach for clip-level instance segmentation is described in subsections 4.1 and 4.2. We also illustrate it in
Figure 2. The subsequent clip-level instance mask aggregation method is presented in subsection 4.3.
Frame t
Bbox...
ClsMask
Frame t+...     Mask  Propagation     CNN
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani