time ttime t+10time t-5time t+5time t+10time ttime t-10propagated instance        segmentationfeature tensors for each frame             in a video clip
time t-5time t+5time t+10time ttime t-10
Figure 2: An illustration of our MaskProp system, which takes as input a video clip centered around frame t, and outputs
a clip-level instance track. Our mask propagation framework can be summarized in three high-level steps: 1) An instancespeciﬁc feature map at time tis computed by masking the frame features at time twith the given instance segmentation
for framet(one for each instance detected in frame t). 2) Next, we use our mask propagation mechanism to temporally
propagate instance-speciﬁc features from frame tto all the other frames in the clip. 3) Lastly, our model predicts instancespeciﬁc segmentations in every frame of the clip by implicitly matching the propagated instance features with the frame-level
features computed at each time step. This last step yields clip-level instance tracks centered around frame t.
3) produces higher accuracy on YouTube-VIS [42].
3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani