t, we compute an instance-speciﬁc feature tensor via element-wise multiplication between tensor tand the given frame-level
instance mask at frame t.2)Next, the element-wise difference of the feature tensors associated with frames tandt+is
used to predict motion offsets between frames tandt+. The resulting offsets are used to propagate the instance-speciﬁc
tensors from time tto timet+via deformable convolution. The propagated tensors represent instance-speciﬁc features
predicted for time t+using the tensors computed at time t.3)Lastly, we add the propagated instance feature tensors to the
tensor effectively computed at t+. A convolutional layer applied to these tensors predicts instance masks in frame t+.
The pixels that do not belong to any object instances are zeroed out using an instance-agnostic attention map.
the tensorft+effectively computed from the frame at time
t+. If the object instance prediction is consistent with the
feature computation, the feature tensors will be aligned and
thus, they will reinforce each other in the predicted region.
Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani