4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track
object instances. Given a video clip VtT:t+Tcenteredat framet, our system outputs clip-level instance masks
Mi
tT:t+Tfor each predicted object instance iin framet.
Our mask propagation branch can be described in three
high-level steps: 1) instance-speciﬁc feature computation,
2) temporal propagation of instance features, and 3) propagated instance segmentation. We will now describe each of
these steps in more detail. We introduce our mask propagation with the example of propagating object instance masks
from frametto framet+where2[T:T].
Computing Instance Speciﬁc Features. The mask
branch of our model predicts frame-level instance masks
Mi
t2R1H0W0from single frame inputs. We then
use these frame-level instance masks to compute instancespeciﬁc features for frame t. Speciﬁcally, for each object instancei, we compute an element-wise product between Mi
t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani