Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for
each object instance iin framet+. The masks are normalized with the softmax nonlinearity across all Ntinstances. To zero-out pixels that do not belong to any object
instance, we use a single 33convolution that computes
an instance-agnostic attention map At+from feature tensorft+. We then multiply At+with each of our predicted
instance masks. A detailed illustration of our mask propagation branch is presented in Figure 4.
4.3. Video-Level Segmentation Instances
Given a video of length L, our goal is to produce videolevel segmentation instances Mi2 RLHW. Conceptually, this requires linking clip-level instance tracks
Mi
tT:t+TandMj
t0T:t0+Twheniandjrepresent the same
object instance, i.e., when the instances are matching. We
achieve this by assigning a video-level instance ID to each
of our predicted clip-level instance tracks. Matching instance tracks are assigned the same video-level instance ID.
Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani