Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks
My2RLHWas:
My(t) =(
Mi
tT:t+T(t)ifyi
t=y
0 otherwise .4.4. Implementation Details
Backbone Network. As our backbone we use a Spatiotemporal Sampling Network [5] based on a Deformable
ResNeXt-101-64x4d [12, 41] with a feature pyramid network (FPN) [24] attached on top of it.
Detection Network. For detection, we use a Hybrid Task
Cascade Network [7] with a 3stage cascade.
Mask Propagation Branch. The residual block in the
mask propagation branch consists of two 33convolutions with 128output channels each. The instance feature
propagation is applied to the FPN feature map with the second largest spatial resolution. To propagate instance features from one frame to another, we predict 9 (x;y)offsets
for every pixel, which are then used as input to a 33deformable convolutional layer with 256channels. To capture
motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani