IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT
t=1jMi(t)[~Mj(t)j(1)
where ~Mj(t)is the ground-truth segmentation of object j
in framet. To achieve a large IoU, a model must not only
accurately classify and segment object instances at a framelevel, but also reliably track them over the video sequence.
As in the COCO benchmark for image segmentation [25], the metrics of AP and AR are computed separatelyfor each object category, and then averaged over ten IoU
thresholds from 50% to95% at increments of 5%. Lastly,
the resulting AP and AR metrics are averaged over the category set, which yields the ﬁnal evaluation metric.
4. Mask Propagation
MaskProp takes a video Vof arbitrary length Las input and outputs video-level instance segmentation tracks
Mi, category labels ciand conﬁdence scores sifor all
objectsidetected in the video. In order to achieve this
goal, our method ﬁrst builds clip-level object instance tracks
Mi
tT:t+T2 R(2T+1)1HWfor each individual clip
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani