78:9mAP score. However, increasing kto larger values reduces the accuracy.
Feature Aggregation Weight Analysis. To analyze how much each of
the supporting frame contributes to the ﬁnal object detections, we visualize the
average weight magnitudes wt;t+k(p)for diﬀerent values of k. This visualization
is presented in the right subplot of Figure 3. We note that in this case, the weight
magnitudes correspond to the point p, which is located at the center of an object.
From this plot, we can conclude that the largest contribution comes from the
supporting frames that are near the reference frame ( k=1;0;1). However,
note that even even supporting frames that are further away from the reference
frame (e.g. k=9;9) have non-zero weights, and contribute quite substantially
to the ﬁnal object detection predictions.
5.3 Qualitative Results
To understand how our STSN exploits temporal information from a given video,
we visualize in Figure 4, the average oﬀsets predicted by the STSN sampling
block. These oﬀsets are used by the STSN to decide, which object-level informationfromthesupportingframeshouldbeusedtodetectanobjectinthereference
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi