illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution
layers (only 2are shown in Figure 2). This means that the initially predicted
oﬀsetso(1)
t;t+kand the concatenated temporal features ft;t+kare ﬁrst used as
inputs to a deformable convolution layer that outputs a new feature map g(1)
t;t+k.
Next, we use g(1)
t;t+kto predict oﬀsets o(2)
t;t+k, and a new feature map g(2)
t;t+k. This
continues for 2 more layers until we obtain oﬀsets o(4)
t;t+k, which are then used
to sample the points out of the supporting feature map ft+k. The ﬁnal sampled
feature map g(4)
t;t+kis obtained via another deformable convolutional layer that
takes as inputs oﬀsets o(4)
t;t+kand the original supporting feature map ft+k.
Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi