activation tensor. The feature tensors ft, andft+kare then concatenated into a
new feature tensor ft;t+k2R2chw. Note that this tensor ft;t+know has twice
as many channels as our initial tensors, and that it now contains object-level
information from both the reference and the supporting frame.
Next, we use the tensor ft;t+kto predict (x;y)location oﬀsets, which are
then used to sample the supporting tensor ft+k. The sampling mechanism is
implementedusingadeformableconvolutionallayer,whichtakes1)thepredicted
oﬀsets, and 2) the supporting tensor ft+kas its inputs, and then outputs a
newly sampled feature tensor gt;t+k, which can be used for object detection in
the reference frame. We use subscript t;t+kto denote the resampled tensor
because, although gis obtained by resampling the supporting tensor, the oﬀset
computation uses both the reference as well as the supporting frame. A detailed
illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi