a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)
Since the oﬀset pnis typically fractional, the operation above is implemented using bilinear interpolation. Note that the oﬀsets are obtained by applying a separate convolutional layer to the activation tensor containing the feature
mapx.Thisyieldsanoﬀsetmapthathasthesamespatialresolutionastheinput
feature map. Also, note that the oﬀsets are shared across all feature channels of
a given activation tensor. During training, the weights for the deformable convolution kernel, and the oﬀsets kernel are learned jointly by propagating gradients
through the bilinear interpolation operator. We refer the reader to the original
work that introduced deformable convolutions [25] for further details.
4 Spatiotemporal Sampling Network
Our goal is to design a network architecture that incorporates temporal information for object detection in video.
Let us denote with Itthe frame at time tin the video. Let us consider one
of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi