Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)
sampling locations on a uniformly-spaced grid R, and 2) performing a weighted
summation of sampled values using weights w. For example, if we consider a
standard 2D convolution with a 33kernel, and a dilation factor of 1, the grid
Ris deﬁned asR=f(1;1);(1;0);:::; (0;1);(1;1)g. Under a standard 2D
convolution, to compute a new value at pixel location p0in the output feature
mapy, we would perform the following operation on the input feature map x:
y(p0) =X
pn2Rw(pn)x(p0+pn);(1)
Instead, in a deformable 2D convolution, the grid Ris augmented with dataconditioned oﬀsets fpnjn= 1;:::;Ng, whereN=jRj. We can then compute
a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi