frame. The green square in the reference frame depicts a pixel, for which we want
to compute a convolution output. The red square in the supporting frame represents an average oﬀset, which is used to determine which feature points from the
supporting frame should be sampled. The yellow arrow indicates object’s motion between the reference frame and the supporting frame. Note that despite
a relatively large motion between the reference and the supporting frames, our
STSN samples features from the supporting frame right around the center of the
object, which is exactly what we want. Such spatiotemporal sampling allows us
to detect objects even if they appear blurry or occluded in the reference frame.
12 G. Bertasius, L. Torresani, J. Shi
Reference Frame (t)Supporting Frame (t+9)
Reference Frame (t)Supporting Frame (t+9)
Fig.4: An illustration of our spatiotemporal sampling scheme (zoom-in for a better view). The green square indicates a point in the reference frame, for which
we want to compute a new convolutional output. The red square indicates the
corresponding point predicted by our STSN in a supporting frame. The yellow
arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi