a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from
both frames via a backbone convolutional network (CNN). Next, we concatenate
the features from the reference and supporting frames, and feed them through
multiple deformable convolutional layers. The last of such layers produces oﬀsets
that are used to sample informative features from the supporting frame. Our
spatiotemporal sampling scheme allows us to produce accurate detections even
if objects in the reference frame appear blurry or occluded.
we use 2Ksupporting frames for detection in the reference frame, by taking
theKpreceding frames and the Ksubsequent frames as supporting frames, i.e.
fItK;It(K1);:::;I t1;It+1;:::;I t+(K1);It+Kg. However, for ease of explanation we introduce our STSN by considering a single supporting frame It+k.
To eﬀectively integrate temporal information we need two things: 1) powerful
object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi