pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,
we randomly sample one supporting frame before and one supporting frame after
the reference frame. We observed that using more supporting frames in training
does not lead to a higher accuracy.
For the rest of our training procedure, we follow the protocol outlined in [24].
Speciﬁcally, we train our model in two stages. First, we pre-train our full model
on the Imagenet DET dataset using the annotations of the 30object classes
that overlap with the Imagenet VID dataset. Note that Imagenet DET dataset
contains only images, and thus, we cannot sample meaningful supporting frames
in this case. Therefore, in the case of images, we use the reference frames as our
supportingframes.Afterwards,theentiremodelistrainedfor 120Kiterationson
4Tesla K40 GPUs with each GPU holding a single mini-batch. The learning rate
is set to 0:001and0:0001for the ﬁrst 80Kand the last 40Kiterations respectively. Afterwards, we ﬁnetune the entire model on the Imagenet VID dataset
for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi