for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand
the last 20Kiterations respectively. Note that in the second stage of training
we sample the supporting frames randomly within a certain neighborhood of a
reference frame (as described above).
Inference. During inference, we use T= 27, meaning that we consider
K= 13supporting frames before and after the reference frame. To avoid GPU
memory issues, we ﬁrst extract features from the backbone network for each
image individually, and then cache these features in the memory. Afterwards,
we feed all these features into our spatiotemporal sampling block. At the end,
standard NMS with a threshold of 0:3is applied to reﬁne the detections. To
handle the ﬁrst and the last K= 13frames in the video —two boundary cases
that require sampling the neighboring frames beyond the video start and end,
we pad the start of a video with Kcopies of the ﬁrst frame, and the end of a
video with Kcopies of the last frame.
5 Experimental Results
In this section, we evaluate our approach for video object detection on the ImageNet VID [26] dataset, which has 3;862and555training and testing video
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi