of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal
Sampling Network (STSN), is speciﬁcally designed for a video object detection
task. Unlike standard Deformable CNNs [25], which use deformable convolution
in the spatial domain, our STSN learns to sample features temporally across
diﬀerent video frames, which leads to improved video object detection accuracy.
2.2 Object Detection in Videos
Up until the introduction of the ImageNet VID challenge [26], there were no
large-scalebenchmarksforvideoobjectdetection.Thus,thereareonlyfewmethods that we can compare our work to. T-CNNs [20,21] use a video object detection pipeline that involves predicting optical ﬂow ﬁrst, then propagating imagelevel predictions according to the ﬂow, and ﬁnally using a tracking algorithm to
select temporally consistent high conﬁdence detections. Seq-NMS [22] constructs
a temporal graph from overlapping bounding box detections across the adjacent
frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi