defocus, unusual poses, or object occlusions (see Figure 1). Despite these challenges, it is natural to assume that video object detectors should be more powerful than still image detectors because video contains richer information about
the same object instance (e.g., its appearance in diﬀerent poses, and from different viewpoints). The key challenge then is designing a model that eﬀectively
exploits temporal information in videos.
Prior work [20,21,22,23] has proposed to exploit such temporal information
in videos by means of various post-processing steps aimed at making object detections coherent across time. However, since temporal coherence is enforced in
a second stage, typically these methods cannot be trained end-to-end. To overcome this limitation, recent work [24] has introduced a ﬂow-based aggregation
network that is trainable end-to-end. It exploits optical ﬂow to ﬁnd correspondences across time and it then aggregates features across temporal correspondences to smooth object detections over adjacent frames. However, one of the
downsides of this new model is that in addition to performing object detection,
it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi