detection in video. Our STSN learns to spatially sample useful feature points
from nearby video frames such that object detection accuracy in a given video
frame is maximized. To achieve this, we train our STSN end-to-end on a large set
of video frames labeled with bounding boxes. We show that this leads to a better accuracy compared to the state-of-the-art on the ImageNet VID dataset [26],
without requiring complex ﬂow network design, or the need to train the network
on large amounts of ﬂow data.
2 Related Work
2.1 Object Detection in Images
Modernobjectdetectors[9,10,11,12,13,14,15,16,17,18,19]arepredominantlybuilt
on some form of deep CNNs [1,3,5]. One of the earliest deep CNN object detection systems was R-CNN [14], which involved a two-stage pipeline where object
proposals were extracted in the ﬁrst stage, and then each proposal was classiﬁed
using a CNN. To reduce the computational burden, the methods in [9], and [13]
leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi