leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various
region proposal methods by another network to make the entire system trainable end-to-end. Following this work, several methods [18,19] extended Faster
R-CNN into a system that runs in real time with small reduction in performance.
Additionally, recent work [17] introduced position sensitive ROI pooling, which
signiﬁcantly improved the detection eﬃciency compared to prior object detection systems. Finally, two recent methods, Mask R-CNN [10], and Deformable
CNNs [25], improved object detection results even further and they represent
the current state-of-the-art in object detection. Whereas Mask-RCNNs use an
additional branch that predicts a mask for each region of interest, Deformable
Object Detection in Video with Spatiotemporal Sampling Networks 3
CNNs employ deformable convolutions, which allow the network to condition
discriminatively its receptive ﬁeld on the input, and to also model deformations
of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi