Object Detection in Video with
Spatiotemporal Sampling Networks
Gedas Bertasius1, Lorenzo Torresani2, and Jianbo Shi1
1University of Pennsylvania,2Dartmouth College
Abstract. We propose a Spatiotemporal Sampling Network (STSN)
that uses deformable convolutions across time for object detection in
videos. Our STSN performs object detection in a video frame by learning to spatially sample features from the adjacent frames. This naturally
renders the approach robust to occlusion or motion blur in individual
frames. Our framework does not require additional supervision, as it optimizes sampling locations directly with respect to object detection performance. Our STSN outperforms the state-of-the-art on the ImageNet
VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical ﬂow data for training.
1 Introduction
In recent years, deep convolutional networks have achieved remarkable results
in many computer vision tasks [1,2,3,4,5,6,7,8], including object detection in images [9,10,11,12,13,14,15,16,17,18,19]. However, directly applying these imagelevel models to object detection in video is diﬃcult due to motion blur, video
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi