it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training
such a model requires large amounts of ﬂow data, which may be diﬃcult and
costly to obtain, 3) integrating a ﬂow network and a detection network into a
single model may be challenging due to factors such as diﬀerent loss functions,
diﬀering training procedures for each network, etc. arXiv:1803.05549v2  [cs.CV]  24 Jul 2018
2 G. Bertasius, L. Torresani, J. Shi
Motion Blur
Occlusion
Unusual Pose
Video DefocusChallengingEasierChallengingEasier
Fig.1: An illustration of the common challenges associated with object detection
in video. These include video defocus, motion blur, occlusions and unusual poses.
Theboundingboxesdenotetheobjectsthatwewanttodetectintheseexamples.
To address these shortcomings, in this work, we introduce a simple, yet effective Spatiotemporal Sampling Network (STSN) that uses deformable convolutions [25] across space and time to leverage temporal information for object
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi