6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame
of the video. As backbone network, we use a Deformable CNN [25] based on
the ResNet-101 [5] architecture, which is one of the top-performing object detection systems at the moment. Similarly to [25], our backbone network employs
6deformable convolutional layers. We also note that even though we use a Deformable CNN architecture, our system can easily integrate other architectures
and thus it can beneﬁt from future improvements in still-image object detection.
Spatiotemporal Feature Sampling. Our main contribution is the design
of a spatiotemporal sampling mechanism, which seamlessly integrates temporal
information in a given video. As a ﬁrst step, we feed the reference frame Itand
the supporting frame It+kthrough our image-level backbone network, which
produces feature tensors ftandft+k, respectively. Note that ft;ft+k2Rchw
wherec;h, andware the number of channels, the height, and the width of the
activation tensor. The feature tensors ft, andft+kare then concatenated into a
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi