object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.
Weachievetheformerbyemployingastate-of-the-artbackbonenetwork.Forthe
latter, we design a spatiotemporal sampling scheme, which we describe below.
Our STSN can be summarized in four steps. First, a backbone convolutional
network computes object-level features for each video frame individually. Then,
spatiotemporal sampling blocks are applied to the object-level feature maps in
order to sample relevant features from nearby frames conditioned on the input
reference frame.Next,thesampledfeaturesfromeachvideoframearetemporally
aggregated into a single feature tensor for the reference frame using a per-pixel
weighted summation. Finally, the feature tensor is provided as input to the
detectionnetworktoproduceﬁnalobjectdetectionresultsforthegivenreference
frame. We note that our framework integrates these conceptually-distinct four
steps into a single architecture, which we train end-to-end.
6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi