frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video
object detection task as a multi-object tracking problem. Finally, the method of
Feichtenhofer et al. [27] proposes a ConvNet architecture that solves detection
and tracking problems jointly, and then applies a Viterbi algorithm to link the
detections across time.
The approach most similar to our work is the method of Zhu et al. [24], who
proposed an end-to-end trainable network that jointly estimates optical ﬂow and
also detects objects in video. This is accomplished by using the predicted optical
ﬂow to align the features from the adjacent frames. The aggregated features are
then fed as input to the detection network.
OurmethodisbeneﬁcialoverthemethodsthatuseopticalﬂowCNNssuchas
themethodofZhuetal.[24].First,wenotethatpretrainedopticalﬂowCNNsdo
not always generalize to new datasets, which may hinder video object detection
performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi