Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference
frame. Conceptually, it replaces the optical ﬂow used in [24] to establish temporal correspondences with a learnable module that is discriminatively trained
from object detection labels. In our experimental section, we show that such a
sampling scheme allows us to improve video object detection performance over
the still-image baseline and the ﬂow-based method of Zhu et al. [24] without
training our model on optical ﬂow data.
Feature Aggregation. The spatiotemporal sampling procedure is applied
for all the supporting frames in the selected range. Note, that this includes a
Object Detection in Video with Spatiotemporal Sampling Networks 7
special case, when the reference frame is treated as a supporting frame to itself to
produceg(4)
t;t, which is a feature tensor computed from only the reference frame.
The resulting feature tensors have the following form: g(4)
t;t+k2Rc(4)hw.
Thesefeaturetensorsareaggregatedintoanoutputfeaturetensor gagg
t2Rc(4)hw
for the reference frame. This tensor captures information from the reference
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi