Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)
Fig.5: An illustration of using our spatiotemporal sampling scheme in action.
The green square indicates a ﬁxed object location in the reference frame. The
red square depicts a location in a supporting frame, from which relevant features
are sampled. Even without optical ﬂow supervision, our STSN learns to track
these objects in video. In our supplementary material, we include more of such
examples in the video format.
where the bicycle is more clearly visible. Similar behavior also occurs in other
cases where STSN successfully resolves occlusion and blurriness issues.
6 Conclusion
Inthiswork,weintroducedtheSpatiotemporalSamplingNetwork(STSN)which
is a new architecture for object detection in video. Compared to the state-of-theart FGFA [24] method, our model involves a simpler design, it does not require
optical ﬂow computation and it produces higher video object detection accuracy.
Ourmodelisfullydiﬀerentiable,andunlikepriorvideoobjectdetectionmethods,
it does not necessitate optical ﬂow training data. This renders our model easy to
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi