it does not necessitate optical ﬂow training data. This renders our model easy to
train end-to-end. Our future work will include experimenting with more complex
design of spatiotemporal sampling blocks.
7 Acknowledgements
This work was funded in part by NSF award CNS-120552. We gratefully acknowledge NVIDIA and Facebook for the donation of GPUs used for portions
of this work.
14 G. Bertasius, L. Torresani, J. Shi
SSN (Static)STSN (Ours)
Sampled Points in the     Supporting Frame
Fig.6: A ﬁgure illustrating object detection examples where our spatiotemporal
sampling mechanism helps STSN to correct the mistakes made by a static SSN
baseline (please zoom-in to see the class predictions and their probabilities).
These mistakes typically occur due to occlusions, blurriness, etc. STSN ﬁxes
these errors by using relevant object level information from supporting frames.
InColumn 1weillustratethepointsinthesupportingframethatSTSNconsiders
relevant when computing the output for a point denoted by the green square in
Column 2.
Object Detection in Video with Spatiotemporal Sampling Networks 15
References
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi