performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it
does not suﬀer from this issue. Furthermore, our STSN can be trained for video
object detection in a single stage end-to-end. In comparison, methods that rely
on optical ﬂow require an additional stage to train an optical ﬂow CNN, which
renders the training procedure more cumbersome and lengthy. For example, we
notethatitwouldtakeaboutfourdaystotrainanopticalﬂowCNNofFGFA[24]
from scratch and then four additional days to train FGFA [24] for video object
detection, making it eight days of total training time. In contrast, our STSN is
trained in a single stage in only 4 days. Finally, we point out that our STSN also
yields a gain —albeit moderate— in video object detection accuracy.
4 G. Bertasius, L. Torresani, J. Shi
3 Background: Deformable Convolution
Before describing our method, we ﬁrst review some background information on
deformable convolution [25], which is one of the key components of our STSN.
Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi