post-processing scheme on top of our STSN predictions, we can further improve
our results and outperform all the other baselines.
to make its predictions. Furthermore, we also want to show that our spatiotemporal sampling scheme works as eﬀectively as the optical ﬂow network in [24],
but without requiring optical ﬂow supervision. To do so, we replace the optical
ﬂow network from [24], with our spatiotemporal sampling mechanism. The rest
of the architecture and the training details are kept the same for both baselines.
Such an experimental design allows us to directly compare the eﬀectiveness of
our spatiotemporal sampling scheme and the optical ﬂow network of FGFA [24].
Finally, we demonstrate that our method performs better than the D&T [27]
method in two scenarios: 1) when we only useCNN-level outputs forvideo object
detection, and also 2) when we allow temporal post-processing techniques such
as Seq-NMS to be applied on top of the CNN outputs. We note that in Table 1,
D&T [27] and STSN refer to the CNN-level baselines whereas D&T+ [27] and
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi