the FGFA [24] baseline, it is much harder to make a direct comparison between
STSN and D&T. Whereas our STSN aims to produce powerful spatiotemporal features, the method of D&T [27] is targeted more for smoothing the ﬁnal
bounding box predictions across time. Thus, we believe that these two methods
are complementary, and it would be possible to integrate them together for the
model that produces both: temporally smooth features, as well as temporally
smooth bounding box predictions. We also note that our STSN and D&T [27]
use slightly diﬀerent architectures (both based on ResNet-101 though).
First, we compare STSN and D&T in a setting when no temporal postprocessing(i.e.Seq-NMS[22],object-tubelinking[27],etc)isused,andshowthat
ourSTSNoutperformstheD&Tbaselinebyasubstantialmargin( 78:9vs75:8).
Object Detection in Video with Spatiotemporal Sampling Networks 11
These results indicate, that our STSN is able to learn powerful spatiotemporal
features, and produce solid video object detection results even without temporal
post-processing algorithms that link bounding box detections over time.
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi