arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,
our STSN learns to sample from the supporting frame at locations that coincide almost perfectly with the same object. This allows our method to perform
accurate object detection even if objects in the reference frame are blurry or
occluded.
In addition, based on the results in Figure 4, we observe that even without
an explicit optical ﬂow supervision, our STSN learns to accurately capture the
motion of the objects, which is another appealing property of our model. In
fact, in Figure 5, we illustrate several examples of using our STSN to track
objects in a given video. From Figure 5, we observe that despite a relatively
large motion in each sequence, our STSN accurately samples features around
objects in every supporting frame. Such results indicate that we may be able
to use our sampling mechanism for discriminative object tracking. In fact, we
note that the commonly used dense optical ﬂow methods are often redundant
because most applications do not require ﬂow prediction for every single pixel. In
comparison, we point out that our STSN captures a more discriminative form of
motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi