motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.
In Figure 6, we also illustrate object detections of the static SSN baseline,
and those of our full STSN model (zoom-in to see the probabilities and class
predictions). In all of these cases, we observe that incorporating temporal information helps STSN to correct the mistakes made by the static baseline. For
instance, in the third row of Figure 6, a static SSN baseline incorrectly labels
an object in the reference frame as a bird, which happens due to the occluded
head of the lizard. However, STSN ﬁxes this mistake by looking at the supporting frames, and by sampling around the lizard body and its head (See Row 3,
Column 1 in Figure 6). Furthermore, in the last row, a static SSN baseline fails
to detect one of the bicycles because it is occluded in the reference frame. STSN
ﬁxes this error, by sampling around the missed bicycle in the supporting frame
Object Detection in Video with Spatiotemporal Sampling Networks 13
Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi