of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that
a nearby frame It+kincludes the same object clearly visible and in a relatively
standard pose. If we only had access to It, accurate object detection would be
very challenging. However, leveraging information from It+kmay enable more
robust detection in the frame It. Thus, the main challenge in this setting is
incorporating object-level information from the supporting frameIt+kfor an improved object detection accuracy in the reference frameIt. Note that in our
system each frame in the video is treated in turn as a reference frame in order to
produce object detection in every frame of the video. Furthermore, in practice
Object Detection in Video with Spatiotemporal Sampling Networks 5
Reference Frame (time t)
Supporting Frame (time t+k)
Def. Conv.OffsetsDef. Conv.Offsets
Sampled Feature Map
Concat
Fig.2: Our spatiotemporal sampling mechanism, which we use for video object
detection. Given the task of detecting objects in a particular video frame (i.e.,
a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi