t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)
t;t+k)(p)j!
(4)
Finally,allweights warefedintothesoftmaxlayer,toensurethattheweights
sum up to 1at each pixel location p(i.e.,PK
k=Kwt;t+k(p) = 18p).
Object Detection. Finally, the aggregated feature tensor gagg
tis used as
input to the detection network, which outputs the ﬁnal bounding box predictions
and their object class probabilities. We describe more details related to the
detection network in the next section along with other implementation details.
4.1 Implementation Details
For our experiments we use the MXNet [28] library. Below we provide details
related to our STSN architecture, and our training and inference procedures.
Architecture. For our backbone network we adopt a state-of-the-art Deformable CNN [25] based on the ResNet-101 [5] architecture. Our spatiotemporal sampling block consists of four 33deformable convolutional layers each
with 1024output channels. In addition, it also has four 33convolutional layers
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi