with 1024output channels. In addition, it also has four 33convolutional layers
predicting (x;y)oﬀsets. To implement a subnetwork S(x)that predicts feature
aggregation weights, we use a sequence of 11,33and11convolutional
layers with 512;512and2048output channels respectively. Our detection network is implemented based on the deformable R-FCN design [17,29,25]. When
feeding the aggregated feature gagg
tto the detection network, we split its 1024
channels into two parts, and feed the ﬁrst and the last 512channels to the
RPN and R-FCN sub-networks respectively. For the RPN, we use 9anchors and
300proposals for each image. Furthermore, for the R-FCN, we use deformable
position-sensitive ROI pooling with 77groups.
8 G. Bertasius, L. Torresani, J. Shi
Training. Our entire STSN model is fully diﬀerentiable, and thus, trainable
end-to-end. During training, we resize all input images to a shorter side of 600
pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi