t2Rc(4)hw
for the reference frame. This tensor captures information from the reference
frame, itsKpreceding frames and its Ksubsequent frames. The output tensor
valuegagg
t(p)for frametat pixelpis computed as a weighted summation:
gagg
t(p) =KX
k=Kwt;t+k(p)g(4)
t;t+k(p) (3)
Inspired by strong results presented in [24], we use their proposed feature
aggregation method where the weights windicate the importance of each supporting frame to the reference frame. To compute the weights w, we attach a
3-layer subnetwork S(x)to the features g(4)
t;t+kand then compute their intermediate feature representations S(g(4)
t;t+k). We then obtain the weights wby applying
an exponential function on the cosine similarity between each corresponding
feature point in a reference frame and a supporting frame:
wt;t+k(p) = exp 
S(g(4)
t;t)(p)S(g(4)
t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
Gedas Bertasius Lorenzo Torresani Jianbo Shi