  In this paper, we empirically study how to make the most of low-resolution
frames for efficient video recognition. Existing methods mainly focus on
developing compact networks or alleviating temporal redundancy of video inputs
to increase efficiency, whereas compressing frame resolution has rarely been
considered a promising solution. A major concern is the poor recognition
accuracy on low-resolution frames. We thus start by analyzing the underlying
causes of performance degradation on low-resolution frames. Our key finding is
that the major cause of degradation is not information loss in the
down-sampling process, but rather the mismatch between network architecture and
input scale. Motivated by the success of knowledge distillation (KD), we
propose to bridge the gap between network and input size via cross-resolution
KD (ResKD). Our work shows that ResKD is a simple but effective method to boost
recognition accuracy on low-resolution frames. Without bells and whistles,
ResKD considerably surpasses all competitive methods in terms of efficiency and
accuracy on four large-scale benchmark datasets, i.e., ActivityNet, FCVID,
Mini-Kinetics, Something-Something V2. In addition, we extensively demonstrate
its effectiveness over state-of-the-art architectures, i.e., 3D-CNNs and Video
Transformers, and scalability towards super low-resolution frames. The results
suggest ResKD can serve as a general inference acceleration method for
state-of-the-art video recognition. Our code will be available at
https://github.com/CVMI-Lab/ResKD.

Computer Vision and Pattern Recognition