  Two-stream networks have achieved great success in video recognition. A
two-stream network combines a spatial stream of RGB frames and a temporal
stream of Optical Flow to make predictions. However, the temporal redundancy of
RGB frames as well as the high-cost of optical flow computation creates
challenges for both the performance and efficiency. Recent works instead use
modern compressed video modalities as an alternative to the RGB spatial stream
and improve the inference speed by orders of magnitudes. Previous works create
one stream for each modality which are combined with an additional temporal
stream through late fusion. This is redundant since some modalities like motion
vectors already contain temporal information. Based on this observation, we
propose a compressed domain two-stream network IP TSN for compressed video
recognition, where the two streams are represented by the two types of frames
(I and P frames) in compressed videos, without needing a separate temporal
stream. With this goal, we propose to fully exploit the motion information of
P-stream through generalized distillation from optical flow, which largely
improves the efficiency and accuracy. Our P-stream runs 60 times faster than
using optical flow while achieving higher accuracy. Our full IP TSN, evaluated
over public action recognition benchmarks (UCF101, HMDB51 and a subset of
Kinetics), outperforms other compressed domain methods by large margins while
improving the total inference speed by 20%.

Computer Vision and Pattern Recognition