Multi Memory Convolutional Neural Network for Video Super Resolution
Video super resolution (SR) is focused on reconstructing high resolution frames from consecutive low resolution (LR) frames. Most previous video SR methods based on convolutional neural networks (CNN) use a direct connection and single memory module within the network, and thus, they fail to make full use of spatio temporal complementary information from LR observed frames. To fully exploit spatio temporal correlations between adjacent LR frames and reveal more realistic details, this paper proposes a multi memory CNN (MMCNN) for video SR, cascading an optical flow network and an image reconstruction network. A series of residual blocks engaged in utilizing intra frame spatial correlations is proposed for feature extraction and reconstruction. Particularly, instead of using a single memory module, we embed convolutional long short term memory into the residual block, thus forming a multi memory residual block to progressively extract and retain inter frame temporal correlations between the consecutive LR frames. We conduct extensive experiments on numerous testing datasets with respect to different scaling factors. Our proposed MMCNN shows superiority over the state of the art methods in terms of PSNR and visual quality and surpasses the best counterpart method by 1 dB at most.
IEEE Transactions on Image Processing