Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution
Super resolving a low-resolution video is usually handled by either single-image super-resolution (SR) or multi-frame SR. Single-Image SR deals with each video frame independently, and ignores intrinsic temporal dependency of video frames which actually plays a very important role in video super-resolution. Multi-Frame SR generally extracts motion information, e.g., optical flow, to model the temporal dependency, which often shows high computational cost. Considering that recurrent neural networks (RNNs) can model long-term contextual information of temporal sequences well, we propose a bidirectional recurrent convolutional network for efficient multi-frame SR. Different from vanilla RNNs, 1) the commonly-used recurrent full connections are replaced with weight-sharing convolutional connections and 2) conditional convolutional connections from previous input layers to the current hidden layer are added for enhancing visual-temporal dependency modelling. With the powerful temporal dependency modelling, our model can super resolve videos with complex motions and achieve state-of-the-art performance. Due to the cheap convolution operations, our model has a low computational complexity and runs orders of magnitude faster than other multi-frame methods.
This paper proposes a bidirectional recurrent convolutional network for multi-frame super resolution. In a sense the paper formulates the problem in a way that I think most people familiar with RNNs would consider as a straightforward and sensible way to formulate the problem. The approach appears novel. The Dong et al. ECCV 14 work on learning deep convolutional networks for image super resolution appears to indeed be the most relevant recent work, but it was applied to static frames. This work seems like a very natural extension in the general trajectory of deep convents for super resolution. This combined with the fact that this work provides an evaluation of reasonable quality makes the paper acceptable for NIPS in my view.The most interesting aspect of this work is perhaps the exploration in Table 2 where the effect of feedforward convolutions, recurrent convolutions and conditional convolutions are evaluated. The statements about MATLAB vs Python when considering different explanations for the differences in run times seems like a side issue compared to the more important question of GPU acceleration. Modern convolutional neural networks are almost always implemented using GPUs in state of the art systems. This paper states on line 362 that the implementation of the approach presented here is in Python, but there is no discussion either way concerning the use of GPU acceleration.Please clarify this issue. If a GPU was not used, this method could be dramatically faster. If a GPU was used, then the comparison with prior work really needs to be cast in that light. Many prior methods are likely amenable to GPU acceleration as well.Accurate motion estimation in particular is given as the traditional bottleneck for non-RNN based methods. Such techniques could potentially be quite effectively accelerated with GPU methods.Language Issues:Abstract:* Considering that recurrent neural network[s] (RNNs) can... * Different from vanilla RNN[s]... * conditional convolutional connections from previous input layers to [the]current hidden layer are added for enhancing visual-temporal dependency mod[el].Please proof read the body text for other language issues.Conclusions: In the future, we will [perform] comparison[s] with [other] multi-frame SR methods