RESIDUAL NON-LOCAL ATTENTION NETWORKS FOR IMAGE RESTORATION
In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial-and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block. The trunk branch is used to extract hierarchical features. Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions. The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map. Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network. Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution. Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.
The paper proposes a convolutional neural network architecture that includes blocks for local and non-local attention mechanisms, which are claimed to be responsible for achieving excellent results in four image restoration applications.
# Results
The strongest point of the paper is that the quantitative and qualitative image restoration results appear to be very good, although they seem almost a bit too good.
# Novelty
I'm not sure about the novelty of the paper, but I suspect it to be rather incremental. The paper says "To the best of our knowledge, this is the first time to consider residual non-local attention for image restoration problems." Does that mean non-local attention (in a very similar way) has already been used, just not in a residual fashion? If so, that would not constitute much novelty. I have to admit that I'm not familiar with the related work on attention, but I did not understand *why* the results of the proposed method are supposed to be much better than that of previous work.
# Clarity
I think the paper is not self-contained enough, since it seems to implicitly assume substantial background knowledge on attention mechanisms in CNNs. 
Furthermore, the introduction of the paper identifies three problems with existing CNNs that I don't necessarily fully agree with. None of these supposed problems are backed up by (experimental) evidence.
I don't think it is sufficient to just show superior results than previous methods. It is also important to disentangle why the results are better. However, the presented ablation experiments are not very illuminating to me.
The attempts at explaining what the novel attention blocks do and why they lead to superior results are very vague to me. Maybe they are understandable in the context of related work, but I found many statements, such as the following, devoid of meaning:
- "Without considering the uneven distribution of information in the corrupted images, [...]"
- "However, in this paper, we mainly focus on learning non-local attention to better guide feature extraction in trunk branch."
- "We only incorporate residual non-local attention block in low-level and high-level feature space. This is mainly because a few non-local modules can well offer non-local ability to the network for image restoration."
- "The key point in mask branch is how to grasp information of larger scope, namely larger receptive field size, so that it’s possible to obtain more sophisticated attention map."
# Experiments
- The experimental results are the best part of the paper. However, it would've been nice to include some qualitative results in the main paper.
- The proposed RNAN model is trained on a big dataset (800 images with ~2 million pixels each). Are the competing methods trained on datasets of similar size? If not, this could be a major reason for improved performance of RNAN over competing methods. At least in the appendix, RNAN and FFDNet are compared more fairly since they are trained with the same/similar data.
- The qualitative examples in the appendix mostly show close-ups/details of very structured regions (mostly stripy patterns). Please also show some other regions without self-similar structures.
# Misc
- Residual non-local attention learning (section 3.3) was not clear to me.
- The word "trunk" is used without definition or explanation.
- Fig. 2 caption is too short, please expand.
# Update (2018-11-29)
Given the substantial author feedback, I'm willing to raise my score.