Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling
Nonlocal neural networks [25] have been proposed and shown to be effective in several computer vision tasks, where the nonlocal operations can directly capture long-range dependencies in the feature space. In this paper, we study the nature of diffusion and damping effect of nonlocal networks by doing spectrum analysis on the weight matrices of the well-trained networks, and then propose a new formulation of the nonlocal block. The new block not only learns the nonlocal interactions but also has stable dynamics, thus allowing deeper nonlocal structures. Moreover, we interpret our formulation from the general nonlocal modeling perspective, where we make connections between the proposed nonlocal network and other nonlocal models, such as nonlocal diffusion process and Markov jump process.
This paper studies the damping effect of a non-local neural network (Want et al., CVPR 2018) by performing spectrum analysis on the weight matrices obtained after well training. Given the eigenvalues of the weight matrices, the paper claimed that non-local blocks, different from local learning, have the nature of diffusion. The authors then proposed an alternative formula of the non-local blocks, which is inspired by some previous classical non-local modeling works (Buades et al. 2005, Coifman et al. 2006, Gilboa et al. 2007, Du et al. 2012). A similarity between the proposed non-local operation and other non-local models, i.e., nonlocal diffusion process and Markov jump process, has been shown in the paper to demonstrate the diffusive nature and stability of the proposed non-local blocks. The experimental results also showed that the modified non-local network not only gets higher accuracy on image classification tasks than the original one, but also is more robust to multiple blocks.  Strengths: 1) The paper is well written and organized, and technically correct. 2) I appreciate the connections with a wide range of existing non-local models. It is always an enjoy to read papers that try to find theoretical justifications for deep neural architectures. In this sense, the paper is a refreshing read. Although the spectrum analysis technique has been widely studied and applied, using it to interpret deep learning seems useful yet simple. 3) Other than the theoretical findings, another contribution of this paper is that it provides a modified non-local network architecture. Although the two formulations look similar, I am convinced by the justifications in Section 4 of the paper and Section 3 of the supplementary material that the modification makes the whole system more stable and well-defined. 4) The experiments are self-contained and kept to the point. In the original paper of non-local network (Wang et al., CVPR 2018), although it was investigated in many datasets and tasks that introducing non-local blocks can improve the learning performance, the non-local blocks are always added to different places. While in this paper, the authors studied the non-local network from a different perspective, namely stacking multiple blocks to the same place, serving for the study of stability and robustness. Then the bottom results in Table 1 also gave a comparison of the two non-local networks with non-local blocks incorporated into different places.  Weaknesses: Although this paper paid more attention to the theoretical justification of the non-local network architecture, and the empirical results are presented to validate the theoretical findings, I am curious about the application of the modified non-local network. Since in (Wang et al., CVPR 2018), the authors provided complete experiments on a few tasks and datasets, especially the video classification on Kinects. I am willing to see how the modified non-local network performs on such a task with multiple non-local blocks.  Overall, the tools used in this paper are useful and extendable. The topic raised in the paper is interesting to the community. The proposed model architecture is incremental, but the theoretical study is original and refreshing. The significance of the paper does not lie in the improvement of experimental performance, but a new angle of studying the neural network architecture. Using some mathematical tools, such as PDEs, ODEs, and stochastic processes, to interpret and explore deep learning techniques seems promising.   This is a nice paper. The rebuttal makes the research idea clearer. I keep my vote to accept this paper.