Learned D-AMP: Principled Neural Network Based Compressive Image Recovery
Compressive image recovery is a challenging problem that requires fast and accurate algorithms. Recently, neural networks have been applied to this problem with promising results. By exploiting massively parallel GPU processing architectures and oodles of training data, they can run orders of magnitude faster than existing techniques. However, these methods are largely unprincipled black boxes that are difficult to train and often-times specific to a single measurement matrix.
It was recently demonstrated that iterative sparse-signal-recovery algorithms can be "unrolled" to form interpretable deep networks. Taking inspiration from this work, we develop a novel neural network architecture that mimics the behavior of the denoising-based approximate message passing (D-AMP) algorithm. We call this new network Learned D-AMP (LDAMP).
The LDAMP network is easy to train, can be applied to a variety of different measurement matrices, and comes with a state-evolution heuristic that accurately predicts its performance. Most importantly, it outperforms the state-of-the-art BM3D-AMP and NLR-CS algorithms in terms of both accuracy and run time. At high resolutions, and when used with sensing matrices that have fast implementations, LDAMP runs over 50× faster than BM3D-AMP and hundreds of times faster than NLR-CS.
Summary of the work:
The authors present a theoretically motivated NN architecture for solving inverse problems arising in compressed sensing. The network architecture arises from ‘unrolling’ the denoiser-based approximate message passing (D-AMP) algorithm with convolutional denoiser networks inside. The work is nicely presented, and to my knowledge, original. The authors present competitive results on known benchmark data.
Overall evaluation:
I enjoyed reading this paper, well presented, theoretically motivated work. I was pleased to see that the method achieves very competitive performance in the end both in terms of accuracy and in terms of speed.
Comments and questions:
1. Why is the Onsager correction so important?: The Onsager correction is needed because the error distribution differs from the white Gaussian noise assumption typically used to train/derive denoising methods. However, if the whole D-IT unrolled network is trained (or only fine-tuned after initial training) end-to-end, then surely, subsequent denoisers can adopt to the non-Gaussian noise distribution and learn to compensate for the bias themselves. This would mean that the denoisers would no longer be good denoisers when taken out of context, but as part of the network they could co-adapt to compensate for each other’s biases. If this is not the case, why?
2. Validation experiments for layer-wise-training: My understanding is that all experiments use the layer-wise training procedure, and there is no explicit comparison between layer-wise or end-to-end training. Is this correct?
3. Connections to the theory denoising autoencoders: This paper is presented from the perspective of compressive sensing, with solid theory, and unsurprisingly most of the references are to work from this community/domain. However, denoisers are applied more generally in unsupervised learning, and there is a relevant body that the NIPS community will be familiar with which might be worth discussing in this paper. Specifically, (Alain and Bengio, 2012) https://arxiv.org/abs/1211.4246 show that neural networks trained for denoising learn approximate the gradients of the log data generating density. Therefore, one could implement iterative denoising as “taking a gradient step” along the image prior. This is exploited in e.g. (Sonderby et al, 2017) https://arxiv.org/abs/1610.04490 where the connection to AMP is noted - although not discussed in detail.
4. Lipschitz continuity remark: the authors say ConvNets are clearly Lipschitz, otherwise their gradients would explode during training. While this is true, and I would assume doesn’t cause any problems in practice, technically, the Lipschitz constant can change with training, and is unbounded. A similar Lipschitz-continuity requirement arises in - for example - the Wasserstein GAN’s discriminator network, where Lipschitz continuity with a certain constant is ensured by weight clipping. Can the authors comment on the effect of the Lipschitz constant’s magnitude on the findings?
Minor comments:
Table 1. would benefit from highlighting the fastest/best methods (and those that come within one confidence interval from the winner)