Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner workings of such computational brains is both fascinating basic science that is interesting in its own right-similar to why we study the human brain-and will enable researchers to further improve DNNs. One path to understanding how a neural network functions internally is to study what each of its neurons has learned to detect. One such method is called activation maximization (AM), which synthesizes an input (e.g. an image) that highly activates a neuron. Here we dramatically improve the qualitative state of the art of activation maximization by harnessing a powerful, learned prior: a deep generator network (DGN). The algorithm (1) generates qualitatively state-of-the-art synthetic images that look almost real, (2) reveals the features learned by each neuron in an interpretable way, (3) generalizes well to new datasets and somewhat well to different network architectures without requiring the prior to be relearned, and (4) can be considered as a high-quality generative method (in this case, by generating novel, creative, interesting, recognizable images).
The authors adopted a pretrained GAN network as prior for visualizing preferred inputs for deep neurons.
Visual quality of preferred inputs obtained using the proposed method is significantly improved compared to existing methods. Generalizability of the method is also supported by extra experiments. Overall the paper should be interesting to the deep learning community. However, there're also many key questions left untouched. For example, is visual interpretability a sufficient and necessary condition for good network performance? How exactly can we use visual interpretability to improve a network? Also, what are the differences in neural responses induced by natural images vs. preferred inputs generated by existing/proposed methods? The main text also cites supplementary materials very frequently, which might not be very convenient for readers. The authors may consider rearrange their manuscript (e.g. moving details of methods into supplementary materials).  %% After Rebuttal %% (No score change.) As most of the reviewers (including myself) agreed that this paper is at least poster worthy for its comprehensive experiments & visually impressive results, I do however want to discuss about the general direction/motivation of this work, corresponding to the authors' response -- "we found no correlation between the activation of a neuron and the recognizability of its visualization" (also in main text):  A deep CNN is essentially a nonlinear function that maps an input image to a real value (a target neuron being studied), and activation maximization (AM) is generally considered useful since it tries to discover what images drive a target neuron the most, and those images can be interpreted as "the meaning(s)" of the neuron. However, if like the authors said, recognizability has no correlation against the functional value (activation) of the function (DCNN) being studied, how can we be convinced that better visual recognizability (this work) is a substantial improvement to existing AM methods in helping us better understand these functions? Put another way, how much more understanding about DCNNs have we gained using the proposed method, compared to using methods from e.g. Simonyan et al., whose results are described as uninterpretable (but may actually induce higher activations, since less regularization on the input space is imposed -- unfortunately one of my questions not answered by the authors)? My arguments are in no way trying to reject this paper, but hopefully to help the authors reevaluate their view on AM/preferred inputs and on toward understanding DCNNs. That is, preferred inputs (alone) does not describe/characterize a neuron very well. For example, (Machens et al.) gave a nice example of the efficient coding hypothesis (Barlow; Simoncelli) that the (most) preferred inputs of a grasshopper's auditory neuron would be something completely out of its natural environment, and what really matters should be how well the neuron's tuning curve matches the statistics of natural inputs. Similarly, preferred inputs of DCNN neurons may not be revealing very much about those neurons, even though we feel visually they are. I hope this example can help the authors further improve their paper, especially the discussion/future direction section (but up to the authors' discretion).  Machens et al., "Testing the efficiency of sensory coding with optimal stimulus ensembles," Neuron, 2005 Barlow, "Possible principles underlying the transformations of sensory messages," Sensory Communication, 1961 Simoncelli, "Vision and the statistics of the visual environment," Current Opinion in Neurobiology, 2003