Deep neural networks can learn meaningful representations of data. However, these representations are hard to interpret. For example, visualizing a latent layer is generally only possible for at most three dimensions. Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer. Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions. To do so, we propose a class of regularizations we call Graph Spectral Regularizations that impose graphstructure on latent layers. This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters. This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data. First, we show a synthetic example that the graph-structured layer can reveal topological features of the data. Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets. Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation. In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations. Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.
The paper introduces a spectral regularization with the aim of obtaining representations
that are easier to interpret.
Some sentences are often confusing and, in general, clarity needs to be improved.
The motivation of the work is not very strong in my opinion, in particular by adding such
a prior the space of possible solutions greatly shrinks and I am afraid
that interesting solutions will be lost. I think one should focus on properties
rather than visual inspection.
Also, isn't it that if we can clearly see the pattern, perhaps that pattern is
linear and of easy discovery also by simpler models?
More importantly, it seems that all experiments are performed on tasks where the
underlying structure is known, however this is almost never the case in practice.
Assuming one uses the proposed spectral regularization, how would one interpret
it in such cases?
In section 2 please clarify the paragraph on bounded Lp norm.
I am sorry but why isn't there a relation, for convolutional nets,
between neurons in different channels? Each element in the feature map represents
the input surrounding that location in a k dimensional space.
The authors state that the usual bottleneck for autoencoders is composed of 2/3
neurons, this is simply not true. There has been extensive work on
overcomplete representations that shows that is better to have many more dimensions
but only few degrees of freedom.
The spectral bottleneck should cite VQVAE as the approach is very similar and the 
authors should compare to it.
For the topological inference experiment it is assumed that one knows the structure,
but how to address the more general problem?
More practically, the regularization enforces smoothing (if few eigenfunctions
are used, which is never explained in the paper) between connected nodes, did
the authors try to have a simple L2 penalty instead? E.g. minimize the difference
between activations in the group.
Regarding the capsule network example, when you write that without regularization
each digit responds differently to perturbation of the same dimension, isn't it
possibly true only up to a, unknown, permutation of the neurons?
To summarize, while the idea sounds interesting, I miss to find the easy interpretability
of results and also the overall motivation sounds a bit weak. 
More importantly the selection of W, crucial for defining structure, is not discussed at all in the paper.
Experiments are performed on toy examples only whereas here, given that we can
possibly interpret the results I would have liked something more involved to
better show that this kind of interpretability is needed.
Missing cites:
[1] van den Oord et al, Neural Discrete Representation Learning.
[2] Koutnik et al, Evolving neural networks in compressed weight space.