THE NEURO-SYMBOLIC CONCEPT LEARNER: INTERPRETING SCENES, WORDS, AND SENTENCES FROM NATURAL SUPERVISION
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

To achieve the state-of-the-art on the CLEVR and the variations of this, the authors propose a method to use object-based visual representations and a differentiable quasi-symbolic executor. Since the semantic parser for a question input is not differentiable, they use REINFORCE algorithm and a technique to reduce its variance.
Quality:
The issue of invalid evaluation should be addressed. CLEVR dataset has train, validation, and test sets. Since the various hyper-parameters are determined with the validation set, the comparison of state-of-the-art should be done using test set. As the authors mentioned, REINFORCE algorithm may introduce high variance, this notion is critical to report valid results. However, the authors only report on the validation set in Table 2 including the main results, Table 4. For Table 5, they only specify train and test splits. Therefore, I firmly recommend the authors to report on the test set for the fair comparison with the other competitive models, and please describe how to determine the hyperparameters in all experimental settings.

Clarity:
As mentioned above, please specify the experimental details regarding setting hyperparameters.
In Experiments section, the authors used less than 10% of CLEVR training images. How about to use 100% of the training examples? How about to use the same amount of training examples in the competitive models? The report is incomplete to see the differential evident from the efficient usage of training examples.
Originality and significance:
The authors argue that object-based visual representation and symbolic reasoning are the contributions of this work (excluding the recent work, NS-VQA < 1 month). However, bottom-up and top-down attention work [1] shows that attention networks using object-based visual representation significantly improve VQA and image captioning performances. If the object-based visual representation alone is the primary source of improvement, it severely weakens the argument of the neuro-symbolic concept learner. Since, considering the trend of gains, the contribution of the proposing method seems to be incremental, this concern is inevitable. To defend this critic, the additional experiment to see the improvement of the other attentional model (e.g, TbD, MAC) using object-based visual representations, without any other annotations, is needed.
Pros:
- To confirm the effective learning of visual concepts, words, and semantic parsing of sentences, they insightfully exploit the nature of the CLEVR dataset for visual reasoning diagnosis.
Cons:
- Invalid evaluation to report only on the validation set, not test set.
- The unclear significance of the proposed method combining object-based visual representations and symbolic reasoning
- In the original CLEVR dataset paper, the authors said "we stress that accuracy on CLEVR is not an end goal in itself" and "..CLEVR should be used in conjunction with other VQA datasets in order to study the reasoning abilities of general VQA systems." Based on this suggestion, can this work generalize to real-world settings? This paper lacks to discuss its limitation and future direction toward the general problem settings.
Minor comments:
In 4.3, please fix the typos, "born" -> "brown" and "convlutional" -> "convolutional".
[1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. IEEE Computer Vision and Pattern Recognition (CVPR'18).


We propose to tackle the mode collapse problem in generative adversarial networks (GANs) by using multiple discriminators and assigning a different portion of each minibatch, called microbatch, to each discriminator. We gradually change each discriminator's task from distinguishing between real and fake samples to discriminating samples coming from inside or outside its assigned microbatch by using a diversity parameter α. The generator is then forced to promote variety in each minibatch to make the microbatch discrimination harder to achieve by each discriminator. Thus, all models in our framework benefit from having variety in the generated set to reduce their respective losses. We show evidence that our solution promotes sample diversity since early training stages on multiple datasets.

The submission proposes to increase the variety of generated samples from GANs by a) using an ensemble of discriminators, and b) tasking them with distinguishing not only fake from real samples, but also their fake samples from the fake samples given to the respective other discriminators. The cost function of the minimax GAN optimization problem is changed accordingly. Experimental results suggest that this approach leads to improved results, both visually and w.r.t. FID metric.
Improving on the well known problem of mode collapse in GAN training scenarios is without a doubt an important endeavor. Various methods have been proposed, as curtly summarized in Section 1.1 of the submission.
With respect to the proposed method, I am not completely clear on how it can increase sample variety, if it does. In understand the arguments brought forward in Appendix B, but:
Consider a minimax game involving one generator G and one discriminator D, where each batch of generated samples from G(Z), is split up into two parts, A and B, via selection without replacement using uniform sampling. Z is a matrix of noise inputs, where each column corresponds to one item of the batch. D is now tasked to differentiate whether a sample came from A or B. It seems intuitive to say that in this case, D can neither win, nor provide any useful signal to G, since the sets A and B were split randomly, and there is no influence on G during training. The variety of samples in A as well as in B will be identical to the variety in the set (A and B).
Yet this random microbatch splitting is what seems to be happening here, if I understood Section 3 correctly; just with an ensemble of discriminators, and not just with one.
While it is thus not completely clear to me *why* the proposed additional term seems to bring increased variety, experiments strongly suggest that it does.
As described in Section 3.1, choice of the weighting parameter alpha seems crucial, and additionally alpha needs to depend on the iteration index. Different schedules are demonstrated, but optimality of either is not guaranteed. This makes the actual influence of the additional loss term even harder to judge and evaluate.
Section 4.1 seems to confirm the increase in variety via the self-defined "Intra FID" measure. I would have liked to see this measure evaluated on conventionally trained GANs as a baseline, as well on the methods compared to in Section 5.
In Table 1, both min and mean FID are given over 50k iterations. Instead of reporting the minimum, it might be fairer to compare FIDs after a fixed number of iterations (i.e. 50k in this case).
The method comparison in Section 5 is generally appreciated, but I think some of its flaws are:
- Datasets, including ImageNet, are all downsampled to 32x32 pixels. We have seen generators in recent work that produce interesting high-resolution output in even megapixel size; the tiny size seems like a pessimization of overall approaches.
- The proposed method is compared to other methods using only 2 discriminators, although Section 4.3 suggest a larger number is better.
- MicroGAN does not compare favorably to many of the compared to methods in Table 2. This may not necessarily by a flaw of the MicroGAN contributions, but is rather a problem of an apples-to-oranges comparison, as the authors readily admit ("the use of more powerful architectures [...] plays a big role"). I question the value of such a comparison, if not only the method differ, but also implementation details such as network architectures.
Overall the submission is quite interesting, but not without the above-mentioned flaws.


Deep reinforcement learning (RL) algorithms have shown an impressive ability to learn complex control policies in high-dimensional environments. However, despite the ever-increasing performance on popular benchmarks like the Arcade Learning Environment (ALE), policies learned by deep RL algorithms can struggle to generalize when evaluated in remarkably similar environments. These results are unexpected given the fact that, in supervised learning, deep neural networks often learn robust features that generalize across tasks. In this paper, we study the generalization capabilities of DQN in order to aid in understanding this mismatch between generalization in deep RL and supervised learning methods. We provide evidence suggesting that DQN overspecializes to the domain it is trained on. We then comprehensively evaluate the impact of traditional methods of regularization from supervised learning, ℓ 2 and dropout, and of reusing learned representations to improve the generalization capabilities of DQN. We perform this study using different game modes of Atari 2600 games, a recently introduced modification for the ALE which supports slight variations of the Atari 2600 games used for benchmarking in the field. Despite regularization being largely underutilized in deep RL, we show that it can, in fact, help DQN learn more general features. These features can then be reused and fine-tuned on similar tasks, considerably improving the sample efficiency of DQN.

I totally disagree with the authors that any of their observations are surprising. Indeed the fact that an RL agent does not generalizes to small modifications of the task (either visual or in the dynamics) is well known. If the agent should generalize though is a different question. And I do not mean this in the sense that it is an undesirable property but rather if it is outside of what “learning one task” means. Particularly I feel this is a very pessimistic view of RL and potentially not even in-line with what happens in supervised learning.
I think one mantra of deep learning (and deep RL needs to obey by it) is that one should test in the same setting as training for things to truly work. For supervised, there is a distribution of data, and the test set are samples from the same distribution. However the testing scenario used here is slightly different. During training, if I do not see car accelerating, I think it makes no sense to expect to generalize to a new game that has this property as it is out-of-distribution. Of course it would be ideal if it could do that. And to clarify, while for us some of these extensions seem very similar and minimal changes, hence it should generalize to rather than transfer to, this is just the effect of imposing our own biases on the learning process. Deep Nets do not learn like we do, and in their universe they have never seen a car accelerating -- it makes sense that it might not to be able to generalize to it. Again, I’m not arguing that we don’t want this, but rather if we should expect it as part of what the system should normally generalize to.
To that end I think this paper enters in that unresolved dispute of what generalization should be versus what is transfer. At what point do we have truly a new task vs a sample from the same task. I don’t think there is an answer.
Going back to the observations in this work. I think the fact that the environment is not stochastic reinforces this overfitting (as in the extreme you end up with a policy that just repeats the optimal sequence of actions). I think in this particular case I can see how finetuning to a variation of the task fails. However true stochasticity in the environment (e.g. having a distribution of variations) like is done in Distral paper (where each episode is a different layout) can behave as a regularizer that will mitigate a bit the overfitting. That is to say that I believe the observed behaviour will be less pronounced in complex stochastic setting.
Nevertheless the paper seems to highlight an important observation (and back it up with empirical evidence), namely we should use more regularization like L2 or otherwise in practice. Which is mostly absent from publications. And I think this on its own is valuable.


Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use lowrank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts.

The paper proposes an input-dependent low rank approximations of weight matrices in neural nets with a goal of accelerating inference. Instead of decomposing a matrix (a layer) to a low rank space like previous work did, the paper proposes to use a linear combination/mixture of multiple low rank spaces. The linear combination is dynamic and data dependent, making it some sense of non-linear combination. The paper is interesting, however,
I doubt its significance at three aspects:
(1) computation efficiency: the primary motivation of this paper is to accelerate inference stage; however, it might not be wise to break computation in a single low-rank space to segments in multiple low-rank spaces. In the original low-rank approximation, only two matrix-vector multiplications are needed, but this paper increases it to 2*K plus some additional overheads. Although the theoretical FLOPs can be cut, but modern hardware runs much faster when a whole bulk of data are computed all together. Because of this, the primary motivation in this paper wasn't successfully supported by wall-clock time;
(2) low-rank approximation: low-rank approximation only makes sense when matrices are known and redundant, otherwise, no approximation target exists (i.e., what matrix is it approximating?). Because of this, low-rank neural nets [1][2] start from trained models, approximate it and fine-tune it, while this method trains from scratch without an approximation target. Although, we can fit the method to approximate trained matrices, then decomposing a matrix to a mixture of low-rank spaces is equivalent to decomposing to one single low-rank space (the only difference is the combination is data dependent). Therefore, I view this paper more in a research line of designing compact neural nets, which brings me to a concern in (3).
(3) efficient architecture design: essentially, the paper proposes a class of compact neural nets, at each layer of which there are K "low-rank" branches with a gating mechanism to select those branches. However, branching and gating is not new [3][4]. I like the results in Table 2, but, to show the efficiency, it should have been compared with more SOTA compact models like CondenseNet.

Clarity:
How FLOPs reduction are exactly calculated? I am not convinced by FLOPs reduction in the LSTM experiments, since in LSTM (especially in language modeling), FLOPs in the output layer are large because of a large vocabulary size (650x10000 in the experiments). However, output layer is not explicitly mentioned in the paper.
Improvement:
(1) Accuracy improvement in Table 1 is not statistically significant, but used more parameters. For example, an improvement of 93.01% over 92.92% is within an effect of training noise;
(2) It is a little hacking to conclude that a random matrix  P_random has a small storage size because we can storage a seed for recovery. When we deploy models across platforms, we cannot guarantee they use the same random generator and has a consistent implementation;
(3) sparse gating of low-rank branches may make this method more computation efficient.
[1] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems (NIPS). 2014.
[2] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference (BMVC), 2014.
[3] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going deeper with convolutions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015.
[4] Huang, Gao, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. "Multi-scale dense networks for resource efficient image classification." arXiv preprint arXiv:1703.09844 (2017).


Probability distillation has recently been of interest to deep learning practitioners as it presents a practical solution for sampling from autoregressive models for deployment in real-time applications. We identify a pathological optimization issue with the commonly adopted stochastic minimization of the (reverse) KL divergence, owing to skewed gradient distribution due to curse of dimensionality. We also explore alternative principles for distillation, and show that one can achieve qualitatively better results than with KL minimization.

This paper proposes new methods for distilling a feed-forward generative model (student) from an autoregressive generative model (teacher) as an alternative to the reverse-KL divergence. The first part of the paper analyses optimization issues with the reverse KL divergence while in the second part of the paper alternatives are proposed (x-reconstruction and z-reconstruction).
Detailed comments:
1.
In abstract and other places: "sparse gradient signal from the teacher".
Sparsity implies that many of the values are exactly zero, while Section 3.1 seems to imply that some of the values might be small (or pointing towards the origin).
2.
In Section 3.1 and 3.2 the authors discuss a potential failure mode of the reverse KL:
But, proposition 1 boils down to the fact that if the student's mass is more spread out than the teacher is some direction, that it should shrink that mass closer to zero as well.
In the example of the paper: if an eigenvalue of T is smaller than 1, it would mean that the student which is spherical Gaussian, would adjust its probability mass to also be smaller in that eigenvector's direction.
As training progresses, the students mass would be much closer to the teacher and the probability of 'pointing away' from the origin would be about as likely as pointing towards.
So it's not clear at all that the described property is problematic for optimization, as it could as well be interpreted as the student trying to fit the teacher's distribution better.
3.
Was the KL between P_S(x_i | z_<i) and P_T(x_i | x_<i) computed analytically? If these conditional distributions are Gaussian (which they are in many of the examples) this should be trivial.
4.
Section 4 about the neural vocoder needs to be expanded: many details are missing here and although it's one of the more important experiments in the paper it's relatively neglected compared to the other parts of the paper.
5.
In the Section 4: the experiment with reverse-KL is a straw man comparison: For audio the reverse KL was only proposed in combination with the power loss (Oord et al). Two additional experiments would make the result a lot stronger: KL+power-loss and X-recon+power-loss. Because if the x-recon method does not work well together with the power-loss, its practical applicability seems limited.
The proposed methods are interesting, because they are elegant and seems to work reasonably well on the tasks tried. The first part of the paper about gradient sparsity/orientation needs to be addressed. Section 4 should be expanded and an additional comparison should be made.
I would change my rating if these issues were addressed.


Despite recent progress, Generative Adversarial Networks (GANs) still suffer from training instability, requiring careful consideration of architecture design choices and hyper-parameter tuning. The reason for this fragile training behaviour is partially due to the discriminator performing well very quickly; its loss converges to zero, providing no reliable backpropagation signal to the generator. In this work we introduce a new technique -progressive augmentation of GANs (PA-GAN) -that helps to overcome this fundamental limitation and improve the overall stability of GAN training. The key idea is to gradually increase the task difficulty of the discriminator by progressively augmenting its input space, thus enabling continuous learning of the generator. We show that the proposed progressive augmentation preserves the original GAN objective, does not bias the optimality of the discriminator and encourages the healthy competition between the generator and discriminator, leading to a better-performing generator. We experimentally demonstrate the effectiveness of the proposed approach on multiple benchmarks (MNIST, Fashion-MNIST, CIFAR10, CELEBA) for the image generation task.

This paper proposes a new trick to improve the stability of GANs. In particular the authors try to tackle the vanishing gradient problem in GANs, when the discriminator becomes to strong and is able to perfectly separate the distribution early in training, resulting in almost zero gradient for the generator. The authors propose to increase the difficulty of the task during training to avoid the discriminator to become too strong.
The paper is quite well written and clear. However there is several unsupported claims (see below).
A lot of work has been proposed to regularize the discriminator, it's not clear how different this approach is to adding noise to the input or adding dropout to the discriminator.
Pros:
- The experimental section is quite thorough and the results seem overall good.
- The paper is quite clear.
Cons:
- There is a major mistake in the derivation of the proposed method. In eq. (6) & (7), (c) is not an equivalence, minimizing the KL divergence is not the same as minimizing the Jensen-Shannon divergence. The only thing we have is that: KL(p||q) = 0 <=> JSD(p||q) = 0 <=> p=q . The same kind of mistake is made for (d). Note that the KL-divergence can also be approximated with a GAN see [1]. Since the equivalence between (6) and (7) doesn't hold, the equation (11) doesn't hold either.
- The authors say that the discriminator can detect the class of a sample by using checksum, the checksum is quite easy for a neural networks to learn so I don't really see how the method proposed actually increase the difficulty of the task for the discriminator. Especially if the last layer of the discriminator learns to perform a checksum, and the discriminator architecture has residual connections, then it should be straight-forward for the discriminator to solve the new task given it can already solve the previous task. So I'm not sure the method would still works if we use ResNet architecture for the discriminator.
- I believe the approach is really similar to adding noise to the input. I think the method should be compared to this kind of baseline. Indeed the method seems almost equivalent to resetting some of the weights of the first layer of the discriminator when the discriminator becomes too strong, so I think it should also be compared to other regularization such as dropout noise on the discriminator.
- The authors claim that their method doesn't "just memorize the true data distribution". It's not clear to me why this should be the case and this is neither shown theoretically or empirically. I encourage the author to think about some way to support this claim.
- The authors states that "adding high-dimensional noise introduces significant variance in the parameter estimation, which slows down training", can the author give some references to support that statement ?
- According to the author: "Regularizing the discriminator with the gradient penalty depends on the model distribution, which changes during training and thus results in increased runtime". While I agree that computing the gradient penalty slightly increase the runtime because we need to compute some second order derivatives, I don't see how these increase of runtime is due to change in the model distribution. The authors should clarify what they mean.
Others:
- It would be very interesting to study when does the level number increase and what happens when it increase ? Also what is the final number of level at the end of training ?
Conclusion:
The idea has some major flaws that need to be fixed. I believe the idea has similar effect to adding dropout on the first layer of the discriminator. I don't think the paper should be accepted unless those major concerns are resolved.
References:
[1] Nowozin, S., Cseke, B., & Tomioka, R. (2016). f-gan: Training generative neural samplers using variational divergence minimization. NIPS


We perform completely unsupervised one-sided image to image translation between a source domain X and a target domain Y such that we preserve relevant underlying shared semantics (e.g., class, size, shape, etc). In particular, we are interested in a more difficult case than those typically addressed in the literature, where the source and target are "far" enough that reconstruction-style or pixel-wise approaches fail. We argue that transferring (i.e., translating) said relevant information should involve both discarding source domain-specific information while incorporate target domain-specific information, the latter of which we model with a noisy prior distribution. In order to avoid the degenerate case where the generated samples are only explained by the prior distribution, we propose to minimize an estimate of the mutual information between the generated sample and the sample from the prior distribution. We discover that the architectural choices are an important factor to consider in order to preserve the shared semantic between X and Y . We show state of the art results on the MNIST to SVHN task for unsupervised image to image translation.

==== After rebuttal ===
I thank the authors for responses. I carefully read the response. But it is difficult to find a reason to increase the score. So, I keep my score.
====================
Unsupervised image-to-image (I2I) translation is an important issue due to various applications and it is still challenging when applied to diverse image data and data where domain gap is large. This paper employs a neural mutual information estimator (MINE) to deal with I2I translation between two domains where there is a large gap. However, this paper contains several issues.
1. Pros. and Cons.
   (+) Mathematical definition of I2I translation
   (+) Application of mutual information for conserving content.
   (-) Lack of comparison with recent I2I models
   (-) Lack of experimental results and ablation studies
   (-) Unclear novelty
2. Major comments
   - The novelty of this paper is not clear. Excluding the mathematical definition, it seems that the proposed TI simply combines DCGAN and MINE-based statistical networks. For clarifying the novelty, the detailed architecture and final objective functions can be helpful.
   - Recent works on unsupervised I2I translation are omitted including UNIT [1], MUNIT [2], and DRIP [3]. Also, the authors need to clarify the main difference of TI-GAN from comparing models.
   - It is not clear to relate the mathematical definition of domain transfer to one-to-many translation within large domain gap.
   - It is not clear how to use mutual information (MINE) for learning. There is no explicit definition of loss function considering MINE term.
   - It is short of comparing other state-of-the art models such as UNIT, MUNIT, DRIP, and AugCycleGAN. They compared their results with CycleGAN only.
   - Experiments are not enough to support the authors’ insist. There is not any quantitative metric or qualitative result on generating edge-to-shoes.
   - It is difficult to read due to inconsistent usage of terms (e.g., Figure 3 and 4 (c)s)
   - For better understanding, it requires to compare the patterns of MINE loss and adversarial loss.
   - Experiments on more datasets such as animal, season, faces or USPS datasets.
   - What is the main difference in the results between DCGAN-based and UNet-based models?
Minor
   - cicle_times symbol looks the product between distribution. But it should be defined before being used.
   - A reference of CycleGAN is incorrectly cited.
   - There are some typos in the paper.
   - page 1: dependent → depend
   - page 3: by separate → by separating
   - page 6: S a I → S and I
1. Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks, CoRR, abs/1703.00848, 2017
2. Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz, Multimodal Unsupervised Image-to-Image Translation, CoRR. abs/1804.04732
3. Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang, Diverse Image-to-Image Translation via Disentangled Representations, ECCV 2018.


A DATA-DRIVEN AND DISTRIBUTED APPROACH TO SPARSE SIGNAL REPRESENTATION AND RECOVERY
In this paper, we focus on two challenges which offset the promise of sparse signal representation, sensing, and recovery. First, real-world signals can seldom be described as perfectly sparse vectors in a known basis, and traditionally used random measurement schemes are seldom optimal for sensing them. Second, existing signal recovery algorithms are usually not fast enough to make them applicable to real-time problems. In this paper, we address these two challenges by presenting a novel framework based on deep learning. For the first challenge, we cast the problem of finding informative measurements by using a maximum likelihood (ML) formulation and show how we can build a data-driven dimensionality reduction protocol for sensing signals using convolutional architectures. For the second challenge, we discuss and analyze a novel parallelization scheme and show it significantly speeds-up the signal recovery process. We demonstrate the significant improvement our method obtains over competing methods through a series of experiments.

This paper proposes a (CNNs) architecture for encoding and decoding images for compressed sensing.
In standard compressed sensing (CS), encoding usually is linear and corresponds to multiplying by a fat matrix that is iid gaussian. The decoding is performed with a recovery algorithm that tries to explain the linear measurements but also promotes sparsity. Standard decoding algorithms include Lasso (i.e. l1 regularization and a MSE constraint)
or iterative algorithms that promote sparsity by construction.
This paper instead proposes a joint framework to learn a measurement matrix Phi and a decoder which is another CNN in a data-driven way. The proposed architecture is novel and interesting.
I particularly liked the theoretical motivation of the used MSE loss by maximizing mutual information.
The use of parallel convolutions is also neat and can significantly accelerate inference, which can be useful for some applications.
The empirical performance is very good and matches or outperforms previous state of the art reconstruction algorithms D-AMP and Learned D-Amp.
On comparisons with prior/concurrent work: The paper is essentially a CNN autoencoder architecture but specifically designed for compressed sensing problems.
There is vast literature on CNN autoencoders including (Jiang 2017 and Shi 2017) paper cited by the authors. I think it is fine to not compare against those since they divide the images into small blocks and hence have are a fundamentally different approach. This is fine even if block-reconstruction methods outperform this paper, in my opinion: new ideas should be allowed to be published even if they do not beat SOTA, as long as they have clearly novel ideas. It is important however to discuss these differences as the authors have done in page 2.
Specific comments:
1. It would be interesting to see a comparison to D-Amp and LDAmp for different number of measurements or for different SNRs (i.e. when y = Phi x+ noise ). I suspect each method will be better for a different regime?
2. The paper: `The Sparse Recovery Autoencoder' (SRA) by Wu et al. https://arxiv.org/abs/1806.10175
is related in that it learns both the sensing matrix and a decoder and is also focused on compressed sensing, but for non-image data. The authors should discuss the differences in architecture and training.
3. Building on the SRA paper, it is possible that the learned Phi matrix is used but then reconstruction is done with l1-minimization. How does that perform for the matrices learned with DeepSSRR?
4. Why is Figure 1 going from right to left?


The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs). However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters. This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters. The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model. The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

This paper proposes an algorithm for optimizing neural networks parametrized by Tensor Train (TT) decomposition based on the Riemannian optimization and rank adaptation, and designs a bidirectional TT LSTM architecture.
I like the topic chosen by the authors, using TT to parametrize layers of neural networks proved to be beneficial and it would be very nice to exploit the Riemannian manifold structure to speed up the optimization.
But, the paper needs to be improved in several aspects before being useful to the community. In particular, I found the several mathematical errors regarding basic definitions and algorithms (see below the list of problems) and I’m not happy with lack of baselines in the experimental comparison (again, see below).
The math problems
1) In equations (1), (2), (7), and (8) there is an error: one should sum out the rank dimensions instead of fixing them to the numbers r_i. At the moment, the left-hand side of the equations doesn’t depend on r and the right-hand side does.
2) In two places the manifold of d-dimensional low-rank tensors is called d-dimensional manifold which is not correct. The tensors are d-dimensional, but the dimensionality of the manifold is on the order of magnitude of the number of elements in the cores (slightly smaller actually).
3) The set of tensors with rank less than or equal to a fixed rank (or a vector of ranks) doesn’t form a Riemannian (or smooth for that matter) manifold. The set of tensors of rank equal to a fixed rank something does.
4) The function f() minimized in (5) is not defined (it should be!), but if it doesn’t have any rank regularizer, then there is no reason for the solution of (5) to have rank smaller then r (and thus I don’t get how the automatic rank reduction can be done).
5) When presenting a new retraction algorithm, it would be nice to prove that it is indeed a retraction. In this case, Algorithm 2 is almost certainly not a retraction, I don’t even see how can it reduce the ranks (it has step 6 that is supposed to do it, but what does it mean to reshape a tensor from one shape to a shape with fewer elements?).
6) I don’t get step 11 of Alg 1, but it seems that it also requires reshaping a tensor (core) to a shape with fewer elements.
7) The rounding algorithm (Alg 3) is not correct, it has to include orthogonalization (see Oseledets 2011, Alg 2).
8) Also, I don’t get what is r_max in the final optimization algorithm (is it set by hand?) and how the presented rounding algorithm can reduce the rank to be lower than r_max (because if it cannot, one would get the usual behavior of setting a single value of rank_max and no rank adaptivity).
9) Finally, I don’t get the proposition 1 nor it’s proof: how can it be that rounding to a fixed r_max won’t change the value of the objective function? What if I set r_max = 1? We should be explained in much greater detail.
10) I didn’t get this line: “From the RSGD algorithm (Algorithm 1), it is not hard to find the sub-gradient gx = ∇f(x) and Exp−1 x (y) = −η∇xf(x), and thus Theorem 3 can be derived.” What do you mean that it is not hard to find the subgradient (and what does it equal to?) and why is the inverse of the exponential map is negative gradient?
11) In general, it would be beneficial to explain how do you compute the projected gradient, especially in the advanced case. And what is the complexity of this projection?
12) How do you combine optimizing over several TT objects (like in the advanced RNN case) and plain tensors (biases)? Do you apply Riemannian updates independently to every TT objects and SGD updates to the non-TT objects? Something else?
13) What is E in Theorem 3? Expected value w.r.t. something? Since I don’t understand the statement, I was not able to check the proof.
The experimental problems:
1) There is no baselines, only the vanilla RNN optimized with SGD and TT RNN optimized with your methods. There should be optimization baseline, i.e. optimizing the same TT model with other techniques like Adam, and compression baselines, showing that the proposed bidirectional TT LSTM is better than some other compact architectures. Also, the non-tensor model should be optimized with something better than plain SGD (e.g. Adam).
2) The convergence plots are shown only in iteration (not in wall clock time) and it’s not-obvious how much overhead the Riemannian machinery impose.
3) In general, one can decompose your contributions into two things: an optimization algorithm and the bidirectional TT LSTM. The optimization algorithm in turn consist in two parts: Riemannian optimization and rank adaptation. There should be ablation studies showing how much of the benefits come from using Riemannian optimization, and how much from using the rank adaptation after each iteration.
And finally some typos / minor concerns:
1) The sentence describing the other tensor decomposition is a bit misleading, for example CANDECOMP can also be scaled to arbitrary high dimensions (but as a downside, it doesn’t allow for Riemannian optimization and can be harder to work with numerically).
2) It’s very hard to read the Riemannian section of the paper without good knowledge of the subject, for example concepts of tangent space, retraction, and exponential mapping are not introduced.
3) In Def 2 “different function” should probably be “differentiable function”.
4) How is W_c represented in eq (25), as TT or not? It doesn’t follow the notation of the rest of the paper. How is a_t used?
5) What is “score” in eq (27)?
6) Do you include bias parameters into the total number of parameters in figures?
7) The notation for tensors and matrices are confusingly similar (bold capital letters of slightly different font).
8) There is no Related Work section, and it would be nice to discuss the differences between this work and some relevant ones, e.g. how is the proposed advanced TT RNN different from the TT LSTMs proposed in Yang et al. 2017 (is it only the bidirectional part that is different?) and how is the Riemannian optimization part different from Novikov et al. 2017 (Exponential machines), and what are the pros and cons of your optimization method compared to the method proposed in Imaizumi et al. 2017 (On Tensor Train Rank Minimization: Statistical Efficiency and Scalable Algorithm).
Please, do take this as a constructive criticism, I would be happy to see you resubmitting the paper after fixing the raised concerns!


We present an extension of a variational auto-encoder that creates semantically rich coupled probabilistic latent representations that capture the semantics of multiple modalities of data. We demonstrate this model through experiments using images and textual descriptors as inputs and images as outputs. Our latent representations are not only capable of driving a decoder to generate novel data, but can also be used directly for annotation or classification. Using the MNIST and Fashion-MNIST datasets we show that the embedding not only provides better reconstruction and classification performance than the current state-of-the-art, but it also allows us to exploit the semantic content of the pretrained word embedding spaces to do tasks such as image generation from labels outside of those seen during training.

Summary
=========
The authors present an extension to the VAE model by exploring the possibility of using the label space to create a new embedding space, which they call Probabilistic Semantic Embedding (PSE).
They present two different extension of PSE, PSE and PSE*.
The idea of additionally supporting the latent embedding, created by a VAE, by using available textual descriptors seems promising.
The proposed model was evaluated on two tasks, label-to-image generation and image annotation.
Although the work is interesting, there are a few questions that I am not clear about and have several comments.
Questions
=========
1. How was the word2vec model trained? Did you use an existing pretrained model (e.g. available as download) or did you train the embedding model yourself? If so, on what data?
2. The major novelty of this approach is the use of annotations supporting images and textual (pretrained) embedding spaces, but no related work regarding Wes was neither introduced in the Related Work section nor was it clearly explained in the text.
3. Why did the authors focus on the w2v model instead of more promising approaches as fastText or ELMo?
4. How does your model deal with OOV word(s) as input? For example, when used as Image Generator.
5. Table 1 shows results achieved on MNIST but not Fashion-MNIST; was the evaluation performed on MNIST only?
6. Table 2 presents the impact of the use of pretrained embeddings (word2vec) instead of one-hot vectors for labels. Which one do the models presented in Table 1 use?
7. Could you explain the small difference between using one-hot vs pretrained label encodings, presented in Table 2?
8. Also, can you explain how the numbers in table 2 were achieved (e.g. sum over all, average of all, etc.). When comparing the values presented here to the values of the same measure in table 1, one does notice the big difference between them.
Comments
=========
1. Section 2, page2: “As derived in the original paper…” references which paper (i.e. Kingma et al)?
2. VAE or beta-VAE model is not referenced (mentioned on page 5);
3. Authors do agree that the corpora used is not optimal for the adequate evaluation of the proposed model. It would be interesting to see the use of this approach on a more realistic data set;
4. Unclear sentence: “Compared with the VAE, latent codes where images with the same labels are clustered.”;
5. The authors claim that one of the results of this work is the possibility to generalize for unseen cases (zero-shot learning). It would be interesting to see the performance of this model compared to SOTA in CV in terms of the zero-shot learning task;
6. Figure 2 visualizes proposed embedding space (2D) but it shows VAE and beta-VAE models and omits to show PSE. VAE and beta-VAE are neither introduced nor referenced in text;
7. Table 1: mark best performing with bold. It does, however, outperform other evaluated models when using 20D embedding space;
8. Page 7: in text you mention generation accuracy and in Table 1 the same value is defined as Generation Correctness (%).


Using stochastic gradient descent (SGD) with large batch-sizes to train deep neural networks is an increasingly popular technique. By doing so, one can improve parallelization by scaling to multiple workers (GPUs) and hence leading to significant reductions in training time. Unfortunately, a major drawback is the socalled "generalization gap": large-batch training typically leads to a degradation in generalization performance of the model as compared to small-batch training. In this paper, we propose to correct this generalization gap by adding diagonal Fisher curvature noise to large-batch gradient updates. We provide a theoretical analysis of our method in the convex quadratic setting. Our empirical study with state-of-the-art deep learning models shows that our method not only improves the generalization performance in large-batch training, but furthermore, does so in a way where the training convergence remains desirable and the training duration is not elongated. We additionally connect our method to recent works on loss surface landscape in the experimental section.

It has previously been observed that training deep networks using large batch-sizes leads to a larger generalization gap compared to the gap when training with a relatively small batch-size. This paper proposes to add noise sampled from diagonal "empirical" Fisher matrix to the large batch gradient as a method for closing the generalization gap. The authors motivate the use of empirical Fisher for sampling noise by arguing that the covariance of gradients from small batch-sizes can be seen as approximately equal to a scaled version of the Fisher matrix. It is then pointed out that using the Fisher matrix directly to sample noise could in principle close the generalization gap but would lead to slow converegence similar to SGD with a small batch-size. The authors then claim that the convergence speed is better when noise is sampled from the diagonal Fisher matrix instead of the full Fisher matrix. This claim is proven in theory for a convex quadratic loss surface and experiments are conducted to empirically verify this claim both in the quadratic setting are for realistic deep networks. Finally an efficient method for sampling noise from the diagonal empirical Fisher matrix is proposed.
Comments:
I think the paper is very well written and the results are presented clearly. In terms of novelty, I found the argument about convergence using diagonal Fisher being faster compared with full Fisher quite interesting, and its application for large batch training to be insightful.
As a minor comment, for motivating theorem 3.1, it is pointed out by the authors that the diagonal Fisher acts as an approximation of the full Fisher and hence their regularization effects should be similar while convergence should be faster for diagonal Fisher. As a caveat, I think the authors should also point out that the convergence rate would be best when C is set to 0 in the result of the theorem. This implies no noise is used during SGD updates. However, this would imply the regularization effect from the noise will also vanish which would lead to poor generalization.
However, there is a crucial detail that makes the main argument of the paper weak. In the main experiments in section 4.3, for the proposed large batch training method, the authors mention that they use a small batch-size of 128 for the first 50 epochs similar to Smith et al (2017) and then switch to the large batch-size of 4096, at which point, the learning rate is linearly scaled proportional to the large batch-size with a warmup scheme similar to Goyal et al (2017) and ghost batch normalization is used similar to Hoffer et al (2017). The former two tricks have individually been shown on their own to close the generalization gap for large batch-size training on large datasets like ImageNet. This paper combines these tricks and adds noise sampled from the diagonal Fisher matrix on top when switching to large batch-size after epoch 50 and reports experiments on smaller datasets-- MNIST, Fashion MNIST and the CIFAR datasets. Finally, the accuracy numbers for the proposed method is only marginally better than the baseline where isotropic noise is added to the large batch-size gradient. For these reasons, I do not consider the proposed method a significant improvement over existing techniques for closing the generalization gap for large batch training.
There is also a statement in the paper that is problematic but can be fixed by re-writing. In the paper, empirical Fisher matrix, as termed by the authors in the paper, refers to the Fisher matrix where the target values in the dataset is used as the output of the model rather than sampling it from the model itself as done for computing the true Fisher matrix. This empirical (diagonal) Fisher matrix is used to sample noise which is added to the large batch gradient in the proposed method. It is mentioned that the covariance of the noise in small batch SGD is exactly same as the empirical Fisher matrix. This claim is premised on the argument that the expected gradient (over dataset) is unconditionally roughly 0, i.e., throughout the training. This is absolutely false. If this was the case, gradient descent (using full dataset) should not be able to find minima and this is far from the truth. Even if we compare the scale of expected gradient to the mini-batch gradient (for small batch-size), the scale of these two gradients at any point during training (using say small batch-size SGD) is of the same order. I am saying the latter statement from my personal experience. The authors can verify this as well.
Overall, while I found the theoretical argument of the paper to be mostly interesting, I was dissapointed by the experimental details as they make the gains from the proposed method questionable when considered in isolation from the existing methods that close the generalization gap for large batch training.


The information bottleneck principle (Shwartz-Ziv & Tishby, 2017)  suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective. However, this claim was established on toy data. The goal of the work we present here is to test whether the information bottleneck principle is applicable to a realistic setting using a larger and deeper convolutional architecture, a ResNet model. We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding. We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder. Sampling images by conditioning on hidden layers' activations offers an intuitive visualisation to understand what a ResNets learns to forget.
Conditionally generated images are shown in (b) -(f). Ten epochs is the peak of fitting, and 200 epochs is the end of compression. These samples enable an intuitive illustration of compression in hidden layers. Based on this example it seems that a compressed representation (f) results in varied samples because it compresses class-irrelevant information. Compare the beginning (d) to the end (f) of compression: there is greater variety at the end without losing the essence of 'horse'.
Deep neural networks are ubiquitous in machine learning and computer vision. Unfortunately, the popularity of neural networks for applications is unmatched by an agreed upon and clear understanding of how exactly they work, and why they generalise well. The field of deep learning will advance with more comprehensive theory and empirical studies that better characterise neural networks.
Generalisation, in the context of learning, means the extraction of typical abstract properties of training data that can be successfully used for inference on unseen data. Neural networks generalise well even when they are over-parameterised. Zhang et al. (2017) highlighted the need to rethink generalisation, because conventional wisdom is not readily applicable to neural networks. A number of avenues for new generalisation bounds for neural networks (Bartlett et al., 2017;Golowich et al., 2017;Neyshabur et al., 2017) exemplify how inapplicable conventional methods for understanding model generalisation can be.
One approach to better understanding neural networks is the Information Bottleneck (IB, Section 2) interpretation of deep learning (Tishby et al., 2000;Tishby & Zaslavsky, 2015;Shwartz-Ziv & Tishby, 2017). The IB accredits deep learning success to compression in hidden layers via the noise of parameter updates in stochastic gradient descent (SGD). Information compression results in optimal representations that discard task-irrelevant data while keeping task-relevant data.
The IB principle has since been actively debated (Saxe et al., 2018), partially motivating this work.
The novelty is that we apply information theoretic analyses to modern convolutional residual neural networks (ResNets, Section 5) trained on realistic images (Section 5.2). These choices complicate analysis since information quantities are non-trivial to compute for high-dimensions. Our solution is to define a lower bound on the mutual information (MI, Section 4) and to estimate it by training decoder models (Section 4.1). The decoder model for the MI between images and hidden layers is a conditional PixelCNN++ and samples generated therefrom illustrate visually MI ( Figure  1).

* Summary:
This work is an empirical study of the relevance of the Information Bottleneck principle as a way of understanding deep-learning. It is carried out in the setting of realistically sized networks trained on natural images dataset. This is, in spirit, a meaningful and sensible contribution to the ongoing debate. Being a largely empirical contribution, its value hinges on the exhaustivity and clarity of the experiments carried out. As it stands, I believe that these should be, and can be, improved. Details to support this opinion are given below. A summary of my expectations is given at the end.
* Summary of the approach and significance:
The IB principle relies on estimating the mutual information between i) the input and an intermediate layer, I(x,h), and an intermediate layer and the output, I(h, y). Previous work has relied on binning strategies to estimate these quantities. This is not applicable in a real-sized problem such as classification of natural images with deep networks. This paper proposes to invert a first deep model using a second, generative, model which must reconstruct the input of the first given some intermediate layer. The information progressively discarded by the first network should be modelled as uncertainty by the second. This yields a lower bound on the mutual information, with a tightness that depends on the expressivity of the generative model.
I believe the goal to be meaningful and a valuable contribution: going forward, testing this assumption in realistic setting is essential to the debate. The proposed approach to do this seems sensible to me. It is similar to cited work by Nash et al., however both works are concurrent and so far unpublished and should be considered as complementary point of views on the same problem.
Partial conclusion: The goal is meaningful and sensible.
* Quality of writing.
In sections 1, 2 and 3, the motivation is clear and contains relevant information. I feel it could be polished further. In particular, some redundant information could be condensed. The introduction to the IB principle, though understandable, could be improved. Here are some opinions:
>> Paragraphs 3 and 4 of 1.0: A reference to M. Saxe et al. could already be made there: it is a major 'opponent' in the debate to which you are empirically contributing.
>> In paragraph 1.1: Points 1) and 2) are redundant. So are 3) and 4). Paragraph 1.1 as a whole is largely redundant with the previous paragraph, these could be collapsed.
>> In section 2: I feel that the main intuitions of the encoder / decoder distributions (I(y,h) / I(x,h)), the information plane, and the optimal bottleneck representations (Sections 2.1 to 2.3 of Schwartz-Ziv & Tishby) could be better conveyed in 2, though I understand the need for brevity.
>> Related works: Descriptions of related works could be condensed. On the other hands, the points of these papers that you contradict in you experiment section could be explicitly mentioned again there.
>> End of section 3, section 4: The fact that estimation of the MI by traditional means is not applicable in your setting is repeated many times throughout the paper, in noticeably rapid succession in that region. Some mentions should be removed.

### More minor points:
>> Paragraphs 2 of 1.0:
- 'the extraction of typical abstract properties...' this statement is vague and thus debatable: the channel-wise mean in RGB space, for instance, is not an especially abstract property.
- Reference is made to Zhang et al. to justify the need for more analysis of generalization in deep-learning. This paper can be considered controversial. Mention could be made of other works, for instance about the inaplicability of traditional generalization bounds and attempts to improve them.
>> Pseudo code algorithm.
- Could be summarized in the main body and pushed to the annex.
>> The choice of pixCNN as a generative model could be discussed more. There are some good reasons to prefer this to L2 regression for instance.
Partial conclusion: The description of the method contains relevant information and is functional, but the writing could be improved.

* Experimental results.
> The contribution and novelty of this paper is largely empirical. Therefore the experimental results should be held to a high standard of clarity and exhaustivity.
*** The choice of dataset:
The experimental setup seems to be fair in terms of dataset / split chosen: the abundance of data for the three steps (encoding, decoding, evaluation) is a notable strength.
*** The quality of the lower bound: Uncertainty when reconstructing the image may come from the fact that information has been discarded. Variance may also come from the pixCNN++, which is imperfect. You mention this (paragraph 4.1) but do not take experimental steps to measure it. Please consider reporting the performance of your generative model i) without conditioning, ii) conditioned on one-hot ground truth labels, and optionally iii) on grayscale/downsampled versions of the image without otherwise modifying the training setup. These values will give the reader an idea of the significance of variations in MI measured and give a 'scale' to your figures, strengthening your claims.
*** The evolution of compression *accross iterations* for a fixed layer
I will focus on the classification setting for now.
Qualitatively: Figures 1, 4 and 5 do not convince me that a meaningful evolution in the type of information discarded *across training iterations* can be observed visually. In figures 1 and 4, the network seems to learn invariances to some coloring, and its preferred colours vary across iterations. Beyond that I cannot see much, except maybe for column (f) of figure 4, despite your claim in section 2, paragraph 2.
Quantitatively: Curves in Figure 2 a) are more convincing, though a notion of scale is missing, as already discussed. The evolution of I(y; h) across iterations is very clear, in Figure 2 a) and especially 3 a). The evolution of I(x, h) much less so. h3 and h4 do not seem to show anything meaningful. In h2 the decrease in I(x, h) is supported by only 2 points in the curve (epoch 10 to epoch 100, and epoch 100 to epoch 200, figures 2a and 3c). Epochs displayed are also incoherent from one curve to the next (epoch 15 is missing for h2 in fig 3c) which raises suspicion. It appears important to i) display more points, to show that this is not just noise and ii) track more layers to confirm the trend, supported by a single layer so far (see next paragraph). I understand that each of these points require training of a generative model, but I feel it is necessary to make reliable conclusions.
Minor: In figure 2, epochs should be added as labels to the colours.
*** The evolution of compression *across layers* for a fixed iteration

Conversely, the evolution of the MI across layers is very convincingly demonstrated, and I feel this is perhaps the main strength of this paper. All curves display consistent trends across layers, and Figure 5 qualitatively displays much more invariance to pose, detail, etc than Figure 4. This is interesting, and could be made more central: i) by making a figure that compares samples across layers, for a fixed iteration, side by side.
On the downside, I believe it is important to track more layers, as it is to me the main interest of your results. The second paragraph of section 5 does not give a good idea of the spread of these layers to someone not familiar with the resnet architecture used. For example, the penultimate layer of the network could be used (the layer at which the most compression is to be expected).
*** On the auto-encoder experiments.
> Little detail is given about the way the auto-encoder is constructed. In particular, one expects the type of bottleneck used (necessary so that the network does not learn the identity function) to have large impact on the amount of information discarded in the encoding process. This dependency is not discussed. More crucially, experiments with different types / strength of bottleneck are not given, and would, in my opinion, be key to an analysis of this dependency through the IB principle.
> Furthermore, no qualitative analysis is provided in this setting.
> Without these additions, I find the Auto-encoding setting an unconvincing distraction from the main contribution of this paper.
***  main avenues of improvement:

> Two kinds of progression in compression are demonstrated in your paper: across layers, and across iterations.
As it stands, results evidence the former more convincingly than the latter, both qualitatively and quantitatively.
I believe results could be presented in a way that clearly takes better advantage of this, as I will detail further.
More data points (across layer and epochs) would be beneficial. I feel that the auto-encoder setting, as it stands, is a distraction.
I would find this paper more convincing if experiments focused more on showing how layers progressively discard information, and less on the 'training phases' that are so far less clear.
*** Additional comments
The following are a number of points that would be worthwhile to discuss in the paper
> As it stands, it seems the line of reasoning and experimental setup seems to rely on the chain-structured nature of the considered neural net architecture. Can the same line of reasoning be applied to networks with more general computational graphs, such as dense-nets [a], mulit-scale denseness [b], fractal nets [c] etc.
[a] Huang, G.; Liu, Z.; van der Maaten, L. & Weinberger, K. Densely connected convolutional networks CVPR, 2017
[b] Huang, G.; Chen, D.; Li, T.; Wu, F.; van der Maaten, L. & Weinberger, K. Multi-Scale Dense Networks for Resource Efficient Image Classification ICLR, 2018
[c] https://arxiv.org/abs/1605.07648
> Why is it that earlier layers are estimated to have larger MI with the target y than later layers before convergence? Sure, later layers compress certain information about the input x, which could be informative on the response variable y. But since the MI estimate for early layers depends on the same network architecture as the one used to compute the later layers from the early ones, the result seems counter intuitive. See paragraph "forward direction" in section 4.1.
> The orange curve in fig 3a estimating I(x;y) is not commented upon. How was it obtained, what is its relevance to the discussion?


In semi-supervised learning, Bad GAN approach is one of the most attractive method due to the intuitional simplicity and powerful performances. Bad GAN learns a classifier with bad samples distributed on complement of the support of the input data. But Bad GAN needs additional architectures, a generator and a density estimation model, which involves huge computation and memory consumption cost. VAT is another good semi-supervised learning algorithm, which utilizes unlabeled data to improve the invariance of the classifier with respect to perturbation of inputs. In this study, we propose a new method by combining the ideas of Bad GAN and VAT. The proposed method generates bad samples of high-quality by use of the adversarial training used in VAT. We give theoretical explanations why the adversarial training is good at both generating bad samples and semi-supervised learning. An advantage of the proposed method is to achieve the competitive performances with much fewer computations. We demonstrate advantages our method by various experiments with well known benchmark image datasets.

This paper makes the interesting observation that the generative procedure proposed by Bad GAN paper can be replaced by a slightly modified VAT procedure. The reasoning is sound and leverages the intuition that adversarial examples (subject to a sufficiently small perturbation radius) are likely to be closer to a decision boundary than the original sample.
The paper is generally easy to follow but the presentation could be improved. In particular more could be done to describe the terms in Equation 5. I’m also curious about the behavior of L^true, which is equivalently the fourth term in Eq 1. Even when reading Bad GAN paper, I did not quite understand their claim that this can be correctly interpreted as a conditional entropy term (if they really wanted conditional entropy, they should probably have either done H(p(k|x)) or H(p(k|x, k <= K))). I agree with the authors that the roles of the second and fourth terms overlapped, and I think this is sufficiently interesting to warrant some further elaboration in the paper. I also liked the reminder that power iteration selects a non-unique sign for the first eigenvector (subject to the random vector initialization); I encourage the authors to do an ablation test to convince the reader that “this modification helps to improve convergence speed of the test accuracy.”
The propositions in this paper were, in my opinion, not particularly insightful. While I think it is nice that the authors went through the effort of providing some formalism to the intuition that VAT has a “push decision boundary away from high-density regions”, I’m less sure if propositions 1 and 2 really provides any additional insight the behavior of VAT. Proposition 1 is pretty weak in that it only covers a 2-class logistic regression; it seems obvious that the adversarial perturbation points in the direction toward the decision hyperplane. If the authors could extend this to more general non-linear classifiers (perhaps subject to some assumptions), that would be more interesting. I don’t think Proposition 2 has any real value and recommend its relegation to the appendix.
I think the biggest weakness of this paper is the experiments. Taking Table 1 at face value, the conclusion that FAT is simply competitive with existing approaches suggests that the additional machinery isn’t particularly useful, providing little more than a vanilla VAT. I also think MNIST/SVHN has run its course as good semi-supervised learning benchmarks and would prefer to see such algorithms being scaled to more complex data. The main argument for why FAT should be prefered over VAT comes from Section 6.2. Figure 4 is more interesting, but is complicated by the fact that FAT checks both possible eigenvectors (+/- u) during training, which requires two forward passes in the classifier; did the authors give a similar treatment to VAT? Please show wall-clock time too. Unfortunately the computational efficiency gain seems to only hold true for MNIST/SVHN, but not for CIFAR. I worry that the observed gains will not sustain once we move to more complicated datasets.
Pros:
+ Simple and clean proposal
+ Easy to read
Cons:
- Limited insight
- Weak experiments


Bayesian Hierarchical Community Discovery
We propose an efficient Bayesian nonparametric model for discovering hierarchical community structure in social networks. Our model is a tree-structured mixture of potentially exponentially many stochastic blockmodels. We describe a family of greedy agglomerative model selection algorithms that take just one pass through the data to learn a fully probabilistic, hierarchical community model. In the worst case, Our algorithms scale quadratically in the number of vertices of the network, but independent of the number of nested communities. In practice, the run time of our algorithms are two orders of magnitude faster than the Infinite Relational Model, achieving comparable or better accuracy.

The authors propose a generalization of the Bayesian
      nonparamtric stochastic block model to the hierarchical setting, in order
      to infer a hierarchy rather than a flat clustering of entities when
      conditioned on observed links between entities. They propose a learning
      algorithm based on agglomerative clustering to select a tree which
      represents valid possible partitions of the entities. They then present
      empirical results from two corpuses of social connectivity data and
      conclude that their method is both faster and more accurate than the
      Infinite Relational Model (IRM), a widely-used model for tasks of this
      kind. I believe this paper tackles an interesting issue that a
      significant fraction of the NIPS audience will be interested in. On the
      technical side, Bayesian nonparametric methods, while attractive from a
      conceptual point of view, are often quite slow in practice. On the proble
      domain side, social scientsits are increasingly using sophisticated
      Bayesian models for analyzing data and so a hierarchical extension to a
      popular network-discovery model should be appreciated. This paper touches
      on both those issues. The key insight is that by restricting
      allowed partitions to conform to a tree structure, the marginal likelihood
      of the data can be computed efficiently. The IRM must essentially consider
      the exponential number of possible partitions of the data. However, I was
      left somewhat confused on how the learning procedure of the trees relates
      to the generative story behind the Bayesian model. Since it is greedy, I
      do not think it can be seen as performing true posterior inference over a
      distribution of trees or finding some kind of point MAP or ML estimate.
      But I may have misunderstood. I would recommend that the authors try to
      improve the clarity of Section 4. I found the results to be
      impressive. In terms of several performance measures (AUC, predictive llh,
      accuracy), the proposed method performed comparably or slightly better
      than the IRM, the closest competitor. While the asymptotic improvements in
      these scores are perhaps not that impresive in absolute terms, the claimed
      runtime is orders of magnitude faster than IRM inference. I think
      the overall clarity and writing quality could be improved. First, there
      are a series of distracting spelling and grammar errors (eg line 126).
      Second, the prose is sometimes hard to understand. Line 47-50 in the
      introduction has some vague phrasing (what are 'local changes' in this
      setting? What are the 'iterations' referring to? Iterations of an MCMC
      algorithm?), for instance. The authors do not clearly describe the content
      of the datasets they train on (Monastery networks and NIPS authorship),
      which could confuse readers who are not already familiar with them. I
      occasionally found the math somewhat obtuse and unnecessary, such as Eq.
      5. I also found Figure 3 confusing. Being unfamiliar with the monastery
      dataset, it is visually unclear which method is performing the best.
      Perhaps a dataset that more readers will be familiar with would be
      helpful. Section 4, which presents the core novel contribution of the
      paper, required several readings to understanding. If these issues
      could be remedied, however, I believe the proposed approach has enough
      novelty and the results are strong enough to warrant publication. The
      authors propose a novel model of graph edge structure that can be seen as
      a Bayesian version of agglomerative clustering inspired by the IRM. I
      found the core learning algorithm to be insightful and the experimental
      results to be strong, but the quality of writing in need of improvement.


Learning Trajectory Preferences for Manipulators via Iterative Improvement
We consider the problem of learning good trajectories for manipulation tasks. This is challenging because the criterion defining a good trajectory varies with users, tasks and environments. In this paper, we propose a co-active online learning framework for teaching robots the preferences of its users for object manipulation tasks. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this co-active preference feedback can be more easily elicited from the user than demonstrations of optimal trajectories, which are often challenging and non-intuitive to provide on high degrees of freedom manipulators. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We demonstrate the generalizability of our algorithm on a variety of grocery checkout tasks, for whom, the preferences were not only influenced by the object being manipulated but also by the surrounding environment.

The paper present an co-active learning approach to
      generate user- or task-preferred trajectories for manipulation accounting
      for task-, object-, environment- and human-related context features.
      The paper addresses a timely topic with an interesting approach
      based on the rather novel idea of incorporating task context into the
      problem of trajectory learning. It is understandable and easy to read, and
      presents an evaluation with quantitative and subjective measures, the
      former carried out in simulation with a 7-dof PR2 arm However,
      there are weaknesses. Related literature is poorly discussed with
      several omissions of related works and a rather shallow level of
      discussion. Highly related are the lines of work by Sisbot and Alami et
      al. (e.g. [2]) which is complementary to yours -- they take a model-based
      approach, no learning -- but are, to the knowledge of the reviewer, the
      first to address the issue of user preferences in robot motion planning
      and the work by Cakmak et al. [1] on complex manipulation tasks in an
      active learning framework. The line of work by Ijspeert and Schaal on DMPs
      seem to be underrepresented in particular given the many variations and
      related extensions of their original idea. The works by Calinon and
      Billard et al. (e.g. [3]) on probabilistic motion generalization are not
      at all cited and discussed. Also, it appears unjustified to dismiss the
      cited and highly related work [32] so easily ("since they learn in
      discretized action space their approach is not directly applicable to
      manipulators with large action spaces"). Their work was evaluated on four
      (continuous-state) domains similar to yours and the question of the
      applicability of their method should at least be elaborated in more
      detail. The paper's contribution is limited by the fact that the
      core theoretical properties such as the TPP algorithm and the regret
      bounds are from previous work [28] and that the amount of manual effort in
      designing the features is very high (admittedly, certain context
      information such as object properties are hard to get automatically). This
      makes that the main contribution of this paper lies in the feature
      definition, the implementation and evaluation. Also, the paper assumes
      perfect knowledge of the environment and all static and dynamic objects.
      Some questions remain unclear regarding interactive feedback: It
      is not clear what you mean by correcting waypoints, can the user just
      confirm/dismiss them or edit their 3D position? How many waypoints did a
      typical trajectory have in the experiments? How much time did the training
      phase take (for both kinds of feedback) and how many times bad waypoints
      needed to be corrected during training? In the evaluation, the
      nDCG metric is not sufficiently explained neither is Table 1 (values in
      the cells?). The user study lacks a hypothesis and is of limited
      significance with N = 2. References: [1] M. Cakmak et al.,
      "Designing Robot Learners that Ask Good Questions", HRI 2012 [2] E.A.
      Sisbot et al., "A human aware mobile robot motion planner", TRO, 2007
      [3] S. Calinon et al., "Active Teaching in Robot Programming by
      Demonstration", RO-MAN 2007


Capacity of strong attractor patterns to model behavioural and cognitive prototypes
We solve the mean field equations for a stochastic Hopfield network with temperature (noise) in the presence of strong, i.e., multiply stored, patterns, and use this solution to obtain the storage capacity of such a network. Our result provides for the first time a rigorous solution of the mean filed equations for the standard Hopfield model and is in contrast to the mathematically unjustifiable replica technique that has been used hitherto for this derivation. We show that the critical temperature for stability of a strong pattern is equal to its degree or multiplicity, when the sum of the squares of degrees of the patterns is negligible compared to the network size. In the case of a single strong pattern, when the ratio of the number of all stored pattens and the network size is a positive constant, we obtain the distribution of the overlaps of the patterns with the mean field and deduce that the storage capacity for retrieving a strong pattern exceeds that for retrieving a simple pattern by a multiplicative factor equal to the square of the degree of the strong pattern. This square law property provides justification for using strong patterns to model attachment types and behavioural prototypes in psychology and psychotherapy.

In this paper the authors consider applying a mean
      field approach to a Hopfield model which has presented with strong
      patterns (patterns that have been presented to the model more than once).
      By using a mean field approach and applying the Lyapunov condition they
      show that they can arrive at mean field equations for two cases:
      1) When the model is presented with a finite collection of strong
      patterns and a collection of simple patterns such that the total number of
      stored patterns over the total number of neurons tends to zero as the
      number of neurons becomes large. 2) When the model is presented
      with a single strong pattern and a collection of simple patterns such that
      the total number of stored patterns over the total number of neurons tends
      to some constant alpha as the number of neurons becomes large. In
      both cases they arrive at self consistent equations, in the first case
      they then evaluate how noise level changes the models ability to recall
      the strong patterns and in the second they look at how the constant alpha
      changes the recovery of the strong patterns. Quality
      ------------------------------------------------------------ There
      a few typos and bad sentences in this paper (see below) but on the whole
      the quality of the writing is OK. My main problems, however, are with the
      proof/statement of lemma 4.1 and the use of lemma 4.1 in the proof of
      theorem 4.3. Proof/Statement of lemma 4.1
      --------------------------------- I think that lemma 4.1 should
      have m_mu instead of m_nu for it to be applied later to theorem 4.3 and
      I'm going to assume this in what follows. I think the proof of lemma 4.1
      does not work. In this proof the authors assume that \xi^1, \xi_i^\mu and
      m_\mu are independent, this is not true since m_{\mu} is constructed from
      a sum including \xi_i^\mu, granted this is a dependency that becomes
      weaker with increasing N, but it is still present and should be addressed.
      Secondly the authors state that m_\nu are independent, I think this is an
      assumption rather than a statement since eqn 11 shows that one may be
      written as a function of the other m_\mu. Proof of Theorem 4.3
      -------------------------------------- I think that eqn 15 should
      have as the sum index mu\neq 1 and not mu\neq 1,\nu. In which case either
      lemma 4.1 cannot be applied at eqn 16 or the statement of lemma 4.1 must
      be changed. If this can be addressed then I agree that the rest of the
      proof of 4.3 follows. Typos etc --------------------------
      - Index should be j around eqns 2 and 3 - I would like the authors
      to mention for some constant m or similar after the first eqn in section 3
      - Statement of lemma 4.1 (see above) - square at end of eqn 13
      should be at the end of the proof - brackets around the expectation in
      eqn 13 - lower limits of the integrals in eqn 12 and the text after
      eqn 13 - The paragraph above theorem 4.3 is missing a word or two I
      think - Eqn 15 (see theorem 4.3 above) - The + symbol in the
      variance just above and in eqn 18. Clarity
      --------------------------------------- Mistakes, typos, lemma 4.1
      and the beginning of the proof of theorem 4.3 aside the paper is clear and
      well written. I think that if the problems outlined above were addressed
      the paper would present a concise and comprehensible discussion on the
      storage of strong patterns in the Hopfield model. Originality
      --------------------------------------- As far as I know this
      material is original. Significance
      ---------------------------------------------- The results seem
      significant enough to be published.


Multisensory Encoding, Decoding, and Identification
We investigate a spiking neuron model of multisensory integration. Multiple stimuli from different sensory modalities are encoded by a single neural circuit comprised of a multisensory bank of receptive fields in cascade with a population of biophysical spike generators. We demonstrate that stimuli of different dimensions can be faithfully multiplexed and encoded in the spike domain and derive tractable algorithms for decoding each stimulus from the common pool of spikes. We also show that the identification of multisensory processing in a single neuron is dual to the recovery of stimuli encoded with a population of multisensory neurons, and prove that only a projection of the circuit onto input stimuli can be identified. We provide an example of multisensory integration using natural audio and video and discuss the performance of the proposed decoding and identification algorithms.

Reviewer's response to the rebuttal:
      '"Theorem 1 needs to be corrected/clarified." We
      apologize for the confusion: part of the theorem was accidentally omitted.
      The sufficient condition is to have a large enough population of
      (different) neurons: N >= SUM_m[(2L_1+1)(2L_2+1)…(2L_{n_m-1} + 1)]. The
      necessary condition is that the total number of spikes is greater than
      (2L_{n_m}+1)*SUM_m[(2L_1+1)(2L_2+1)…(2L_{n_m-1} + 1)] +N" OK,
      that seems more acceptable, even though I don't fully understand the
      condition -- what prevents these spikes from being completely synchronized
      across neurons, and hence being uninformative? '"…what would
      happen if all filter kernels were the same? Then no matter how many
      neurons you added, you would not gain any further information.” That
      is not entirely correct. For example, if all filters are Dirac-delta
      functions, the inputs are encoded directly without processing. However,
      one can recover all inputs if the neurons have different parameters (e.g.,
      bias, capacitance, threshold of the IAF neuron). Imposing linear
      independence between kernels is another way to guarantee signal recovery.'
      True, but in your definition of the IAF-TEM (eqn3 and following)
      it does not appear that the neurons have different parameters, hence my
      remark. Please say that somewhere. Review for "Multisensory
      Encoding, Decoding, and Identification" The paper describes an
      extension of Time Encoding Machines (TEM) to the
      multiple-input-multiple-output (MIMO) setting, using the formalism of
      Reproducing Kernel Hilbert Spaces (RHKS). The approach is motivated from a
      neuroscientific perspective: in the introduction, it is claimed
      (correctly) that multisensory integration (and processing) provides many
      benefits to organisms, but that it is still poorly understood. What
      follows is a particular extension of TEMs to the MIMO setting, with
      trignometric polynomials as basis functions. Necessary conditions for
      perfect decoding and/or system identification are given. The feasibility
      of the resulting algorithms are demonstrated on an audio/video decoding
      task, and an audio/video encoder identification task. The paper concludes
      with the remark that this work constitutes the first tractable
      computational model for multisensory integration (which is wrong) and that
      extensions to the noisy case would be straightforward. Clarity:
      The paper is mostly well written, even though the notation in section
      3 is cumbersome, but I do not see how that could be avoided.
      Originality: the paper's most original contribution is the
      extensions of TEMs to the MIMO setting, the connection to the brain is
      tenuous. Quality: I believe some parts of the paper need to be
      improved before publication. Significance: from a technical
      perspective, the paper demonstrates the feasibility of TEMs for
      multi-modal encoding, which might prove useful. There is no clear
      contribution to neural coding, other than the motivation in the
      introduction. Detailed comments: line 100:
      "bandlmited" -> bandlimited line 137: j is the complex unit,
      please say that somewhere. line 144: the u_n in eqn.(1) are not
      the same as the u_n^i in figure 1, right? I propose to use a different
      letter. line 166: "BIBO-stable" bounded-input-bounded-output?
      please define. line 166: "...finite temporal support of lenghth
      S_i \leq T_i...": would that be spatio-temporal support for the video
      filters in example 2? line 177: Say here that the h_{l_1 \ldots
      l_n} are the filter coefficients, to connect this formula to theorem 1.
      line 182: "...IAF TEM...": IAF="integrate and fire"? Please
      specify. line 185: eqn. 3: what is the index k? From other
      literature about IAF-TEMs, I would infer that it's the spike index? If
      that is correct, please also say that t_k is a spike time, and that these
      are strictly monotonically increasing with k etc. In fact, it would be
      very useful if there was a short introduction to IAF-TEMs: up to here, the
      paper only talks about continuous signals, now spikes are introduced (if I
      interpret everything correctly). line 195-195: "Then there exists
      a number N ... can be perfectly recovered" this sounds like a sufficient
      condition. But in the proof of this theorem 1, line 224ff: "A necessary
      condition for the latter (solvability)...." so you only show a necessary
      condition? Hence, theorem 1 remains unproven, I think. In fact, I wonder
      what would happen if all filter kernels were the same? Then no matter how
      many neurons you added, you would not gain any further information, thus
      the signal recovery would not improve. To remedy this, I think the kernels
      would have to be constrained by some sort of linear independence
      condition, similar to [15]. figures 1,2,4: the v_i do not appear
      anywhere else in the paper. Please clarify their purpose. section
      5, examples. The examples are instructive. But I wonder why you did not
      choose a real video+audio for the examples -- it would have strengthened
      the connection to neuroscience if you showed that your approach works
      on relevant datasets. line 417: "...the first tractable
      computation model for multisensory integration..." wrong, see e.g. the
      works by Wei & Koerding, or Ernst & Banks. Please remove this
      sentence.


Adaptive Step-Size for Policy Gradient Methods
In the last decade, policy gradient methods have significantly grown in popularity in the reinforcement-learning field. In particular, they have been largely employed in motor control and robotic applications, thanks to their ability to cope with continuous state and action domains and partial observable problems. Policy gradient researches have been mainly focused on the identification of effective gradient directions and the proposal of efficient estimation algorithms. Nonetheless, the performance of policy gradient methods is determined not only by the gradient direction, since convergence properties are strongly influenced by the choice of the step size: small values imply slow convergence rate, while large values may lead to oscillations or even divergence of the policy parameters.
Step-size value is usually chosen by hand tuning and still little attention has been paid to its automatic selection. In this paper, we propose to determine the learning rate by maximizing a lower bound to the expected performance gain. Focusing on Gaussian policies, we derive a lower bound that is second-order polynomial of the step size, and we show how a simplified version of such lower bound can be maximized when the gradient is estimated from trajectory samples. The properties of the proposed approach are empirically evaluated in a linear-quadratic regulator problem.

This paper proposes a new policy-gradient
      reinforcement learning method in which the step size is set adaptively by
      maximizing a lower bound on the expected performance gain. The authors
      first consider the case where the policy gradient is known and establish a
      lower bound on the difference in performance between the original policy
      and the modified policy resulting from an update that follows the policy
      gradient. Since the above bound is a fourth-order polynomial, the authors
      then consider the special case in which the stochastic policy is
      represented as a Gaussian distribution with a fixed standard deviation and
      a mean that is a linear combination of the state features. In this case,
      they derive a lower bound that is quadratic in the step size and has a
      single maximum for positive step sizes. Next, they remove the assumption
      that the policy gradient is known and establish a high-probability lower
      bound given an epsilon-optimal estimate of the policy gradient. Finally,
      they show how two existing techniques, REINFORCE and G{PO)MDP/PGT, can
      provide such policy gradient estimates, leading to an adaptive step size
      policy gradient method that does not require a model of the environment
      and can learn only from sample trajectories. The proposed method is
      evaluated in a simple simulation experiment using a 1-dimensional toy
      problem. Overall, this is a high-quality, well written paper. The
      authors make a good argument that most research on policy-gradient methods
      has focused on finding better gradient estimators and thus there remains
      room for improvement in automating the step size in such algorithms. The
      theoretical results in the paper make a substantial contribution by
      providing a principled framework for the automatic selection of such step
      sizes by maximizing lower bounds on expected performance gain. The paper
      contributes not only a specific algorithm but more general results that
      could be used to derive algorithms with different policy representations
      and gradient estimators. The main weakness in the paper is the
      empirical evaluation, which considers only a toy problem involving the
      optimization of one policy parameter. Results obtained using the true
      gradient show that manual choices of the step size can lead to slow
      learning (too small) or divergence (too big) whereas the adaptive method
      always converges within the threshold. For large values of the standard
      deviation of the Gaussian policy, the adaptive method performs better than
      any of the fixed step-sizes tested. However, for small standard deviations
      it performs worse, which the authors suggest reflects an inherent
      trade-off between the determinism of the policy and tightness of the lower
      bound. The authors also present results for the case where the
      true gradient is not known and the REINFORCE and PGT estimators are used
      instead. However, it is not clear what these results demonstrate, other
      than that, unsurprisingly, increasing the number of trajectories used to
      estimate the gradient improves the gradient estimate and thus learning
      performance. More worryingly, however, the results also show that even
      with many trajectories, the errors are quite large, leading to loose
      bounds that prevent strong performance. This casts doubt on whether the
      results have practical implications or are only of theoretical interest.
      Furthermore, the empirical results compare only to fixed-step size
      variants and not to other baseline methods. In particular, the EM-based
      policy-gradient-like methods of Kobers & Peters and Vlassis et al.
      mentioned in the introduction seem like particularly relevant baselines
      since they do not need to tune a free parameter.