Hyperparameter tuning is a bothersome step in the training of deep learning models. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gradient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems. Notably, Alrao is robust: all runs were able to learn without any tuning. This makes Alrao a candidate for situations in which a neural network has to work from the first try. This text comes with a PyTorch implementation of the method, which can be plugged on an existing PyTorch model.

POST REBUTTAL: I think the paper is decent, there are some significant downsides to the method but it could constitute a first step towards a more mature learning-rate-free method. However, in its current state the paper is left with some gaping holes in its experiment section. The authors tried to add experiments on Imagenet, but these experiments apparently didn't finish before the end of the rebuttal period. For that reason, the paper probably should not be accepted for publication (even if the authors manage to finish running these experiments, we would not have a chance to review these results).
------
Original review
------
In this paper, the authors present a method for training deep networks with randomly sampled feature-wise learning rates, removing the need for fixed learning rates and their tuning. The method is shown to perform comparatively to SGD with a learning rate roughly optimized with regards to validation performance. The method applies to the most popular types of deep learning architectures, which includes fully connected layers, convolutional layers and recurrent cells.
Quality: The paper is of a decent quality in general, I noticed no glaring omissions while reading the paper. However, I do worry that the method provides little gain for a lot of work. It is becoming more and more easy to tune the learning rate of deep learning models with strategies such as early stopping, and this method comes at a high cost for models with a big final layer.
Clarity: The paper is well written, but the reader is often (too often?) sent to the Appendix, which is itself ordered in a strange way (e.g., the first reference to the Appendix in the paper refers to Appendix F?). If some sections of the Appendix are not needed, I would remove them.
Originality: The work is original in the approach, i.e. randomization as a way to get rid of learning rates is a novel method. However, there was one work presented last year at NIPS which concerns itself with the same problem, which is getting rid of learning rates:
“Training Deep Networks without Learning Rates Through Coin Betting” by Francesco Orabona and Tatiana Tommasi, NIPS, 2017.
They don’t compare on the same methods and the same datasets, but I think the authors should be aware of this work and perhaps compare themselves with it. The work takes a very different approach to solve the problem so I don’t think it’s an issue for this paper.
Significance: I think the work is important, in that it adds another tool to solve the learning rate problem. I would not say it is likely to have a very high impact, because it involves a lot of work, for little benefit. Furthermore, the cost of reproducing multiple times the last layer of the network will be prohibitive in many cases for NLP.
The method feels ad-hoc in many respects, and there are no guarantees that it would work any better than Adam does on pathological cases. Perhaps some mathematical analysis on simpler problems would help make the contributions stronger.
The authors state that the learning rate range has little impact on performance, yet it still has enough impact to justify tuning it for different models and datasets (on CIFAR it is 10^-5 to 10^1, on Pennbank it is 10^-3 to 10^2). I would tend to agree that the alrao method is more robust to the choice of learning rate than plain SGD, however the fact of the matter is that there are still parameters to tune.
Figure 5. also seems to suggest that the range is important, although the models were not trained until the end, so it is not clear.
Some additional comments:
Nitpicking: In Section 2, most sub-sections (or paragraphs titles?) have the name of the method in them. That’s redundant. Instead of “Alrao principle”, “Alrao update”, etc., just write “Principle.”, “Update.”.
Is there a justification for using the same learning rate for all weights in an LSTM unit?
I believe there is a mistake in Equation 2. The denominator should be log(\eta_{max}) - log(\eta_{min})
[second paragraph on page 4.] Once again nitpicking for the sake of clarity: “For each classifier Cθ cl j, we set a learning rate log eta_j = …” this reads as if the learning rate would be set to log eta_j, but you probably mean you will set the learning rate to eta_j = exp(...).
Figure 5b in the appendix does not specify which curve has which learning rate interval.


LEARNING ROBUST REPRESENTATIONS BY PROJECT- ING SUPERFICIAL STATISTICS OUT
Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level cooccurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' outof-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's. We test our method on battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

Summary:
The paper proposes an unsupervised approach to identify image features that are not meaningful for image classification tasks. The goal is to address the domain adaptation (DA)/domain generalization (DG) issue. The paper introduces a new learning task where the domain identity is unavailable during training, called unguided domain generalization (UDG). The proposed approach is based on an old method of using gray level co-occurence matrix, updated to allow for differentiable training. This new approach is used in two different ways to reduce the effect of background texture in a classification task. The paper introduces a new dataset, and shows extensive and carefully designed experiments using the new data as well as existing domain generalization datasets.
This paper revisits an old idea from image processing in a new way, and provides an interesting unsupervised method for identifying so called superficial features. The proposed block seems to be very modular in design, and can be plugged into other architectures. The main weakness is that it is a bit unclear exactly what is being assumed as "background texture" by the authors.
Overall comments:
- Some more clarity on what you mean by superficial statistics would be good. E.g. by drawing samples. Are you assuming the object is centered? Somehow filling the image?  Different patch statistics? How about a texture classification task?
- please derive why NGLCM reduces to GLCM in the appendix. Also show the effect of dropping the uniqueness constraint.
- Section 3.2: I assume you are referring to an autoencoder style architecture here. Please rewrite the first paragraph. The current setup seems to indicate that you doing supervised training, since you have labels y, but then you talk about decoder and encoder.
- Section 3.2: Please expand upon why you use F_L for training but F_P during testing
Minor typos/issues:
- Last bullet in Section 1: DG not yet defined, only defined in Section 2.
- page 2, Section 2, para 1: data collection conduct. Please reword.
- page 2, Section 2, para 2: Sentence: For a machine learning ... There is no object in this sentence. Not sure what you are trying to define.
- page 2, Section 2, para 2: Is $\mathcal{S}$ and $\mathcal{T}$ not intersecting?
- page 2, Section 2.1: Heckman (1977), use \citep
- page 2, Section 2.1: Manski, citep and missing year
- page 3, Section 2.1: Kumagai, use citet
- page 3, Section 3.1: We first expand ... --> We first flatten A into a row vector
- page 4, Section 3.1: b is undefined. I assume you mean d?
- page 4, Section 3.1: twice: contrain --> constraint
- page 4, Section 3.2: <X,y> --> {X,y} as used in Section 3.1.
- page 4, Section 3.2, just below equation: as is introduced in the previous section. New sentence about MLP please. And MLP not defined.
- page 4, Section 3.2, next paragraph: missing left bracket (
- page 4, Section 3.2: inferred from its context.
- page 5, Section 4: popular DG method (DANN)
- page 7: the rest one into --> the remaining one into
- page 8: rewrite: when the empirical performance interestingly preserves.
- page 8, last sentence: GD --> DG
- A2.2: can bare with. --> can deal with.
- A2.2: linear algebra and Kailath Variant. Unsure what you are trying to say.
- A2.2: sensitive to noises --> sensitive to noise.


It is usually hard for a learning system to predict correctly on the rare events, and there is no exception for segmentation algorithms. Therefore, we hope to build an alarm system to set off alarms when the segmentation result is possibly unsatisfactory. One plausible solution is to project the segmentation results into a low dimensional feature space, and then learn classifiers/regressors in the feature space to predict the qualities of segmentation results. In this paper, we form the feature space using shape feature which is a strong prior information shared among different data, so it is capable to predict the qualities of segmentation results given different segmentation algorithms on different datasets. The shape feature of a segmentation result is captured using the value of loss function when the segmentation result is tested using a Variational Auto-Encoder(VAE). The VAE is trained using only the ground truth masks, therefore the bad segmentation results with bad shapes become the rare events for VAE and will result in large loss value. By utilizing this fact, the VAE is able to detect all kinds of shapes that are out of the distribution of normal shapes in ground truth (GT). Finally, we learn the representation in the one-dimensional feature space to predict the qualities of segmentation results. We evaluate our alarm system on several recent segmentation algorithms for the medical segmentation task. The segmentation algorithms perform differently on different datasets, but our system consistently provides reliable prediction on the qualities of segmentation results.

This paper presents an approach to evaluate the quality of segmentations. To achieve this, a variational auto-encoder (VAE) is trained on the ground truth masks to extract shape-relevant information in the feature space, assuming that incorrect segmentations will be far from the normal distribution. Then, a regression model is trained to predict the quality of the segmentation based on the shape-learned features. The authors use several datasets focusing on pancreas segmentation to evaluate their quality-assessment approach, showing competitive performance with respect to other approaches.
The paper is well written, easy to follow in general, and the methodology is sound. Nevertheless, I have some concerns related to the applicability of this approach.
- Closely related works in the literature are missing:
There is a closely related recent work that used auto-encoders on the sets of ground-truth masks to build representations of shape and constrain the outputs of deep networks: Otkay et al., Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation, IEEE TMI 2017
This work does not focus directly on quality assessment. However, I believe the loss in this work, which evaluates the difference between the obtained segmentation (characterized by the outputs of a deep network) and an auto-encoder description of shape, can be used directly as a criterion for evaluating the quality of segmentation (on validation data) in term of consistency with the shape prior. I think this work is very closely related and should be discussed.
Also, a quick google search provided some missing references related to this work. I think including comparisons to the recent work in [1], for example, would be appropriate. As the focus is on quality assessment of medical image segmentation, I would suggest a deeper review of the literature.
[1] Vanya V Valindria, Ioannis Lavdas, Wenjia Bai, Konstantinos Kamnitsas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Reverse classification accuracy: Predict- ing segmentation performance in the absence of ground truth. IEEE Transactions on Medical Imaging, 2017.
[2] S. Chabrier, B. Emile, C. Rosenberger, and H. Laurent, “Unsupervised performance evaluation of image segmentation,” EURASIP Journal on Applied Signal Processing, vol. 2006, pp. 217–217, 2006.
[3] Gao H, Tang Y, Jing L, Li H, Ding H. A Novel Unsupervised Segmentation Quality Evaluation Method for Remote Sensing Images. Sensors. 2017 Oct 24;17(10):2427.
- The proposed quality assessment uses the learned shape features.  Even though it is strong prior information, there  might be situations where the predicted segmentation might be plausible in terms of shape, but is not a good segmentation.
-  I wonder how this approach works in problems with a high size/shape variation. For example, in the case of tumors, where their shape is unpredictable and each unknown case can be seen as a ‘rare’ example.
- To better capture the shape in the proposed approach, images need to be aligned, which limits the applicability of this approach to aligned volumes only.
- This approach gives a global hint about a given segmentation result, as a whole. I think it would be more interesting to provide local information on a segmentation, as it may happen that a predicted contour is generally correct, but there are some crispy borders in some points due to low contrast, for example. Even though the quality assessment would say that the prediction is correct, the contour may be unusable for certain applications, where a minimal surface distance is required (e.g., radiotherapy).
- As the quality assessment is based on shape and not in image information, it would be interesting to see how accurately it predicts the performance on different image modalities (for example, the method is trained on ground truth masks corresponding to CT images and quality is assessed in segmentations performed in MRI).
If I understood correctly, comparison with other methods is done with the same dataset under the same conditions (i.e., all the images are pre-aligned). As the other methods might not have the limitation of requiring aligned images, it would be interesting to compare also the performances in this situation.
How the training (or the VAE) is adapted for DeepLab-3, as it is based on 2D convolutions?
Minor: The paper needs a proof-read to fix some issues (e.g. ‘the properties of F is encoded’)


A ROTATION-EQUIVARIANT CONVOLUTIONAL NEURAL NETWORK MODEL OF PRIMARY VISUAL CORTEX
Classical models describe primary visual cortex (V1) as a filter bank of orientationselective linear-nonlinear (LN) or energy models, but these models fail to predict neural responses to natural stimuli accurately. Recent work shows that convolutional neural networks (CNNs) can be trained to predict V1 activity more accurately, but it remains unclear which features are extracted by V1 neurons beyond orientation selectivity and phase invariance. Here we work towards systematically studying V1 computations by categorizing neurons into groups that perform similar computations. We present a framework for identifying common features independent of individual neurons' orientation selectivity by using a rotation-equivariant convolutional neural network, which automatically extracts every feature at multiple different orientations. We fit this rotation-equivariant CNN to responses of a population of 6000 neurons to natural images recorded in mouse primary visual cortex using two-photon imaging. We show that our rotation-equivariant network outperforms a regular CNN with the same number of feature maps and reveals a number of common features, which are shared by many V1 neurons and are pooled sparsely to predict neural activity. Our findings are a first step towards a powerful new tool to study the nonlinear functional organization of visual cortex.

In this interesting study, the authors show that incorporating rotation-equivariant filters  (i.e. enforcing weight sharing across filters with different orientations) in a CNN model of the visual system is a useful prior to predict responses in V1. After fitting this model to data, they find that the RFs of model V1 cells do not resemble the simple Gabor filters of textbooks, and they present other quantitative results about V1 receptive fields. The article is clearly written and the claims are supported by their analyses. It is the first time to my knowledge that a rotation-equivariant CNN is used to model V1 cells.
The article would benefit from the following clarifications:
1. The first paragraph of the introduction discusses functional cell types in V1, but the article does not seem to reach any new conclusion about the existence of well-defined clusters of functional cell types in V1. If this last statement is correct, I believe it is misleading to begin the article with considerations about functional cell types in V1. Please clarify.
2. For clarity, it would help the reader to mention in the abstract, introduction and/or methods that the CNN is trained on reproducing V1 neuron activations, not on an image classification task as in many other studies (Yamins 2014, etc).
3. “As a first step, we simply assume that each of the 16 features corresponds to one functional cell type and classify all neurons into one of these types based on their strongest feature weight.” and “The resulting preferred stimuli of each functional type are shown in Fig. 6.“
Again, I think these statements are misleading because they suggest that V1 cells indeed cluster in distinct functional cell types rather than form a continuum. However, from the data shown, it is unclear whether the V1 cells recorded form a continuum or distinct clusters. Unless this question is clarified and the authors show the existence of functionally distinct clusters in their data, they should preferably not mention "cell types" in the text.
Suggestions for improvement and questions (may not necessarily be addressed in this paper):
4. “we apply batch normalization”
What is the importance of batch normalization for successfully training the model? Do you think that a sort of batch normalization is implemented by the visual system?
5. “The second interesting aspect is that many of the resulting preferred stimuli do not look typical standard textbook V1 neurons which are Gabor filters. ”
OK but the analysis consists of iteratively ascending the gradient of activation of the neuron from an initial image. This cannot be compared directly to the linear approximation of the V1 filter that is computed experimentally from doing a spike-triggered average (STA) from white noise. A better comparison would be to do a single-step gradient ascent from a blank image. In this case, do the filters look like Gabors?
6. Did you find any evidence that individual V1 neurons are themselves invariant to a rotation?
7. The article could be more self-contained. There are a lot of references to Klindt et al. (2017) on which this work is based, but it would be nice to make the article understandable without having to read this other article.
Typo: Number of fearture maps in last layer
Conclusion:
I believe this work is significant and of interest for the rest of the community studying the visual system with deep networks, in particular because it finds an interesting prior for modeling V1 neurons, that can probably be extended to the rest of the visual system. However, it would benefit from the clarifications mentioned above.


Existing methods of image to image translation require multiple steps in the training or modification process, and suffer from either an inability to generalize, or long training times. These methods also focus on binary trait modification, ignoring continuous traits. To address these problems, we propose ModifAE: a novel standalone neural network, trained exclusively on an autoencoding task, that implicitly learns to make continuous trait image modifications. As a standalone image modification network, ModifAE requires fewer parameters and less time to train than existing models. We empirically show that ModifAE produces significantly more convincing and more consistent continuous face trait modifications than the previous state-of-the-art model.

Overview and contributions: The authors propose the ModifAE model that is based on an autoencoder neural network for continuous trait image modifications. ModifAE requires fewer parameters and less time to train than existing generative models. The authors also present experiments to show that ModifAE produces more convincing and more consistent continuous face trait modifications than the current baselines.
Strengths:
1. Nice presentation of the model.
2. Good experiments to justify improved running time and fewer number of parameters.
Weaknesses:
1. I am not completely convinced by the results in Figure 4. It doesn't seem like the model is able to pick up on subtle facial expressions and generate them in a flexible manner. In fact, the images look very very similar regardless of the value of the traits. Furthermore, the authors claim that "In general, as she becomes more emotional, her smile increases, and as she is made more attractive, her smile increases as well, as smiling subjects are judged as more attractive". I believe attractiveness and emotions are much more diverse and idiosyncratic than just the size of her smile...
2. From Figure 5 it seems like ModifAE generates images that are lower in quality as compared to StarGAN. Can the authors comment on this point? How can ModifAE be improved to generate higher-quality images?
Questions to authors:
1. Weakness points 1 and 2.
2. This did not affect my rating, but I am slightly concerned by the labelings as seen in Figure 1. Is it reasonable to infer traits like "trustworthy", "attractive", "aggressive", "responsible" from images? Are these traits really what we should be classifying people's faces as, and are there any possible undesirable/sensitive biases from the dataset that our models could learn? I would like to hear the author's opinions on the ethical implications of these datasets and models.
Presentation improvements, typos, edits, style, missing references:
None


EQUI-NORMALIZATION OF NEURAL NETWORKS
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the SinkhornKnopp algorithm, we introduce a fast iterative method for minimizing the 2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batchand group-normalization on CIFAR-10 and ImageNet with a ResNet-18.

Summary:
This paper introduces equi-normalization (ENorm): a normalization technique that relies on the scaling invariance properties of the ReLU, similarly to Path-SGD. Their method explicitly use this property to balance the weights of the network, without changing the function computed by the network. The main difference with Path-SGD is that the network is explicitly balanced, while Path-SGD uses a regularizer to implicitly balance the network. Since it doesn’t rely on mini-batch to normalize the network, Equi-normalization could be a good alternative to BN in small mini-batch regime. The method is validated on 3 tasks (MLP on CIFAR10, CNN on CIFAR10, Reidual Network on ImageNet).
Clarity:
The paper is quite clear, although a bit long (10 pages). The related work section is particularly nice. I really appreciated the “positioning” paragraph, which really explains how the method differs from others.
Novelty:
The paper is quite incremental, due to its similarities with Path-SGD. It seems quite close to what Weight Normalization is doing as well (see detailed comments).
Pros and Cons:
+ Clearly written
+ Clearly motivated
+ Nice review of literature
- Quite incremental (close to Path-SGD / Weight Normalization), and missing actual comparison with Weight Normalization, which seems to be the direct competitor of ENorm (see detailed comments)
- Some flaws in the experimental setup (see detailed comments), particularly in the fully-connected experiment.
- Doesn’t scale (yet) to deeper architectures, which is precisely where small batch sizes become a problem for BN and thus where ENorm would be needed.
Detailed Comments:
1. Differences with Weight Normalization:
I have trouble seeing the difference between the proposed method and Weight Normalization (WN), or the more advanced Normalization Propagation (NP). It seems that both methods are performing quite similar normalization: WN reparameterize the network such that ||W[:,j]||_2 = 1, so it seems that you wouldn’t need to re-balance the network when using WN. Could the authors elaborate on this? Moreover WN is simple to implement, fast, and also works in the small batches settings as well. Finally, since those methods are quite similar, the authors should compare their method against WN in their experimental setup.
2. Initialization of the Weights:
The Xavier initialization you are using in the CIFAR10 experiments is designed to work with Tanh activations functions. For ReLUs, one should use the Kaiming initialization, which has been proven to work way better for ReLUs (He et al., 2015a).  This certainly explains the poor performances of your baseline when you increase the number of layers in Figure 4, and probably explains why you need to add a BN layer at the end of your network to help with the training of the baselines. I suggest the authors to re-run the baselines using proper initialization for ReLUs.
3. Fully-Connected Layers Benchmark:
I think the fully-connected benchmark you used is quite poor. A baseline with 1 layer only reaches ~54 % test accuracy and your method needs a 11 layer model to increase this baseline performance of about ~0.5 % only. The deep autoencoder on MNIST (see e.g. Desjardins et al., Natural neural networks, NIPS 2015), would probably be a better benchmark for fully-connected layers (of course using ReLUs in place of Sigmoids).  It would also reinforce your empirical results by adding a 3rd dataset.
4. ImageNet Experiment:
Do you use Ghost Batch Normalization in this experiments (i.e. calculating the BN statistics on each GPU separately)? It would certainly explain the poor performances of BN with tiny batch size (32 examples on 8 GPUs means only 8 examples per GPU).
5. Computation Time performances:
It is stated in the conclusion that “using ENorm during training adds a much smaller computational overhead than BN or GN …”. I can see that Table 1 gives an overview of the number of elements accessed during normalization, but I do think that a proper plot showing the accuracy versus the wall-clock time would be a better way of showing how your method compares in practice with BN or GN. Moreover, as stated previously, comparison with WN (and / or NP) need to be performed as well, especially because WN and NP are also way faster to compute than BN and GN.
6. Shortening the paper:
The recommended paper size for ICLR is 8 pages. Here would be a few pointers that could help you reduce a bit the length of the paper: Introduction and Related Work takes up 3 pages already and I think there is quite some overlap between the 2 (about BN in particular), so there is probably quite a lot of space to gain here if the authors were to reduce a bit the introduction or make the related work a sub-section of the introduction. The ENorm presentation is 4 pages long (which is quite a lot). Section 3.6 might be totally discarded since it is vanilla application of the chain rule. The extension to convolution layers and max pooling could be transferred to the appendix.
Minor Comments:
You should add “Optimizing neural networks with kronecker-factored approximate curvature” in the literature review about optimization landscape, as it is an important research direction on natural gradient.
Conclusion:
All in all, I find that this work is a bit too incremental, missing some important comparisons with other techniques and its experimental setup could definitely be improved. Also, the speedup claims should also be supported with empirical experimentation.
Revision:
I thank the authors for the all the extra experiments they performed. The paper looks good to me, and increased my evaluation accordingly.


With innovations in architecture design, deeper and wider neural network models deliver improved performance on a diverse variety of tasks. But the increased memory footprint of these models presents a challenge during training, when all intermediate layer activations need to be stored for back-propagation. Limited GPU memory forces practitioners to make sub-optimal choices: either train inefficiently with smaller batches of examples; or limit the architecture to have lower depth and width, and fewer layers at higher spatial resolutions. This work introduces an approximation strategy that significantly reduces a network's memory footprint during training, but has negligible effect on training performance and computational expense. During the forward pass, we replace activations with lower-precision approximations immediately after they have been used by subsequent layers, thus freeing up memory. The approximate activations are then used during the backward pass. This approach limits the accumulation of errors across the forward and backward pass-because the forward computation across the network still happens at full precision, and the approximation has a limited effect when computing gradients to a layer's input. Experiments, on CIFAR and ImageNet, show that using our approach with 8-and even 4-bit fixed-point approximations of 32-bit floating-point activations has only a minor effect on training and validation performance, while affording significant savings in memory usage.

The authors detail a procedure to reduce the memory footprint of deep networks by quantization of the activations only on back propagation. While this scheme does not benefit from computational speedups of activation quantization on both passes (and indeed has a slight computational overhead), the authors demonstrate that for common convolutional architectures it nicely preserves the accuracy of computation by computing the forward pass at full accuracy and limiting propagation of errors in the backward pass. This is possible because the majority of errors are introduced in gradient calculation of the weights and not the inputs each layer. The authors also wisely perform quantization after batch normalization and use the known mean and variance of the activations to scale the quantization and reduce errors. They demonstrate very slight drops in performance accuracy for ResNets on Cifar10, Cifar100, and ImageNet with memory compression factors up to 8. They also point to natural future directions such as using vector quantization to better leverage the activation statistics. The paper is also very clearly written with appropriate references to the relevant literature.
An area of improvement I could see for the paper would be to demonstrate the utility of the reduced memory footprint. Their motivation clearly outlines that reducing memory can allow for larger batch sizes and larger networks that can improve the performance of training, but the authors do not demonstrate an example of this principle. They do mention that they are able to train with a larger batch size on ImageNet without combining batches, but more quantitative evidence of improvements in wall clock time (for different batch sizes) or improvement in performance (for larger networks) would help support the arguments of the paper. Given that the authors are focusing on single device training, they don't have to necessarily improve the state of the art, but a relative comparison would be illustrative. Also, specific measurements of the change in memory footprint for real networks would be helpful.


GENERATIVE CODE MODELING WITH GRAPHS
Generative models for source code are an interesting structured prediction problem, requiring to reason about both hard syntactic and semantic constraints as well as about natural, likely programs. We present a novel model for this problem that uses a graph to represent the intermediate state of the generated output. Our model generates code by interleaving grammar-driven expansion steps with graph augmentation and neural message passing steps. An experimental evaluation shows that our new model can generate semantically meaningful expressions, outperforming a range of strong baselines.

The paper proposes a code completion task that given the rest of a program, predicts the content of an expression. This task has similarity to code completion tasks in the code editor of an IDE. The paper proposes an interesting problem, but the paper would benefit if writing and evaluation are significantly improved.
The work builds on prior research by Allamanis et al. 2018b that performs such completions of single variables by picking from the variables in the scopes. The difference here is that portions of parse trees are predicted as opposed to a single variables, where the algorithm from the prior research is used to predict single variables.
Writing-wise the paper is hard to read on the technical part with many unclear details and this portion needs a good amount of extra explanations. The Epsilon set includes triples which are not described and need understanding equation (2). The first element of this triple is an edge label <edge>($a$, $v$) where $a$ is an AST and $v$ is a node. Thus, edges of the graph end up between entire ASTs and nodes? While I can see how could this make sense, there is certainly lack of explanation going on here. Overall, this part is hard to parse and time-consuming to understand except at high level. Furthermore, the text has many functions without signatures and they seem to be used before they are defined (e.g. getRepresentation).
Technically, the approach also seems very similar to N3NN by Parisotto et al, ICLR 2017. There should be more elaboration on what is new here. Otherwise, the novelty of the paper really is just combining this work with Allamanis et al. 2018b.
In terms of evaluation, the task seems to be on a different set of expressions than the one explained in the exposition. How many expressions where there in the evaluation programs and how many were chosen to evaluate on and based on what criteria. It seems from the exposition that expressions with field accessed and function calls are not possible to be generated, but then some completions show method calls. How much of the full task is actually solved? In particular, several of the cited prior works solve specific problems like constants that are ignored here.
The evaluation is mostly an ablation studies of the proposed approach by removing edges from the final idea.
Besides this, the paper also introduces a new dataset for showcasing the technique and does not report sizes and running times, essentially not answering basic questions like what is the trade-off between the different techniques. Comparison to actual prior works on similar tasks is also lacking (some TODO is left in the paper), but there is the claim that existing neural techniques such as seq2seq perform "substantially worse". I guess the authors have extra experiments not included for lack of space or that the evaluation was not ready at submission time.


We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).

[UPDATE]
I find the revised version of the paper much clearer and streamlined than the originally submitted one, and am mostly content with the authors reply to my comments. However, I still think the the work would highly benefit from a non-heuristic justification of its approach and some theoretic guarantees on the performance of the proposed framework (especially, in which regimes it is beneficial and when it is not). Also, I still find the presentation of experimental results too convoluted to give a clear and comprehensive picture of how this methods compares to the competition, when is it better, when is it worse, do the observations/claim generalize to other task, and which are the right competing methods to be considering. I think the paper can still be improved on this aspect as well.
As I find the idea (once it was clarified) generally interesting, I will raise my score to 6.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The paper proposes an objective function for learning representations, termed the conditional entropy bottleneck (CEB). Variational bounds on the objective function are derived and used to train classifiers according to the CEB and compare the results to those attained by competing methods. Robustness and adversarial examples detection of CEB are emphasized.
My major comments are as follows:
1) The authors base their 'information-theoretic' reasoning on the set-theoretic structure of Shannon’s information measures. It is noteworthy that when dealing with more than 2 random variables, e.g., when going from the twofold I(X;Y) to the threefold I(X;Y;Z), this theory has major issues. In particular, there are simple (and natural) examples for which I(X;Y;Z) is negative. The paper presents an information-theoretic heuristic/intuitive explanation for their CEB construction based on this framework. No proofs backing up any of the claims of performance/robustness in the paper are given. Unfortunately, with such counter-intuitive issues of the underlying theory, a heurisitc explanation that motivates the proposed construction is not convincing. Simulations are presented to justify the construction but whether the claimed properties hold for a wide variety of setups remain unclear.
2) Appendix A is referred to early on for explaining the minimal necessary information (MNI), but it is very unclear. What is the claim of this Appendix? Is there a claim? It's just seems like a convoluted and long explanation of mutual information. Even more so, this explanation is inaccurate. For instance, the authors refer to the mutual information as a 'minimal sufficient statistic' but it is not. For a pair of random variables (X,Y), a sufficient statistic, say, for X given Y is a function f of Y such X-f(Y)-Y forms a Markov chain. Specifically, f(Y) is another random variable. The mutual information I(X;Y) is just a number. I have multiple guesses on what the authors' meaning could be here, but was unable to figure it out from the text. One option, which is a pretty standard way to define sufficient statistic though mutual information is as a function f such that I(X;Y|f(Y))=0. Such an f is a sufficient statistic since the zero mutual information term is equivalent to the Markov chain X-f(Y)-Y from before. Is that what the authors mean..?
3) The Z_X variable introduced in Section 3 in inspired by the IB framework (footnote 2). If I understand correctly, this means that in many applications, Z_X is specified by a classifier of X wrt the label Y. My question is whether for a fixed set of system parameters, Z_X is a deterministic function of X? If this Z_X play the role of the sufficient statistics I've referred to in my previous comment, then it should be just a function of X.
However, if Z_X=f(X) for a deterministic function f, then the CEB from Equation (3) is vacuous for many interesting cases of (X,Y). For instance, if X is a continuous random variable and Z_X=f(X) is continuous as well, then
I(X;Z_X|Y)=h(Z_X|Y)-h(Z_X|X,Y)
where h is the differential entropy and the subtracted terms equals -\infty by definition (see Section 8.3 of (Cover & Thomas, 2006). Consequently, the mutual information and the CEB objective are infinite. If Z_X=f(X) is a mixed random variable (e.g., can be obtain from a ReLU neural network), then the same happens. Other cases of interest, such as discrete X and f being an injective mapping of the set of X values, are also problematic. For details of such problem associated with IB type terms see:
[1] R. A. Amjad and B. C. Geiger 'Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle', 2018 (https://arxiv.org/abs/1802.09766).
Can the authors account for that?
4) The other two reviews addressed the missing accounts for past literature. I agree on this point and will keep track of the authors' responses. I will not comment on that again.
Beyond these specific issue, they text is very wordy and confusing at times. If some mathematical justification/modeling was employed the proposed framework might have been easier to accept. The long heuristic explanations employed at the moment do not suffice for this reviewer. Unless the authors are able to provide clarification of all the above points and properly place their work in relation to past literature I cannot recommend acceptance.


In this paper, we propose the StochAstic Recursive grAdient Policy Optimization (SARAPO) algorithm which is a novel variance reduction method on Trust Region Policy Optimization (TRPO). The algorithm incorporates the StochAstic Recursive grAdient algoritHm(SARAH) into the TRPO framework. Compared with the existing Stochastic Variance Reduced Policy Optimization (SVRPO), our algorithm is more stable in the variance. Furthermore, by theoretical analysis the ordinary differential equation and the stochastic differential equation (ODE/SDE) of SARAH, we analyze its convergence property and stability. Our experiments demonstrate its performance on a variety of benchmark tasks. We show that our algorithm gets better improvement in each iteration and matches or even outperforms SVRPO and TRPO.
In this paper, we focus on studying variance reduction method on TRPO. We discussed an alternative variance reduction method called SARAH, recently proposed by (Nguyen et al., 2017) whose convergence rate matches that of SVRG in the convex case and has the inner-loop convergence property that SVRG does not possess. We propose a new algorithm named SARAPO which hybrids the SARAH method with TRPO and analyze the dynamics of SVRG/SARAH on natural gradient updates via its approximating ordinary and stochastic differential equation (ODE/SDE).
In experiments, we test the performance and sample efficiency of SARAPO, SVRPO, and TRPO. To allow larger steps in SVRPO/SARAPO, we remove the KL constraint in the line search of TRPO. We verify that variance reduction enables a larger leap towards the optima and find that SARAPO outperforms TRPO and matches the performance of SVRPO in most of the tasks.
The rest of this paper is organized as the following: we introduce some background, propose and analyze our algorithm in Section 2. In Section 3, we theoretically analyze our proposed SARAPO algorithm from a dynamical system viewpoint. We present the numerical results of experiments in Section 4 and conclude our paper in Section 5.

This paper investigates how the SARAH stochastic recursive gradient algorithm can be applied to Trust Region Policy Optimization. The authors analyze the SARAH algorithm using its approximating ordinary and stochastic differential equations. The empirical performance of SARAPO is then compared with SVRPO and TRPO on several benchmark problems.
Although the idea of applying SARAH to reduce the variance of gradient estimates in policy gradient algorithms is interesting and potentially quite significant (variance of gradient estimates is a major problem in policy gradient algorithms), I recommend rejecting this paper at the present time due to issues with clarity and quality, particularly of the experiments.
Not enough of the possible values for experimental settings were tested to say anything conclusive about the performance of the algorithms being compared. For the values that were tested, no measures of the variability of performance or statistical significance of the results were given. This is important because the performance of the algorithms is similar on many of the environments, and it is important to know if the improved performance of SARAPO observed on some of the environments is statistically significant or simply due to the small sample size.
The paper also needs improvements in clarity. Grammatical errors and sentence fragments make it challenging to understand at times. Section 2.3 seemed very brief, and did not include enough discussion of design decisions made in the algorithm. For example, the authors say ``"the Fisher Information Matrix can be approximated by Hessian matrix of the KL divergence when the current distribution exactly matches that of the base distribution" but then suggest using the Hessian of the KL of the old parameters and the new parameters which are not the same. What are the consequences of this approximation? Are there alternative approaches?
The analysis in section 3 is interesting, but the technique has been applied to SGD before and the results only seem to confirm findings from the original SARAH paper.
To improve the paper, I would suggest moving section 3 to an appendix and using the extra space to further explain details and conduct additional simpler experiments. Additional experiments on simpler environments and policy gradient algorithms (REINFORCE, REINFORCE with baseline) would allow the authors to try more possible values for experimental settings and do enough runs to obtain more conclusive results about performance. Then the authors can present their results applying SARAH to TRPO with some measure of statistical significance.


Beyond understanding what is being discussed, human communication requires an awareness of what someone is feeling. One challenge for dialogue agents is recognizing feelings in the conversation partner and replying accordingly, a key communicative skill that is trivial for humans. Research in this area is made difficult by the paucity of suitable publicly available datasets both for emotion and dialogues. This work proposes a new task for empathetic dialogue generation and EMPATHETICDIALOGUES, a dataset of 25k conversations grounded in emotional situations to facilitate training and evaluating dialogue systems. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, while improving on other metrics as well (e.g. perceived relevance of responses, BLEU scores), compared to models merely trained on large-scale Internet conversation data. We also present empirical comparisons of several ways to improve the performance of a given model by leveraging existing models or datasets without requiring lengthy re-training of the full model.

Overall this paper contributes many interesting insights into the specific application of empathetic dialog into chatbot responses.  The paper in particular is contributing its collected set of 25k empathetic dialogs, short semi-staged conversations around a particular seeded emotion and the results of various ways of incorporating this training set into a generative chatbot.
While the results clearly do not solve the problem of automating emapthy, the paper does give insights into which methods perform better than others (Generation vs Retrieval) and explicitly adding emotion predictions vs using an ensemble of encoders.
There is a lot in this paper, and I think it could have been better organized.
I am more familiar with emotion related research and not language to language translation, so I would have appreciated a better explanation of the rationale for using BLEU scores.  I did some online research to understand these Bilingual Evaluation Understudy Scores and while it seems like they measure sentence similarity, it is unclear how they capture ”relevance” at least according to the brief tutorial that I read (https://machinelearningmastery.com/calculate-bleu-score-for-text-python/).  I did not see the paper describing the use of this score in the references but perhaps I missed it – could you please clarify why this is a good metric for relevance?  It seems that these scores are very sensitive to sentence variation.  I am not sure if you can measure empathy or appropriateness of a response using this metric.
For your data collection you have 810 participants and 24,850 conversations.  Are the 810 participants all speakers or speakers and listeners combined?  How many conversations did each speaker/listener pair perform 32?  (one for each emotion) or 64? (two for each emotion) Was the number variable?  If so what is the distribution of the contribution – e.g. did one worker generate 10,000 while several hundred workers did only three of four?  Was it about even?  Just for clarity – how did you enroll participants?  Was it through AMT?  What were the criteria for the workers?  E.g. Native English speaker, etc.
In your supplemental material, I found the interchanging of the words “context” and “emotion” confusing.  The word context is used frequently throughout your manuscript: “dialog context,” “situational context” - emotions are different from situations, the situational utterance is the first utterance describing the emotion if I read your manuscript correctly.  Table 6 should use “Label” or “Emotion” instead of the more ambiguous “Context.”
My understanding is that speakers were asked to write about a time when they experienced a particular feeling and they were given a choice of three feelings that they could write about.  You then say that workers are forced to select from contexts they had not chosen before to ensure that all of the categories were used.  From this I am assuming that each speaker/listener worker pair had to write about all 32 emotions – is this correct?  Another interpretation of this is that you asked new workers to describe situations involving feelings that had not been chosen by other workers as data collection progressed to ensure that you had a balanced data set.  This would imply that some emotional situations were less preferred and potentially more difficult to write about.  It would be interesting if this data was presented.  It might imply that some emotion labels are not as strong if people were forced to write about them rather than being able to choose to write about them.
Were these dialogs ever actually annotated?  You state in section 2, Related Work “we train models for emotion detection on conversation data that has been explicitly labeled by annotators” – please describe how this was done.  Did independent third party annotators review the dialogs for label correctness?  Was a single rater or a majority vote used to decide the final label.  For example, in Table 1, the label “Afraid” is given to a conversation that could also have reasonable been generated by the label “Anxious” a word explicitly used in the dialog.  I am guessing that the dialogs are just labeled according to the label / provocation word and that they were not annotated beyond that, but please make this clear.
In the last paragraph you state “A few works focus..” and then list 5.  This should rather be “several other works have focused on “ …
Conversely, you later state in section 3 “Speaker and Listener”, “We include a few example conversations from the training data in Table 1,” this should more explicitly be “two.”
Also in section 3 when you describe your cross validation process, you state “We split the conversations into approximately 80/10/10 partitions.  To prevent overlap of <<discussed topics>> we split the data so that all the sets of conversations with the same speaker providing the prompt would be in the same partition.
In your supplemental material you state that workers were paired.  Each worker is asked to write a prompt, which also seems to be the first utterance in the dialog they will start.  You state each worker selects one emotion word from a list of three which is somehow generated (randomly?) form your list of 32 .  I am assuming each worker in the pair does this, then the pair has a two “conversations” one where the first worker is the speaker and another where the second worker is the speaker – is this correct?  It is not entirely clear from the description. Given that you have 810 workers and 24,850 conversations, I am assuming that each worker had more than one conversation.  My question is  - did they generate a new prompt / first utterance for each conversations.  I am assuming yes since you say there are 24,850 prompts/conversations.  For each user are all of the situation/prompts they generate  describing the same emotion context?  E.g. would one worker write ~30 conversations on the same emotion.  This seems unlikely, and it seems more likely that given the number of conversations ~30 per participant is similar to the number of emotion words that you asked each worker to cycle through nearly all of the emotions or that given they were able to select, they might describe the same emotion, e.g. “fear” several times.  If the same worker was allowed to select the same emotion context multiple times was it found that they re-used the same prompt several times?  I am assuming that this is the case and that this is what you mean when you say that you “prevent overlap of discussed topics” between sets when you exclude particular workers.  Is this correct?  Or did you actually look and code the discussed topics to ensure no overlap even across workers (e.g. several people might have expressed fear of heights or fear of the dark).
In section 4, Empathetic dialog generator, you state that the dialog model has access to the situation description given by the speaker (also later called the situational prompt) but not the emotion word prompt.  Calling these both prompts makes the statement about 24,850 prompts/conversations a bit ambiguous.  A better statement would be 24,850 conversations based on unique situational prompts/descriptions (if they are in fact unique situational prompts.  I am assuming they are not if you are worried about overlapping “discussed topics” which I am assuming are the situational prompts since the dialogs are very short and heavily keyed off these initial situational prompts)
In your evaluation of the models with Human ratings you describe two sets of tests.  In one test you say you collect 100 annotations per model.  More explicitly, did you select 100 situational prompts and then ask workers to rate the response of each model?  Was how many responses was each worker shown?  How many workers were used?  Are the highlighted numbers the only significant findings or just the max scores?  Annotations is probably not the correct word here.
Please also describe your process for assigning workers to the second human ratings task.
Since the two novel aspects of your paper are the new dataset and the use of this dataset to create more empathetic chatbot responses ("I know the feeling") I have focused on these aspects of the paper in my review.
I found the inclusion of Table 7 underexplained in the text.  The emotion labels for all these datasets are not directly comparable so I would have liked to have seen more explanation around how these classifications were compared.  It would also be helpful to know how more similar emotions such as "afraid" and "anxious" were scored vs "happy" and "sad" confusions


FROM LANGUAGE TO GOALS: INVERSE REINFORCE- MENT LEARNING FOR VISION-BASED INSTRUCTION FOLLOWING
Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a languageconditioned policy leads to poor performance.

Summary:
This paper proposes learning reward functions via inverse reinforcement learning (IRL) for vision-based instruction following tasks like "go to the cup". The agent receives the language instruction (generated via grammar templates) and a set of four images (corresponding to four cardinal directions) from virtual cameras mounted on the agent as input at every time step and its aim is either to 1. navigate to the goal location (navigation task) or 2. move an object from one place to another (pick task).
The really interesting part in this paper is learning reward functions such that they generalize across different tasks and environments (e.g. indoor home layouts). This differentiates it from the standard IRL setting where reward functions are learnt and then policies optimized on this reward function on the *same* environment.
In order to generalize across tasks and environments a slight modification to the max-ent IRL gradient equations are made: 1. Similar to meta-learning the gradient is taken with respect to multiple tasks (in a sampling-based manner) and 2. Making the reward function a function of not just states and actions but also language context. The overall algorithm (Algorithm 1) is simple and the critical step of computing an optimal policy to compute the IRL gradient is done by assuming that one has access to full state and dynamics and essentially running a planner on the MDP. This assumption is not unreasonable since in a simulator one has access to the full dynamics and can hence one can compute the optimal trajectories by planning.
Experiments are presented on the SUNCG dataset of indoor environments. Two baselines are presented: One using behavior cloning (BC) and an oracle baseline which simply regresses to the ground truth reward function which is expected to be an upper bound of performance. Then DQN is used (with and without reward shaping) using the learnt reward functions to learn policies which are shown to have better performance on different tasks.
Comments and Questions:
- The paper is generally well-written and easy to understand. Thanks!
- The idea of using IRL to learn generalizable reward functions to learn policies so as to aid transfer between environments in such vision-language navigation tasks is interesting and clearly shows benefits to behavior cloning.
- One of my main concerns (and an interesting question that this paper naturally raises) is how does this approach compare to imitation learning (not vanilla behavior cloning which is straight-up supervised learning and has been theoretically and empirically shown to have worse performance due to distribution shifts. See Ross and Bagnell, 2011, Ross, Gordon, Bagnell 2012 (DAgger, Ross and Bagnell 2014 (AggreVate), Chang et al., 2015 (LOLS), etc). If the same budget of 10 demonstrations per environment is used via DAgger (where say each iteration of DAgger gets say 2 or 3 demonstrations until the budget is exhausted) how does it compare? Note online version of DAgger has already been used in similar settings in "Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments" by Anderson et al, CVPR 2018. The main difference from their setting is that this paper considers higher level tasks instead of taking as input low-level turn-by-turn language inputs.
- The following papers are relevant and should be cited and discussed:
"Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments" by Anderson et al, CVPR 2018.
"Embodied Question Answering", Das et al, CVPR 2018.
Update:
------------
After looking at other reviews and author rebuttals to all reviews I am raising my grade.


Model compression can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and accuracy, popular compression techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio of each layer. This process is typically costly and suboptimal. In this paper, we propose a Multi-Layer Pruning method (MLPrune), which is theoretically sound, and can automatically decide appropriate compression ratios for all layers. Towards this goal, we use an efficient approximation of the Hessian as our pruning criterion, based on a Kroneckerfactored Approximate Curvature method. We demonstrate the effectiveness of our method on several datasets and architectures, outperforming previous state-of-theart by a large margin. Our experiments show that we can compress AlexNet and VGG16 by 25x without loss in accuracy on ImageNet. Furthermore, our method has much fewer hyper-parameters and requires no expert knowledge.

This paper introduces an approach to pruning the parameters of a trained neural network. The idea is inspired by the Optimal Brain Surgeon method, which relies on second derivatives of the loss w.r.t. the network parameters. Here, the corresponding Hessian matrix is approximated using the Fisher information to make the algorithm scalable to very deep networks.
Strengths:
- The method does not require hyper-parameter tuning.
- The results show the good behavior of the approach.
Weaknesses:
Novelty:
- In essence, this method relies on the work of Marten & Grosse to approximate the Hessian matrix used in the Optimal Brain Surgeon strategy. This is fine, but not of great novelty.
Method:
- It is not clear to me why the notion of binary parameters gamma is necessary. Instead of varying the gammas from 1 to 0, why not directly zero out the corresponding network parameters w?
- In essence, the objective function in Eq. 5 adds an L_1 penalty on the gamma parameters, which would be related to an L_1 penalty on the ws. Note that this strategy has been employed in the past, e.g., Collins & Kohli, 2014, "Memory Bounded Deep Convolutional Networks".
- It is not clear to me how zeroing out individual parameters will truly allows one to reduce the model afterwards. In fact, one would rather want to remove entire rows or columns of the matrix W_l, which would truly correspond to a smaller model. This was what was proposed by Wen et al., NIPS 2016 and Alvarez & Salzmann, NIPS 2016, "Learning the Number of Neurons...".
- In the past, when dealing with the Hessian matrix, people have used the so-called Pearlmutter trick (Pearlmutter, Neural Computation 2014, "Fast exact multiplication by the Hessian". In fact, in this paper, the author mentions the application to the Optimal Brain Surgeon strategy. Is there a benefit of the proposed approach over this alternative strategy?
Experiments:
- While the reported compression rates are good, it is not clear to me what they mean in practice, because the proposed algorithm zeroes out individual parameters in the matrix W_l of each layer.  This does not guarantee entire channels to be removed. As such, I would not know how to make the model actually smaller in practice. It would seem relevant to show the true gains in memory usage and in inference speed (both measured on the computer, not theoretically).
Summary:
I do appreciate the fact that the proposed method does not require hyper-parameters and that it seems to yield higher compression rates than other pruning strategies that act on individual parameters. However, novelty of the approach is limited, and I am not convinced of its actual benefits in practice.


MEASURING COMPOSITIONALITY IN REPRESENTATION LEARNING
Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs' learned representations. While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces. We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives. We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization.
Figure 1: Representations arising from a communication game. In this game, an observation (b) is presented to a learned speaker model (c), which encodes it as a discrete character sequence (d) to be consumed by a listener model for some downstream task. The space of inputs has known compositional structure (a). We want to measure the extent to which this structure is reflected (perhaps imperfectly) in the structure of the learned codes.
The success of modern representation learning techniques has been accompanied by an interest in understanding the structure of learned representations. One feature shared by many humandesigned representation systems is compositionality: the capacity to represent complex concepts (from objects to procedures to beliefs) by combining simple parts (Fodor & Lepore, 2002). While many machine learning approaches make use of human-designed compositional analyses for representation and prediction (Socher et al., 2013;Dong & Lapata, 2016), it is also natural to ask whether (and how) compositionality arises in learning problems where compositional structure has not been built in from the start. Consider the example in Figure 1, which shows a hypothetical character-based encoding scheme learned for a simple communication task (similar to the one studied by Lazaridou et al., 2016). Is this encoding scheme compositional? That is, to what extent can we analyze the agents' messages as being built from smaller pieces (e.g. pieces xx meaning blue and bb meaning triangle)?
A large body of work, from early experiments on language evolution to recent deep learning models (Kirby, 1998;Lazaridou et al., 2017), aims to answer questions like this one. But existing solutions rely on manual (and often subjective) analysis of model outputs (Mordatch & Abbeel, 2017), or at best automated procedures tailored to the specifics of individual problem domains (Brighton & Kirby, 2006). They are difficult to compare and difficult to apply systematically.
We are left with a need for a standard, formal, automatable and quantitative technique for evaluating claims about compositional structure in learned representations. The present work aims at first steps toward meeting that need. We focus on an oracle setting where the compositional structure of model inputs is known, and where the only question is whether this structure is reflected in model outputs. This oracle evaluation paradigm covers most of the existing representation learning problems in which compositionality has been studied.
The first contribution of this paper is a simple formal framework for measuring how well a collection of representations (discrete-or continuous-valued) reflects an oracle compositional analysis of model inputs. We propose an evaluation metric called TRE, which provides graded judgments of compositionality for a given set of (input, representation) pairs. The core of our proposal is to treat a set of primitive meaning representations as hidden, and optimize over them to find an explicitly compositional model that approximates the true model as well as possible. For example, if the compositional structure that describes an object is a simple conjunction of attributes, we can search for a collection of "attribute vectors" that sum together to produce the observed object representations; if it is a sparse combination of (attribute, value) pairs we can additionally search for "value vectors" and parameters of a binding operation; and so on for more complex compositions.
Having developed a tool for assessing the compositionality of representations, the second contribution of this paper is a survey of applications. We present experiments and analyses aimed at answering four questions about the relationship between compositionality and learning:
• How does compositionality of representations evolve in relation to other measurable model properties over the course of the learning process? (Section 4) • How well does compositionality of representations track human judgments about the compositionality of model inputs? (Section 5) • How does compositionality constrain distances between representations, and how does TRE relate to other methods that analyze representations based on similarity? (Section 6) • Are compositional representations necessary for generalization to out-of-distribution inputs?
(Section 7)
We conclude with a discussion of possible applications and generalizations of TRE-based analysis.

Edit and a further question: Reading again Section 7, I'm wondering whether the  the high generalization is possible due to the fact that at test time only one of the two candidates is unseen, and the other is seen. Having *both* candidates to be unseen makes the problem significantly harder since the only way for the listener to get it right is to associate the message with the right candidate, rather than relying in some other strategy like whether the message is novel (thus it's the seen candidate) or new (thus it's the unseen candidate). As such, I don't think I can fully trust your conclusions due to this potential confounder.
--------------------------------------------------------------
The authors propose a measure of compositionality in representations. Given instances of data x annotated with semantic primitives, the authors learn a vector for each of the primitive such that the addition of the vectors of the primitives is very close (in terms of cosine) to the latent representation  z of the input x. The authors find that this measure correlates with the mutual information between the input x and z, approximates the human judges of compositionality on a language dataset and finally presents a study on the relation between the proposed measure and generalizalization performance, concluding that their measure correlates with generalization error as well as absolute test accuracy.
This in an interesting study and attacks a very fundamental question; tracking compositionality in representations could pave the way towards representations that facilitate transfer learning and better generalization. While the paper is very clear with respects to results, I found the presentation of the proposed measure overly confusing (and somewhat more exaggerated that what is really going on).
The authors start with a very clean example, that can potentially facilitate clarifying in a visual way the process of obtaining the measure. However, I feel that clarity is being traded-off for formality. It needs several reads to really distill the idea that essentially the authors are simply learning vectors of primitives that when added should resemble the representation of the input. Moreover, the name of the measure is a bit misleading and not justified by the experiments and the data. The authors do not deal with trees in any of the examples, but rather with a set of primitives (apparent in the use of addition as a composition function which being commutative does not allow for word-order and the like deep syntactic properties).
Now, onto the measure. I like the idea of learning basis vectors from the representations and constraining to follow the primitive semantics. Of course, this constraints quite a bit the form of compositionality that the authors are searching for.
The idea of additive semantics has been explored in NLP, however it's mostly applicable for primitives with intersective semantics (e.g., a white towel is something that is both white and a towel). Do the authors think that this restricts their experiments (especially the natural languages ones)? What about other composition techniques found in the literature of compositional semantics (e.g., by Baroni and Zamparelli, 2010).
This is good to be clarified.  Moreover, given the simplicity of the datasets in the current study, wouldn't a reasonable baseline be to obtain the basis vector of blue by averaging all the latent representations of blue?  Similarly, how sensitive are conclusions with respect to different composition functions?
Section 4 is potentially very interesting, but I don't seem to understand why it's good news that TRE correlates with I(x;\theta). Low TRE indicates high-degree of compositionality. I suspect that low MI means that input and latent representation are somewhat independent but I don't see the connection to compositional components. Can the authors clarify?
Section 5 is a nice addition. The authors mention that they learn word and phrase representations. Where are the word representations used? My understanding is that you derive basis word representations by using SGD and the phrase vectors and compute TRE with these. If this is the case, an interesting experiment would be to report how similar the induced basis vectors are (either some first-order or second-order similarity) to the pre-trained ones.
Section 8 presents results on discrete representations. Since this is the experiment most similar to the recent work that uses topographic similarity (and since the authors already prime the reader at section 7 about relation between the 2 measures), it would be interesting to see the empirical relation between TRE and topographic and its relation to generalization and absolute performance.
Baroni and Zamparelli (2010) Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space


Expectation learning is a continuous unsupervised learning process which uses multisensory bindings to modulate unisensory perception. As humans, we learn to associate a barking sound with the visual appearance of a dog, and we continuously fine-tune this association over time, as we learn, e.g., to associate highpitched barking with small dogs. In this work, we address the problem of building a computational model that captures two important properties of expectation learning, namely continuity and the lack of any external supervision other than temporal co-occurrence. To this end, we present a novel hybrid neural model based on audio/visual autoencoders and a recurrent self-organizing network for stimulus reconstruction and multisensory binding. We demonstrate that the proposed model is capable of learning concept bindings, i.e. dog barking with dogs, by evaluating it on unisensory classification tasks for audi-visual stimuli using the 43,500 Youtube videos in the animal subset of the AudioSet corpus. In addition, our analysis and discussion explain how the expectation learning mechanism enforces the generation of high-level bindings and how they contribute to audiovisual recognition.

The authors proposed an unsupervised learning framework to learn multisensory binding, using visual and auditory domain from animal videos as example. First, the visual and auditory inputs are autoencoded, and these latent codes are binding using a recurrent self-organizing network (Gamma-GWR). Furthermore, the authors proposed the expectation learning idea, which is inspire by psychology literature. In short, after the first pass of training using the real data. The authors fine tuned the model to bind the real data from one domain and the reconstructed data from another domain. This could be a good idea, as the authors pointed out, human usually bind all kinds of yellow bird to a same mental 'chirping' sounds. So, this expectation learning could potentially group the representation to a canonical one. Also, the authors showed in Table 1 that with the expectation learning, the model's recognition accuracy is improved a bit. I think it would be interesting to show the reconstruction output example (as in Fig. 3) for both model with and without expectation learning. To see if it is as the authors claim, that the model with expectation learning is reconstructing the missing modality with more canonical images/sounds. (This may not be the goal in other practice, though I'm convinced it is a potentially good psychological model as it explain well the multisensory imagery effect (Spence & Deroy, 2013).
I found this manuscript quite hard to follow though. The description seems sometime not flowing very smoothly. And there are some clear typos and mess up of math notations make the reading unpleasant. I have noted down several points below, and hope the authors could improve in the next iteration.
1. The description of variational autoencoder is not well written. The citation (Chen, 2016) is not the standard VAE paper people usually cite (unless the author is adopting something specific from the Chen's paper.). For example, the authors wrote "the KL divergence between the encoded representation and a sample from the Gaussian distribution" which sounds incorrect to me.
2. Why a Variational autoencoder is necessary for visual domain, but a regular autoencoder is used in auditory domain?
Typos:
1. page 2, 2nd line: a online --> an online
2. Use subscript I-1 to mean the winner neuron at t-1, I think this is not quite clear. I suggest to follow the notation in (Parisi & Wermter 2017), use I(t-1), which is easier to follow.
3. page 7, 2nd line: more than 17% for audio.  -> for vision.
4. page 8, 3rd line: not on the original high-abstraction data. Do the authors mean highly specific data? That seems make more sense.
5. Several notation mismatch here and there. for example, in formula 6 it is w_j^s, but in the text below it become w_{j,s}.


POSTERIOR ATTENTION MODELS FOR SEQUENCE TO SEQUENCE LEARNING
Modern neural architectures critically rely on attention for mapping structured inputs to sequences. In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output tokens across a predicted sequence. We present an alternative architecture called Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables, proposes two major changes. First, the position where attention is marginalized is changed from the input to the output. Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output. Empirically on five translation and two morphological inflection tasks the proposed posterior attention models yield better BLEU score and alignment accuracy than existing attention models.

This paper proposes a new sequence to sequence model where attention is treated as a latent variable, and derive novel inference procedures for this model. The approach obtains significant improvements in machine translation and morphological inflection generation tasks. An approximation is also used to make hard attention more efficient by reducing the number of softmaxes that have to be computed.
Strengths:
- Novel, principled sequence to sequence model.
- Strong experimental results in machine translation and morphological inflection.
Weaknesses:
- Connections can be made with previous closely related architectures.
- Further ablation experiments could be included.
The derivation of the model would be more clear if it is first derived without attention feeding: The assumption that output is dependent only on the current attention variable is then valid. The Markov assumption on the attention variable should also be stated as an assumption, rather than an approximation: Given that assumption, as far as I can tell the (posterior) inference procedure that is derived is exact: It is indeed equivalent to the using the forward computation of the classic forward-backward algorithm for HMMs to do inference.
The model’s overall distribution can then be defined in a somewhat different way than the authors’ presentation, which I think makes more clear what the model is doing:
p(y | x) = \sum_a \prod_{t=1}^n p(y_t | y_{<t}, x, a_t) p(a_t | y_{<t}, x_ a_{t-1}).
The equations derived in the paper for computing the prior and posterior attention is then just a dynamic program for computing this distribution, and is equivalent to using the forward algorithm, which in this context is:
 \alpha_t(a) = p(a_t = a, y_{<=t}) = p(y_t | s_t, a_t =a) \sum_{a’} \alpha_{t-1}(a’) p(a_t = a | s_t, a_{t-1} = a’)
The only substantial difference in the inference procedure is then that the posterior attention probability is fed into the decoder RNN, which means that the independence assumptions are not strictly valid any more, even though the structural assumptions are still encoded through the way inference is done.
[1] recently proposed a model with a similar factorization, although that model did not feed the attention distribution, and performed EM-like inference with the forward-backward algorithm, while this model is effectively computing forward probabilities and performing inference through automatic differentiation.
The Prior-Joint variant, though its definition is not as clear as it should be, seems to be assuming that the attention distribution at each time step is independent of the previous attention (similar to the way standard soft attention is computed) - the equations then reduce to a (neural) version of IBM alignment model 1, similar to another recently proposed model [2]. These papers can be seen as concurrent work, and this paper provides important insights, but it would strengthen rather than weaken the paper to make these connections clear.
The results clearly show the advantages of the proposed approach over soft and sparse attention baselines. However, the difference in BLEU score between the variants of the prior or posterior attention models is very small across all translation datasets, so to make claims about which of the variants are better, at a minimum statistical significance testing should be done. Given that the “Prior-Joint” model performs competitively, is it computationally more efficient that the full model?
The main missing experiment is not doing attention feeding at all. The other experiment that is not included (as I understood it) is to compute prior and posterior attention, but feed the prior attention rather than the posterior attention.
The paper is mostly written very clearly, there are just a few typos and grammatical errors in sections 4.2 and 4.3.
Overall, I really like this paper and would like to see it accepted, although I hope that a revised version would make the assumptions the model is making clearer and make connections to related models clearer.

[1] Neural Hidden Markov Model for Machine Translation, Wang et al, ACL 2018.
[2] Hard Non-Monotonic Attention for Character-Level Transduction, Wu, Shapiro and Cotterell, EMNLP 2018.


L 2 -NONEXPANSIVE NEURAL NETWORKS
This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers. We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness, with a new regularization scheme for linear layers, new ways to adapt nonlinearities and a new loss function. With MNIST and CIFAR-10 classifiers, we demonstrate a number of advantages. Without needing any adversarial training, the proposed classifiers exceed the state of the art in robustness against white-box L 2 -bounded adversarial attacks. They generalize better than ordinary networks from noisy data with partially random labels. Their outputs are quantitatively meaningful and indicate levels of confidence and generalization, among other desirable properties.

Summary:
The paper presents techniques for training a non expansive network, which keeps the Lipchitz constant of all layers lower than 1. While being non-expansive, means are taken to preserve distance information better than standard networks. The architectural changes required w.r.t standard networks are minor, and the most interesting changes are made to the loss minimized. The main claim of the paper is that the method is robust against adversarial attacks of a certain kind. However, the results presented show that a) such robustness comes at a high cost of accuracy for standard examples, and b) even though the network is preferable to a previous alternative in combating adversarial examples, the accuracy obtained in the face of adversarial attacks is too low to be of practical value. Other properties of the networks, explored empirically, are that the confidence of the prediction is indicative of robustness (to adversarial attacks) and that the networks learn better in the presence of high label noise.
In short, this paper may be of interest to a sub-community interested in defense against certain types of adversarial attacks, even when the defense level is much too low to be practical. I am not part of this community, hence did not find this part very interesting. I believe the regularization results are of wider interest. However, to present this as the main contribution of L2NNN more work is required to find configuration which are resilient to overfit yet enable high training accuracy, and more diverse experiments are required.
Pros:
+ the idea of non expansive network is interesting and important
+ results indicate some advantages in fighting adversarial examples and label noise
Cons:
- the results for fighting adversarial examples are not significant from a practical perspective
- the results for copying with label noise are preliminary and require expansion with more experiments.
- the method has costs in accuracy, which is lower than standard networks and this issue is not faced with enough attention
- presentation clarity is medium: proofs for claims are missing, as well as relevant background on the relevant adversarial attacks. The choice to place the related work at the end also reduces presentation clarity.
More detailed comments:
Pages 1-3: In many places, small proofs are left to the reader as ‘straightforward’. Examples are: the claim in the introduction, in eq. 2, in section 2.2, section 2.3’ last line of page 3, etc.. While the claim are true (in the cases I tried to verify them long enough), this makes reading difficult and not fluent. For some of these claims I do not see the argument behind them. In general, I think proofs should be brought for claims, and short proofs (preferably) should be brought for small claims. Leaving every proof to the reader as an exercise is not a convenient strategy.
Page 4: The loss is complex and its terms utility require empirical evidence. The third term is shown to be clearly useful, enabling a trade off between train accuracy and margin. However, the utility of terms 4) and 5) is not verified. Do we really need both these terms? Cannot we just stay with one?
The main claim is robustness w.r.t “white-box non targeted L2-bounded attacks”. This seems to be a very specific attack type, and it is not explained at all in the text. Hence it is hard to judge the value of this robustness. Explanation of adversarial attack kinds, and specifically of “white-box non targeted
L2-bounded attacks” is required for this paper to be a stand alone readable paper. Similarly ‘L_\infty’-bounded attacks, for which results are shown, should be explained.
Table 1,2: First, the model architecture used in these experiments is not stated. Second, the accuracy of the ‘natural’ baseline classifier, at least in the MNist case, is somewhat low – much better results can be obtained with CNN on MNist. Third, the accuracies of the suggested robust models are very low compared to what can be obtained on these datatsets. Forth, while the accuracies under attack of the proposed method are better than those of Madri et al., both are quite poor and indicate that the classifier is not useful under attack (from a practical perspective).
Page 6: The classifiers which share the work between an L2NNN network and a regular more accurate network may be interesting, as the accuracies reported for them are significantly higher than the L2NNN networks. However, the robustness scores are not reported for these classifiers, so it is not possible to judge if they lead to a practical and effective strategy.
Page 7: For me, the results with partially random labels are the most interesting in the paper. The resistance of L2NNN to overfit and its ability to learn with very noisy data are considerably better than the suggested alternatives.
Relevant work not mentioned “Spectral Norm Regularization for Improving the Generalizability of Deep Learning” - Yuichi Yoshida and Takeru Miyato, Arxiv, 2017.
I have read the rebuttal.
The discussion was interesting, but I do not see a need to change my assessment.
The example of ad-blocking in indeed a case (the first I encounter) where l2- perturbated adversarial examples can be useful for cyber attack. The other ones are less relevant (the attacks are not based on adversarial attacks in the sense used in the paper: images crated with small gradient-direction perturbations). Anyway talking about 'attacks on a self-driving car' are still not neaningful to me: I do not understand what adversarial examples have to do with this.
I do not find the analogy of 'rocket improvements and moon landing' convincing: in 69 rocket improvements were of high interest in multiple applications, and moon landing was visible over the corner.


The most used recommendation method is collaborative filtering, and the key part of collaborative filtering is to compute the similarity. The similarity based on co-occurrence of similar event is easy to implement and can be applied to almost all the situation. So when the word2vec model reach the state-of-art at a lower computation cost in NLP. An correspond model in recommender system item2vec is proposed and reach state-of-art in recommender system. It is easy to see that the position of user and item is interchangeable when their count size gap is not too much, we proposed a user2vec model and show its performance. The similarity based on cooccurrence information suffers from cold start, we proposed a content based similarity model based on doc2vec which is another technology in NLP.

The idea of learning user embeddings for downstream tasks in recommender systems is a good one.
However, this paper proposes no significant methodological developments (e.g., user2vec is an extension of item2vec obtained by transposing the observation matrix). Further, it does not present a thorough study with interesting empirical results (doc2vec does not improve performance, a single dataset is used, baselines are not state of the art).
Overall, this short paper (3 pages + refs) seems a bit preliminary and, in its current state, does not make a significant enough contribution to be accepted at this venue.
I would suggest that a more thorough analysis of similarity methods for NN models could be interesting to a recsys workshop or perhaps a conference focussed on recsys (e.g., ACM recsys).


ADVERSARIAL AUDIO SYNTHESIS
Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that-without labels-WaveGAN learns to produce intelligible words when trained on a smallvocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

*Pros:*
-	Easily accessible paper with good illustrations and a mostly fair presentation of the results (see suggestions below).
-	It is a first attempt to generate audio with GANs which results in an efficient scheme for generating short, fixed-length audio segments of reasonable (but not high) quality.
-	Human evaluations (using crowdsourcing) provides empirical evidence that the approach has merit.
-	The paper appears reproducible and comes with data and code.
*Cons*:
-	Potentially a missing comparison with existing generative methods (e.g. WaveNet). See comments/questions below **
-	The underlying idea is relatively straightforward in that the proposed methods is a non-trivial application of already known techniques from ML and audio signal processing.
*Significance*: The proposed GAN-based audio generator is an interesting step in the development of more efficient audio generation and it is of interest to a subcommunity of ICLR as it provides a number of concrete techniques for applying GANs to audio.
*Further comments/ questions:*
-	Abstract/introduction: I’d suggest being more explicit about the limitations of the method, i.e. you are currently able to generate short and fixed-length audio.
-	SpecGAN (p 4): I’d suggest including some justification of the chosen pre-processing of spectrograms (p. 4, last paragraph).
-	** Evaluation:  The paper dismisses existing generative methods early in the evaluation phase but the justification for doing so is not entirely clear to me: Firstly, if the inception score is used as an objective criterion it would seem reasonable to include the values in the paper. Secondly, as inception scores are based on spectrograms it could potentially favour methods using spectrograms directly (SpecGAN) or indirectly (WaveGAN, via early stopping) thus putting the purely sample based methods (e.g. WaveNet) at a disadvantage. It would seem fair to pre-screen the audio before dismissing competitors instead of solely relying on potentially biased inception scores (which was probably also done in this work, but not clearly stated…)? Finally, while not the aim of the paper, it would have been beneficial to discuss and understand the failures of existing methods in more detail to convince the reader that a fair attempt has been made to getting competitors to work before leaving them out entirely.
-	Results/analysis: It is unclear to me how many people annotated the individual samples? What is the standard deviation over the human responses (perhaps include in tab 1)? Consider including a reflection on (or perhaps even test statistically) the alignment between the qualitative diversity/quality scores and the subjective ratings to justify the use of the objective scores in the training/selection process.
-	Related work: I think it would provide a better narrative if the existing techniques are outlined earlier on in the paper.