Adversarial training has been successfully applied to build robust models at a certain cost. While the robustness of a model increases, the standard classification accuracy declines. This phenomenon is suggested to be an inherent trade-off. We propose a model that employs feature prioritization by a nonlinear attention module and L 2 feature regularization to improve the adversarial robustness and the standard accuracy relative to adversarial training. The attention module encourages the model to rely heavily on robust features by assigning larger weights to them while suppressing non-robust features. The regularizer encourages the model to extracts similar features for the natural and adversarial images, effectively ignoring the added perturbation. In addition to evaluating the robustness of our model, we provide justification for the attention module and propose a novel experimental strategy that quantitatively demonstrates that our model is almost ideally aligned with salient data characteristics. Additional experimental results illustrate the power of our model relative to the state of the art methods.

Summary: This paper argues that improved resistance to adversarial
attacks can be achieved by an implicit denoising method in which model
weights learned during adversarial training are encouraged to stay
close to a set of reference weights using the ell_2
penalty. Additionally, the authors claim that by introducing an
attention model which focuses the model training on more robust
features they can further improve performance. Some experiments are
provided.
Feedback: My main concerns with the paper are:
* The experimental section is fairly thin. There are at this point a
  large number of defense methods, of which Madry et al. is only one. In
  light of these, the experimental section should be expanded. The
  results should ideally be reported with error bars, which would help
  in gauging significance of the results.
* The differential impact of the two contributions is not entirely
  clear. The results in Table 1 suggest that implicit denoising can
  help, yet at the same time, Table 2 suggests that Black-box
  performance is better if we just use the attention model. Overall,
  this conflates the contributions unnecessarily and makes it hard to
  distingish their individual impact.
* The section on gradient maps is not clear. The authors argue that if
  the gradient map aligns with the image the model depends solely on
  the robust features. While this may be (somewhat more) intuitive in
  the context of simple GLMs, it's not clear why it should carry over
  to DNNs. I think it would help to make these intuitions much more
  precise. Secondly, even if this were the case, the methodology of
  using a neural net to classify gradient maps and from this derive a
  robustness metric raises precisely the kinds of robustness questions
  that the paper tries to answer. I.e.: how robust is the neural net
  classifying the gradient images, and how meaningful are it's
  predictions when gradient maps deviate from "clean" images.
Overall, I feel this paper has some potentially interesting ideas, but
needs additional work before it is ready for publication.


We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration. Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other. The former collects training samples for the latter, and its objective is to maximize the error of the latter. The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former. In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples. We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively. We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models. Experimental results show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors.

The paper proposes a novel exploration strategy for self-supervised imitation learning. An inverse dynamics model is trained on the trajectories collected from a RL-trained policy. The policy is rewarded for generating trajectories on which the inverse dynamics model (IDM) currently works poorly, i.e. on which IDM predicts actions that are far (in terms of mean square error) from the actions performed by the policy. This adversarial training is performed in purely self-supervised way. The evaluation is performed by one-shot imitation of an expert trajectory using the IDM: the action is predicted from the current state of the environment and the next state in the expert’s trajectory. Experimental evaluation shows that the proposed method is superior to baseline exploration strategies for self-supervised imitation learning, including random and curiosity-based exploration.
Overall, I find the idea quite appealing. I am not an expert in the domain and can not make comments on the novelty of the approach. I found the writing mostly clear, except for the following issues:
- the introduction has not made it crystal clear that the considered paradigm is different from e.g. DAGGER and GAIL in that expert demonstrations are used at the inference time. A much wider audience is familiar with the former methods, and this distinction should have be explained more clearly.
- Section 4.2.: “As opposed to discrete control domains, these tasks are especially challenging, as the sample complexity grows in continuous control domains.” - this sentence did not make sense to me. It basically says continuous control is challenging because it is challenging.
- I did not understand the stabilization approach. How exactly Equation (7) forces the policy to produce “not too hard” training examples for IDM? Figure 4 shows that it is on the opposite examples with small L_I that are avoided by using \delta > 0.
- Table 1 - it is a bit counterintuitive that negative numbers are better than positive numbers here. Perhaps instead of policy’s deterioration you could report the relative change, negative when the performance goes down and positive otherwise?
I do have concerns regarding the experimental evaluation:
- the “Demos” baseline approach should be explained in the main text! In Appendix S.7 I see that 1000 human demonstrations were used for training. Why 1000, and not 100 and not 10000?  How would the results change? This needs to be discussed. Without discussing this it is really unclear how the proposed method can outperform “Demos”, which it does pretty often.
- it is commendable that 20 repetitions of each experiment were performed, but I am not sure if it is ever explained in the paper what exactly the upper and lower boundaries in the figures mean. Is it the standard deviation? A confidence interval? Can you comment on the variance of the proposed approach, which seems to be very high, especially when I am looking at high-dimensional fetch-reach results?
- the results of “HandReach” experiments, where the proposed method works much worse than “Demos” are not discussed in the text at all
- overall, there is no example of the proposed method making a difference between a “working” and “non-working” system, compared to “Curiosity” and “Random”. I am wondering if improvements from 40% to 60% in such cases are really important. In 7 out of 9 plots the performance of the proposed method is less than 80% - not very impressive. "Demos" baseline doesn't perform much better, but what would happen with 10000 demonstrations?
- there is no comparison to behavioral cloning, GAIL, IRL. Would these methods perform better than learning IDM like "Demos" does?
I think that currently the paper is slightly below the threshold, due to evaluation issues discussed above and overall low performance of the proposed algorithm. I am willing to reconsider my decision if these issues are addressed.


How optimization influences the generalization ability of a DNN is still an active area of research. This work aims to unveil and study a factor of influence: the speed at which each layer trains. In our preliminary work, we develop a visualization technique and an optimization algorithm to monitor and control the layer rotation rate, a tentative measure of layer-level training speed, and show that it has a remarkably consistent and substantial impact on generalization. Our experiments further suggest that weight decay's and adaptive gradients methods' impact on both generalization performance and speed of convergence are solely due to layer rotation rate changes compared to vanilla SGD, offering a novel interpretation of these widely used techniques, and providing supplementary evidence that layer-level training speed indeed impacts generalization. Besides these fundamental findings, we also expect that on a practical level, the tools we introduce will reduce the meta-parameter tuning required to get the best generalization out of a deep network.

Paper summary: The authors propose layer rotation speed as a measure of layer-wise training speed and introduce the Layca optimizer as a means of enforcing uniform layer rotation speed throughout the network. They show empirically that layer rotation speed is linked to the generalization performance of deep neural networks and that weight decay induces uniform layer rotation speeds.
Detailed comments:
Overall, I felt that the paper introduced some interesting ideas but I was not left convinced that layer rotation speed is the correct measure of layer training speed. I hope that the authors can clarify this based on my questions and comments below.
1) In the introduction you refer to input and feedback signals to a layer, I assume this refers to the forward and backward pass. As I understand it, this intuition and the findings of Figure 1 do not immediately relate to the notion of layer rotation speed during training. Could you clarify what you mean by "the input and feedback signals that a layer receives could also influence the generalization ability induced by the layer's training", to me this statement seems obvious as input+feedback signals contains training entirely.
2) In Figure 1 you show that when training one chosen layer and keeping the others fixed, if the chosen layer is deeper into the network the test accuracy is worse. I wonder to what extent this might be remedied by initialization. For example, one might expect that when sampling random square matrices there are some very small eigenvalues which "kill" information in the forward pass. If we train a layer deep in the network it may have access to less information from the data than one earlier on. Whereas training an earlier layer could allow this layer to shift mass into the parts of the eigenspace which are well represented (so-to-speak) in the future layers. Have you thought about this at all? One simple way to evaluate this would be to initialize the weights to be random orthogonal matrices, ensuring that the eigenvalues are equal. With that said, I thought that this was an interesting experiment with a fairly surprising outcome!
3) In related work you discuss the vanishing and exploding gradients problems in terms of layer-level training speed. I think that another relevant research direction may be dynamical isometry [1] which solves this problem by restricting all singular values of the Jacobian matrix to be close to 1. These ideas may also be relevant when discussing Layca and layer-rotation.
4) I found section 3.1. a little unconvincing. It is not obvious to me that layer rotation speed is necessarily a good measure of training speed. In fact, there are many updates which have large cosine distance (as you define it) but do not change the network function (for example, permuting the weight matrices in fully-connected networks). Why is the rotation defined through a vectorization of the weight matrix as opposed to e.g. the polar decomposition? Is this a computational issue? Similarly, in section 3.2 you liken Layca to optimization on a manifold but I am not convinced that this makes sense for matrices which inherently have some structure (e.g, perhaps the Stiefel manifold would be more meaningful).
6) Figure 2 shows that uniform rotation leads to improved test accuracy. But could it be the case that controlling the effective learning rate is sufficient (and layer rotation is one way to achieve this)? For example, we might take the sign of the update and use this to ensure that each weight matrix has the same effective learning rate (something like [2]). Do you expect this would have a similar effect? If not, what is unique about layer rotation that provides good test accuracy?
7) You claim that SGD and adaptive methods with weight decay works without taking extra care to control the layer-rotation rate, as weight decay provides a similar effect. Firstly, you use weight decay and L2 regularization interchangeable, could you be explicit about exactly which you mean (see e.g. [3]). Assuming you mean weight decay (and not L2 regularization), then this could also be due to the effective learning rate ([4,5,6]) which may have some interaction with layer rotation rate (i.e. Figure 4). In summary, I would have liked to see an explanation for why weight decay leads to uniform rotation speeds.
8) If I understand correctly, Figure 5 shows 5 tasks and reportedly 5 optimization schemes - each on a different task? It seems more reasonable to compare these on the same task.
Overall I felt that the paper had some interesting contributions and a fairly comprehensive empirical study. However, I do not feel that the paper gives adequate attention to the notion of effective learning rate induced by weight decay and I was not totally convinced that the way layer rotation speed is defined is the correct way.
Minor comments:
- A lot of white space and a large caption for Figure 1.
- Section 4 opens with "monitor and control", but I think the latter is really presented in section 4 and not section 3.
- I think a diagram of the projection step of the Layca algorithm would be informative (for 2D weight vector).
- Why does `5` appear in equation 1? Is this an arbitrary choice?
- Some of the lighter colors in e.g. Figure 2(b) made some lines hard to read when printed. I do not believe that this affected the image significantly.
Clarity: The paper is well written and is easy to understand. Some of the figures in the experiments are a little cluttered and the lighter colors can be hard to see (e.g. Fig 2(b)), but this is minor.
Significance: The paper presents an interesting view point but I am not convinced that it offers as strong an explanation for these phenomena as other approaches. I believe with some more clarification the results could become more significant. My review score hinges mostly on the interaction between layer rotation speed and the effective layer-wise learning rate.
Originality: To my knowledge, the ideas are presented in the paper are original. In particular, this is a novel way to characterize layer-wise training speed.
References:
[1] Pennington et al. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice" https://arxiv.org/abs/1711.04735
[2] Bernstein et al. "signSGD: Compressed Optimisation for Non-Convex Problems" https://arxiv.org/abs/1802.04434
[3] Loshchilov et al. "Fixing Weight Decay Regularization in Adam", https://arxiv.org/pdf/1711.05101.pdf
[4] Laarhoven, "L2 Regularization versus Batch and Weight Normalization" https://arxiv.org/abs/1706.05350
[5] Hoffer et al. "Norm matters: efficient and accurate normalization schemes in deep networks" https://arxiv.org/abs/1803.01814
[6] Anonymous, "Three Mechanisms of Weight Decay Regularization" https://openreview.net/forum?id=B1lz-3Rct7   (Another ICLR 2019 submission)


We augment adversarial training (AT) with worst case adversarial training (WCAT) which improves adversarial robustness by 11% over the current stateof-the-art result in the 2 norm on CIFAR-10. We obtain verifiable average case and worst case robustness guarantees, based on the expected and maximum values of the norm of the gradient of the loss. We interpret adversarial training as Total Variation Regularization, which is a fundamental tool in mathematical image processing, and WCAT as Lipschitz regularization.

This paper explores augmenting the training loss with an additional gradient regularization term to improve the robustness of models against adversarial examples. The authors show that this training loss can be interpreted as a form of adversarial training against optimal L2 and L_infinity adversarial perturbations. This augmented training effectively reduces the Lipschitz constant of the network, leading to improved robustness against a wide variety of attack algorithms.
While I believe the results are correct and possibly significant, the paper is poorly written (especially for a 10 page submission) and comparison with prior work on reducing the Lipschitz constant of the network is lacking. The authors also made little to no effort in writing to ensure the clarity of their paper. I would like to see a completely reworked draft before opening to the idea of recommending acceptance.
Pros:
- Theoretically intuitive method for improving the model's robustness.
- Evaluation against a wide variety of attacks.
- Empirically demonstrated improvement over traditional adversarial training.
Cons:
- Lack of comparison to prior work. The authors are aware of numerous techniques for controlling the Lipschitz constant of the network for improved robustness, but did not compare to them at all.
- Poorly written. The paper contains multiple missing figure references, has a duplicated table (Tables 1 and 3), and the method is not explained well. I am confused as to how the 2-Lip loss is minimized. Also, the paper organization seems very chaotic and incoherent, e.g., the introduction section contains many technical details that would better belong in related works or methods sections.
--------------------------------------------
Revision:
I thank the authors for incorporating my suggestions and reworking the draft, and I have updated my rating in response to the revision. While I believe the organization is much cleaner and easier to follow, there is still much room for improvement. In particular, the paper does not introduce concepts in a logical order for a non-expert to follow (e.g. Reviewer 1) and leaps into the paper's core idea too quickly. I am strongly in favor of exceeding the suggested page limit of 8 pages and using that space to address these concerns.
A more pressing concern is the evaluation of prior work. The authors added a short section (Section 5.4) comparing their method to that of (Qian and Wegman, 2018). This is certainly a reasonable comparison and the results seem promising, the evaluation lacks an important dimension -- varying the value of epsilon and observing the change in robustness. This is an important aspect for defenses against adversarial examples as certain defense may be less robust but are insensitive to the adversary's strength. Showing the robustness across different adversary strengths gives a more informative view of the authors' proposed method in comparison to others. The evaluation is also lacking in breadth, ignoring other similar defenses such as (Cisse et al., 2017) and (Gouk et al., 2018).


POLICY TRANSFER WITH STRATEGY OPTIMIZATION
Computer simulation provides an automatic and safe way for training robotic control policies to achieve complex tasks such as locomotion. However, a policy trained in simulation usually does not transfer directly to the real hardware due to the differences between the two environments. Transfer learning using domain randomization is a promising approach, but it usually assumes that the target environment is close to the distribution of the training environments, thus relying heavily on accurate system identification. In this paper, we present a different approach that leverages domain randomization for transferring control policies to unknown environments. The key idea that, instead of learning a single policy in the simulation, we simultaneously learn a family of policies that exhibit different behaviors. When tested in the target environment, we directly search for the best policy in the family based on the task performance, without the need to identify the dynamic parameters. We evaluate our method on five simulated robotic control problems with different discrepancies in the training and testing environment and demonstrate that our method can overcome larger modeling errors compared to training a robust policy or an adaptive policy.

The authors propose a policy transfer scheme which in the source domain simultaneously learns a family of policies parameterised by dynamics parameters and then employs an optimisation framework to select appropriate dynamics parameters based on samples from the target domain. The approach is evaluated on a number of simulated transfer tasks (either transferring from DART to MuJoCo or by introducing deliberate model inaccuracies).
This is interesting work in the context of system identification for policy transfer with an elaborate experimental evaluation. The policy learning part seems largely similar to that employed by Yu et al. 2017 (as acknowledged by the authors). This makes the principal contribution, in the eyes of this reviewer, the optimisation step conducted based on rollouts in the target domain. While the notion of optimising over the space of dynamics parameters is intuitive the question arises whether this optimisation step makes for a substantive contribution over the original work. This point is not really addressed in the experimental evaluation as benchmarking is performed against a robust and an adaptive policy but not explicitly against the (arguably) most closely related work in Yu et al. It could be argued, of course, that Yu et al. essentially use adaptive policy generation but they do explicitly learn dynamics parameters based on recent history of actions and observations. An explicit comparison therefore seems appropriate (or alternatively a discussion of why it is not required).
Another point which would, in my view, add significant value is explicit discussion of the baseline performances observed in the various experiments. For example, in the hopper experiment (Sec 5.2) the authors state that the baseline methods were not able to adapt to the new environment. Real value could be derived here if the authors could elaborate on why this is the case. The same applies in Sec 5.3-5.6.
(I would add here, as an aside, that I thought the notion in Sec 5.6 of framing the learning of policies for handling deformable objects as a transfer task based on rigid objects to be a nice idea. And not one this reviewer has come across before - though this could merely be a reflection of limited familiarity with the literature).
The experimental evaluation seems thorough with the above caveat of a seemingly missing benchmark in Yu et al. I would also encourage the authors to add more detail in the experimental section in the main text specifically with regards to number of trials run to arrive at variances in the figures as well as what metric these shaded areas actually signify.
A minor point: the J in equ 1 seems (to me at least) undefined. I suspect that it signifies the expected cumulative reward and was meant to be introduced in Sec 3 where the J may have been dropped from the latex?
If the above points were addressed I think this would make a valuable and interesting contribution to the ICLR community. As it stands I believe it is marginally below the acceptance threshold.
[ADDENDUM: given the author feedback and addition of the benchmark experiments requested I have updated my score.]
Pros:
———
- interesting work
- accessible
- effective
- thorough evaluation (though potentially missing a key benchmark)
Cons:
———
- potentially missing a key benchmark (and therefore seems somewhat incremental)
- only limited insight offered by the authors in the discussion of the experimental results
- some more details needed with regards to the experimental setup


Work on the problem of contextualized word representation-the development of reusable neural network components for sentence understanding-has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo . This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives. The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models. However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.

The work presented in this paper relates to the impact of the dataset on the performance of contextual embedding (namely ELMO in this paper) on many downstream tasks, including GLUE tasks, but also alternative NLP tasks.
The work is focused on experiments, and draws several conclusions that are interesting, mostly around the amount of gain one can expect and the fact that the choice of the dataset is task-dependent.
One of the issue is that the authors if seems to believe that ELMO is the best contextual language model. The field is moving so quickly that the experiments might become invalid pretty soon (e.g. see BERT model referenced below).
Finally, the analysis is mostly descriptive and there is few insight by the author about what should be the future work, apart from "we need a better understanding".
Minor details:
Page 1: "can yield very strong performance on NLP tasks" is a very busy way to express the fact that Sentence Encoders work well in practice.
The field evolves quickly and ELMO has now a competitive models called BERT (arXiv.org > cs > arXiv:1810.04805). I understand that the results of the current papers would hove to be re-run on all these tasks, but I'm afraid the current paper will have a limited impact if it does not use the most effective method at the date of publication...


DYNAMICALLY UNFOLDING RECURRENT RESTORER: A MOVING ENDPOINT CONTROL METHOD FOR IMAGE RESTORATION
In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels using a single model. The proposed control problem contains an image restoration dynamic which is modeled by a convolutional RNN. The moving endpoint, which is essentially the terminal time of the associated dynamic, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage.

Summary:
The authors proposes a new image restoration method that becomes particularly useful for blind restoration setting, e.g., the unknown noise variance setting for denoising. They utilized the moving endpoint control methodology, which essentially is applying reinforcement learning to the image restoration, and devised a method that adaptively decides the unfolding steps for given noisy image. The experimental results show encouraging results.
Pros:
In the experimental result, the proposed DURR outperforms DnCNN-B, a current state-of-the-art. Particularly, while DnCNN-B suffers for the noise level that it was not trained for, DURR can still denoise much better. (Table 2) A similar result is obtained for the JPEG deblocking problem as well.
Cons:
- Since the Deep Q-learning is used to train the policy unit, I suspect the training time could be quite long. How does the reward curve look like while training? How stable is the training? Showing such details should make the paper stronger.
- It will be interesting to see more details on the model. For example, what is the mean/std for the number of folds that model applies for BSD68? What is the distribution (histogram?) of the folds for BSD68? Currently, the paper just simply shows the results and seems to hide many details.
- What was the choice for \lambda in Eq. (3),(4)? How do you choose it?
- How does the result look like on other benchmark datasets other than BSD68? It seems like the specific number of looks for each noise level is important for training. Do the choices of (25,4),(35,6),(45,9),(55,12) generalize well to other datasets as well?
Overall, I think the paper should add more details mentioned above to make the paper stronger.


Ensemble learning is a method of combining multiple trained models to improve model accuracy. We propose the usage of such methods, specifically ensemble average, inside Convolutional Neural Network (CNN) architectures by replacing the single convolutional layers with Inner Average Ensembles (IEA) of multiple convolutional layers. Empirical results on different benchmarking datasets show that CNN models using IEA outperform those with regular convolutional layers and advances the state of art. A visual and a similarity score analysis of the features generated from IEA explains why it boosts the model performance.

IEA proposes to use multiple "parallel" convolution groups, which are then averaged to improve performance.
This fundamental idea of ensembles combined with simple functions has been explored in detail in Maxout (Goodfellow et.  al., https://arxiv.org/abs/1302.4389) in the context of learning activation functions, and greater integration with dropout regularization.
Under the lens of comparison to Maxout (which should be cited, and is a key comparison point for this work), a number of questions emerge. Does IEA also work for feedforward layers? Does IEA give any performance improvement or have some fundamental synergy with the regularizers used here? Is the performance boost greater than simply using an ensemble of m networks directly (resulting in the equivalent number of parameters overall)? The choice of the mean here seems insufficient for creating the types of complexity in activation which are normally desirable for neural networks, so some description of why a simple mean is a good choice would be beneficial since many, many other functions are possible.
Crucially Maxout seems much too close to this work, and I would like to see an indepth comparison (since it appears to be use of mean() instead of max() is the primary difference). I would also significantly reduce the claims of novelty, such as "We introduce the usage of such methods, specifically ensemble average inside Convolutional Neural Networks (CNNs) architectures." in the abstract, given that this is the exact idea explored in other work including followups to Maxout.
For example, MNIST performance here matches Maxout (.45% for both, but Maxout uses techniques known in 2013). CIFAR-10 results are better, but again Maxout first appeared 5 years ago. There are more recent followups that continued on the line of work first shown in Maxout, and there should be some greater comparison and literature review on these papers. The CIFAR-10 baseline numbers are not ideal, and since IEA is basically "plug and play" in existing architectures, starting from one of these settings instead (such as Wide ResNet https://arxiv.org/abs/1605.07146) and showing a boost would be a stronger indication that this method actually improves results. In addition, there are a number of non-image settings where CNNs are used (text or audio), and showing this idea works on multiple domains would also be good.
There seems to be a similarity between ResNet and this method - specifically assuming the residual pathway is convolution with an identity activation, the summation that combines the two pathways bears a similarity to IEA. With multiple combined paths (as in Dense ResNet) this equivalence seems stronger still. A discussion of this comparison in greater detail, or even derivation of IEA as a special setting or extension of ResNet (coupled with stronger performance on the datasets) would help ground the work in prior publication.
The section on visualization and inspection of IEA features seems interesting, but too brief. A greater exploration of this, and possible reduction or removal of the ensemble selection section (which didn't have a clear contribution to the message of the paper, in my opinion) would strengthen the work - and again, comparisons to activations learned by Maxout and followups would make this inspection much stronger.
My key concerns here are on relation to past work, greater comparison to closely related methods, and improvement of baselines results. Given the close similarity of this work to Maxout and others, a much stronger indication of the benefits and improvements of IEA seems necessary to prove out the concepts here.


Deep neural networks can learn meaningful representations of data. However, these representations are hard to interpret. For example, visualizing a latent layer is generally only possible for at most three dimensions. Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer. Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions. To do so, we propose a class of regularizations we call Graph Spectral Regularizations that impose graphstructure on latent layers. This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters. This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data. First, we show a synthetic example that the graph-structured layer can reveal topological features of the data. Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets. Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation. In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations. Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.

The paper introduces a spectral regularization with the aim of obtaining representations
that are easier to interpret.
Some sentences are often confusing and, in general, clarity needs to be improved.
The motivation of the work is not very strong in my opinion, in particular by adding such
a prior the space of possible solutions greatly shrinks and I am afraid
that interesting solutions will be lost. I think one should focus on properties
rather than visual inspection.
Also, isn't it that if we can clearly see the pattern, perhaps that pattern is
linear and of easy discovery also by simpler models?
More importantly, it seems that all experiments are performed on tasks where the
underlying structure is known, however this is almost never the case in practice.
Assuming one uses the proposed spectral regularization, how would one interpret
it in such cases?
In section 2 please clarify the paragraph on bounded Lp norm.
I am sorry but why isn't there a relation, for convolutional nets,
between neurons in different channels? Each element in the feature map represents
the input surrounding that location in a k dimensional space.
The authors state that the usual bottleneck for autoencoders is composed of 2/3
neurons, this is simply not true. There has been extensive work on
overcomplete representations that shows that is better to have many more dimensions
but only few degrees of freedom.
The spectral bottleneck should cite VQVAE as the approach is very similar and the
authors should compare to it.
For the topological inference experiment it is assumed that one knows the structure,
but how to address the more general problem?
More practically, the regularization enforces smoothing (if few eigenfunctions
are used, which is never explained in the paper) between connected nodes, did
the authors try to have a simple L2 penalty instead? E.g. minimize the difference
between activations in the group.
Regarding the capsule network example, when you write that without regularization
each digit responds differently to perturbation of the same dimension, isn't it
possibly true only up to a, unknown, permutation of the neurons?
To summarize, while the idea sounds interesting, I miss to find the easy interpretability
of results and also the overall motivation sounds a bit weak.
More importantly the selection of W, crucial for defining structure, is not discussed at all in the paper.
Experiments are performed on toy examples only whereas here, given that we can
possibly interpret the results I would have liked something more involved to
better show that this kind of interpretability is needed.
Missing cites:
[1] van den Oord et al, Neural Discrete Representation Learning.
[2] Koutnik et al, Evolving neural networks in compressed weight space.


ORDERED NEURONS: INTEGRATING TREE STRUCTURES INTO RECURRENT NEURAL NETWORKS
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference 1 .

Language is hierarchically structured: smaller units (e.g., noun phrases) are nested within larger units (e.g., clauses). This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the different units of an LSTM can learn to track information at different time scales, the standard architecture does not impose this sort of strict hierarchy. This paper proposes to add this constraint to the system by ordering the units; a vector of "master" input and forget gates ensures that when a given unit is reset all of the units that follow it in the ordering are also reset.
Strengths:
* The paper introduces an elegant way of adding a hierarchical inductive bias; the intuition behind this idea is explained clearly.
* The evaluation tasks are very sensible. It's good that the model is shown to obtain good perplexity and slightly improve over an LSTM baseline; it's not the state of the art, but that's not the point of the paper (in fact, I would emphasize that even more than the authors do). The unsupervised parse evaluation (Table 2) is the heart of the paper, in my opinion (and should probably be emphasized more) -- the results from the second layer are quite impressive.
* The (mildly) better performance than LSTMs on long-distance dependencies, and (mildly) worse performance on local dependencies, in the Marvin & Linzen dataset, is interesting (and merits additional analysis).
Weaknesses:
* The discussion of the motivation for unsupervised structure induction in the introduction is somewhat confused. I am not sure that neural networks with latent syntactic structures can really address the seemingly very fundamental question mentioned in the first paragraph (whether syntax is related to "an underlying mechanism of human cognition") - I would suggest eliminating this part. At the same time, the authors might want to add another motivation for studying architectures that discover latent structure (as opposed to being given that structure) - this setting corresponds more closely to human language acquisition, where children aren't given annotated parse trees.
* The authors discuss hierarchy in terms of syntactic structure alone, but it would seem to me that the hierarchy that the LSTM is inducing could just as well include topic shifts, speech acts and others, especially if the network is trained across sentences.
* There is limited analysis of the model. Why does the second layer show better unsupervised parsing performance than the third layer? (Could this be related to syntactic vs. semantic/discourse units I mention in the previous bullet?) Why is the model better at ADJP boundaries than NP boundaries? It would have been more useful to report less experiments but analyze the results of each experiment in greater depth.
* In this vein, I am not sure it's useful to include WSJ10 in Table 2, which is busy as it is. These sentences are clearly too easy, as the right branching baseline shows, and require additional POS tagging.
* I found it difficult to read Figure A.2: could you help us understand what we should take away from it?
* It is not entirely clear why the model needs both unit-specific forget/input gates and the "master" forget/input gates, and there is no discussion of this issue. Have you tried using only the "master" gates?
Minor notes:
* RNNGs are described as having an explicit bias to model syntactic structure; this is an arguably confusing use of the word "bias", in that the architecture has a hard constraint enforcing syntactic structures (bias implies a soft constraint).
* There are some language issues: agreement errors (e.g. "have" in the sentence that starts with "Developing" in the introduction), typos ("A order should exist", "co-occurance"), determiner issues ("values in [the] master forget gate", "when the overlap exists") - I would suggest going through and copy editing the paper.
* "cummax" seems like a better choice of name for cumulative maximum than "cumax".
* It may be helpful to remind the reader of the update equation for c_t in a standard LSTM.
* Did the language model have 1150 units in each layer or in total? Why did you use exactly three layers? Did you try one, two and four?
* It's not clear if the results in Table 2 reflect the best seed out of five (as the title of the column "max" indicates) or the average (as the caption says).


Symbolic logic allows practitioners to build systems that perform rule-based reasoning which is interpretable and which can easily be augmented with prior knowledge. However, such systems are traditionally difficult to apply to problems involving natural language due to the large linguistic variability of language. Currently, most work in natural language processing focuses on neural networks which learn distributed representations of words and their composition, thereby performing well in the presence of large linguistic variability. We propose to reap the benefits of both approaches by applying a combination of neural networks and logic programming to natural language question answering. We propose to employ an external, nondifferentiable Prolog prover which utilizes a similarity function over pretrained sentence encoders. We fine-tune these representations via Evolution Strategies with the goal of multi-hop reasoning on natural language. This allows us to create a system that can apply rule-based reasoning to natural language and induce domainspecific natural language rules from training data. We evaluate the proposed system on two different question answering tasks, showing that it complements two very strong baselines -BIDAF (Seo et al., 2016a) and FASTQA (Weissenborn et al., 2017) -and outperforms both when used in an ensemble.

UPDATE: Given the authors' rebuttal and the clear improvements to their paper, I've increased my rating of the work.
=======================
This paper presents NLProlog, a method that combines logical rules with distributed representations for reasoning on natural language statements. Natural language statements (first converted to logical triples) and templated logical rules are embedded in a vector space using a pretrained sentence encoder. These embedded "symbols" can be compared in vector space, and their similarity used in a theorem prover (Prolog) modified to support weak unification. The theorem prover determines the answer to a natural language query by constructing a proof according to its logical rules.
Training through the non-differentiable theorem prover occurs via an "evolutionary strategy," which enables the model to fine-tune its sentence encoders and learn domain-specific logic rules directly from text. The authors also propose a Gradual Rule Learning (GRL) algorithm that seems necessary for the optimization process to converge on good solutions.
Despite the model's complexity, the paper was fairly clear to me.
Although the proposed model is a conglomeration of pre-existing parts, the combination is original to my knowledge. The use of Open Information Extraction to transform natural language statements to logical statements, which amenable to theorem provers, is novel and also circumvents the complicated preprocessing required by previous related works.
The authors evaluate the proposed approach on subsets of the Wikihop dataset and BABI-1k. NLProlog performs competitively with neural models, similarly augmented with Sent2Vec but lacking explicit logical rules, only on the 'country' subset of Wikihop. It does not compete with or clearly outperform these models in general. As the authors state, it further "struggles to find meaningful rules for the predicates 'developer' and 'publisher'."
NLProlog demonstrates strong performance on a subset of problems labelled unanimously by annotators to require multi-hop reasoning. Unfortunately, this is only done for the "country" subset of Wikihop, on which the model was already shown to have the strongest performance. I'd find this more convincing if similar improvements were shown on the other subsets (publisher, developer).
Taking into account also that the BABI subset was used only for ablation, the limited results call into question the significance of the work. It would definitely benefit from more extensive experimental validation. On the other hand, it's very positive to see that NLProlog seems to succeed where the neural models fail, and vice versa, so that the two approaches can be combined in an ensemble to achieve state-of-the-art results. This suggests that the paper's line of research has something to add to the community and should be pursued further. I'd find this result more interesting if an error analysis elucidated some characteristics of the examples that each approach does well/poorly on.
I'd like to see more analysis in general, that answers questions including:
- How reliable is the Open IE system and how does its performance impact the end task?
- How well-specified must the a priori rule structures be to achieve good performance? Further, how does the number and structure of the rules (a hyperparameter in this work) affect performance?
- What is the run-time/complexity of the exhaustive proof search during training?
- Relatedly, you state that you limit the rule complexity to two body atoms in the rule templates for BABI. Can you estimate what rule complexity is required in the Wikihop tasks?
I would like to recommend this work more confidently because it tackles such an important problem and does so in an interesting, well-conceived way. My reluctance arises from the limited experimental validation and analysis. Given more analysis details and experimental evidence from the authors, I'm happy to raise my recommendation.
Pros:
- the method complements standard deep QA models to achieve state-of-the-art results in an ensemble.
- unifying neural representations with logical/symbolic formalisms is an important research direction.
- a code release is planned.
Cons:
- a very complex model, whose details are occasionally unclear; the algorithms in Appendix A are helpful but they are not in the main text.
- the model expresses only a limited subset of first order logic; dynamically changing world states are not supported (yet).
- limited experimental validation.
- it's good to be able to incorporate prior knowledge, but it seems like it's quite necessary to pre-specify rules (in template form).
Minor quibble: Evolutionary learning strategies, such as genetic algorithms, go back a long way. It's strange using only a reference from 2017 to introduce them.


Gradient-based meta-learners such as MAML (Finn et al., 2017) are able to learn a meta-prior from similar tasks to adapt to novel tasks from the same distribution with few gradient updates. One important limitation of such frameworks is that they seek a common initialization shared across the entire task distribution, substantially limiting the diversity of the task distributions that they are able to learn from. In this paper, we augment MAML with the capability to identify tasks sampled from a multimodal task distribution and adapt quickly through gradient updates. Specifically, we propose a multimodal MAML algorithm that is able to modulate its meta-learned prior according to the identified task, allowing faster adaptation. We evaluate the proposed model on a diverse set of problems including regression, few-shot image classification, and reinforcement learning. The results demonstrate the effectiveness of our model in modulating the meta-learned prior in response to the characteristics of tasks sampled from a multimodal distribution.

Strengths:
+ The paper identifies a valid limitation of the MAML algorithm: With a limited number of gradient descent steps from a single initialization, there is a limit to the ability of a fixed-size neural network to adapt to tasks sampled from a diverse dataset.
+ The tSNE plots show some preliminary interesting structure for the simple regression and RL tasks, but not for the classification task.
Weaknesses:
- The motivation of uncovering latent modes of a task distribution does not align with the proposed method. The algorithm computes a continuous representation of the data from a task (which is fixed during gradient-based fast adaptation). The mode identity, on the other hand, should be a discrete variable.  Such a discrete variable is never explicitly computed in the proposed method.
- The technical writing is unclear and jargon is often used without definition. Importantly, one of the central motivators of the paper, "task modulation", is never given a precise definition.
- The standard few-shot classification task (Omniglot) does not clearly consist of a task distribution that is multimodal, so the method is not well-motivated in this setting.
- Experimental conclusions are weak.
Major comments:
- The paper neglects to discuss how the proposed method could be used in the context of other methods for "gradient-based meta-learning" such as Ravi & Larochelle (2016). I believe the attention-based modulation and the FiLM modulation could be easily adapted to that setting. Why was this not discussed or evaluated?
- Conditioning has been used in the context of few-shot learning before, but this is not discussed (https://arxiv.org/abs/1805.10123, https://arxiv.org/abs/1806.07528).
- The paper often confounds task representation with neural network parameter values. For example, Figure 1 depicts the adaptation of parameter values with gradients (\nabla L), yet the caption describes "task modes." More careful writing would disentangle these two components.
- The motivation for the particular form of the task embedding computation is not given. What were the other options? Why not, for example, an order-invariant function instead of a bidirectional GRU?
- In all of the experiments, there is no appropriate baseline that keeps the parameter dimensionality constant, so it is unclear whether the (marginal) improvement in performance is due to added expressivity by adding more parameters rather than an algorithmic improvement. I suggest an ensembling baseline with an appropriate number of ensemble members.
- There is no evaluation on a standard benchmark for few-shot classification (miniImageNet), and the Omniglot improvement is small.
- The reinforcement learning comparison at some point compares MUMOMAML with modulation applied (therefore with access to task-specific data) to MAML with no adaptation (and therefore no access to task-specific data). This is not entirely fair.
- tSNE results can be misleading (e.g., see https://distill.pub/2016/misread-tsne/), and the task delineation is not extremely clean. I would be more convinced if a clustering algorithm were applied.
Minor comments:
- The paper needs to be checked over for English grammar and style.
- everywhere: The "prior" referred to in this paper is not a prior in the Bayesian sense. I suggest a more careful use of terminology.
- abstract: "augment existing gradient-based meta-learners" You augment a specific variant of gradient-based meta-learning, MAML.
- pg. 1: "carve on a snowboard" don't know what this means
- The terminology of "task distribution" and "modes" thereof is used without introduction in the introduction section. The terminology "model-based meta-learning/adaptation" and "gradient-based meta-learning/adaptation" is also used without introduction here. This makes the introduction unnecessarily opaque. Consider the reader who is not familiar with meta-learning papers; they would have a very hard time parsing, for example, the phrase "...this not only requires additional identity information about the modes, which is not always available or is ambiguous when the modes are not clearly disjoint..." (pg. 1).
- Further, the terminology "model-based" seems non-standard, and is aliased with the term model-based reinforcement learning (which specifically refers to the set of RL algorithms that make use of a "model" of transition dynamics). Since the paper tackles a reinforcement learning benchmark, this may lead to some confusion.
- pg. 3; "our model does not maintain an internal state" Is the task representation/embedding not an internal state?
- pg. 3: "relevant but vaguely related skills" this is imprecise
- pg. 3: The episodic training setup, which is standard to meta-learning setups, could be much better described. The MAML algorithm could be given better intuition.
- everywhere: "task specific" -> task-specific
- Algorithm 1: "infer" is a misuse of terminology that usually refers to an operation in latent variable probabilistic modelling. Since the computation of \tau is purely feedforward, I recommend writing "compute."
- \tau should be used in some places where v is used instead


We present a powerful new loss function and training scheme for learning binary hash codes with any differentiable model and similarity function. Our loss function improves over prior methods by using log likelihood loss on top of an accurate approximation for the probability that two inputs fall within a Hamming distance target. Our novel training scheme obtains a good estimate of the true gradient by better sampling inputs and evaluating loss terms between all pairs of inputs in each minibatch. To fully leverage the resulting hashes, we use multi-indexing. We demonstrate that these techniques provide large improvements to a similarity search tasks. We report the best results to date on competitive information retrieval tasks for ImageNet and SIFT 1M, improving MAP from 73% to 85% and reducing query cost by a factor of 2-8, respectively.

Summary: This paper contributes to the area of learning to hash. The goal is to take high-dimensional vectors in R^n resulting from an embedding and map them to binary codewords with the goal of similar vectors being mapped to close codewords (in Hamming distance). The authors introduce a loss function for this problem that's based on angles between points on the hypersphere, relying on the intuition that angles corresponds to the number of times needed to cross to the other side of the hypersphere in each coordinate. This is approximately the Hamming distance under a simple quantization scheme. The loss function itself forces similar points together and dissimilar points apart by matching the Hamming distance to the binomial CDF of the angle quantization. They also suggest a batching scheme that enforces the presence of both similar and dissimilar matches. To confirm the utility of this loss function, the authors empirically verify similarity on ImageNet and SIFT.
Strengths: The main idea, to match up angles between points on the hypersphere and Hamming distance is pretty clever. The loss function itself seems generally useful.
Weaknesses: First, I thought the paper was pretty difficult to understand without a lot of background from previous papers. For the most part the authors don't actually state what the input/output/goals are, leaving it implied from the context, which is tough for the reader. The overall organization isn't great. The paper doesn't contain any theory even for simplified or toy cases (which actually seems potentially tractable here); there is only simple intuition. I think that is fine, but then the empirical results should be extensive, and unfortunately they are not.
Verdict: I think this work contains a great main idea and could become quite a good paper in the future, but the work required to illustrate and demonstrate the idea is not fully there yet.
Comments and Questions:
- Why do you actually need the embedded points y to be on the unit hypersphere? You could compute distances between points at different radii. The results probably shouldn't change much.
- There's at least a few other papers that use a similar idea, for example
Gong et al "Angular Quantization-based Binary Codes for Fast Similarity Search" at NIPS 2012. Would be good to discuss the differences.
- The experimental section seems very limited for an empirical paper. There's at least a few confusing details, noted below:
- The experimental results for ImageNet comparing against other models are directly taken from those reported by Lu et al. That's fine, but it does mean that it's hard to make comparisons against *any* other paper than the Lu paper. For example, if the selected ImageNet classes are different, then the results of the comparison may well be different. I checked the HashNet paper (Cao et al. 2017), and it papers that their own reported numbers for ImageNet are better than those of the Lu et al paper. That is, I see 0.5059 0.6306 0.6835 for 16/32/64 bit codewords vs Lu's result of 0.442 0.606 0.684, which is quoted in this paper. What's causing this difference? It would probably be a bit less convenient but ultimately better if the results for comparison were reproduced by the authors, and possibly on a different class split compared to the single Lu paper.
- The comparison against PQ should also consider more recent works of the same flavor as PQ, which themselves outperform PQ. For example, "Cartesian k-means" by Norouzi and Fleet, or "Approximate search with quantized sparse representations" by Jain et al. These papers also use the SIFT dataset for their experimental result, so it would be great to compare against them.


There has long been debates on how we could interpret neural networks and understand the decisions our models make. Specifically, why deep neural networks tend to be error-prone when dealing with samples that output low softmax scores. We present an efficient approach to measure the confidence of decision-making steps by statistically investigating each unit's contribution to that decision. Instead of focusing on how the models react on datasets, we study the datasets themselves given a pre-trained model. Our approach is capable of assigning a score to each sample within a dataset that measures the frequency of occurrence of that sample's chain of activation. We demonstrate with experiments that our method could select useful samples to improve deep neural networks in a semi-supervised leaning setting.
Unlike traditional supervised learning that requires fully labeled data to train a classifier, semisupervised learning requires only a portion of data to be labeled, which demands less effort in data preparation and is more practical in real life. Consider a semi-supervised leaning task where both the labeled and the unlabeled data come from the same sampling distribution. A straightforward technique is that we first train a model on the labeled data, and then use this model to classify the unlabeled parts of the dataset to enlarge the training set. After this, all the data are labeled regardless of correctness. The model could then be re-trained by iteratively performing the above steps. However, experiments have shown that this approach does not work well on CNNs (see section 3 for details). The reason is two-fold:
• Parts of the data are labeled by our model correctly with extreme confidence, i.e. probability generated by softmax layer is close to 1. This could happen only if our model already learned the features during training, therefore adding these data into the training set will not provide any improvement.
• The model does not pick up some of the features that are contained in the unlabeled data during training. Therefore, those data are doomed to be labeled incorrectly and could potentially weaken our model upon added into the training set.
To summarize these two problems: could we use a particular sample to improve our model; and could we trust our model to label a particular sample correctly. We propose a method to automatically assign scores to data that represents both confidence (could we trust it) and interpretability (could we use it) based on a pre-trained network. Our approach mainly focuses on activations of individual neurons within a network. For a pre-trained CNN and some images within one category, we focus on images' activation maps generated by all of the convolutional and linear layers. The activation maps contain information about each image's relationship with the activated neurons. Then our method could identify neurons that are commonly activated and thereby assigning scores to each individual image based on it's chain of activation. Samples with high scores are assumed to be interpretable and experiments have shown that they are also trustworthy. This entire procedure is unsupervised, i.e. our approach does not need label information to assign image scores. Therefore, we could apply our method to semi-supervised learning tasks including image classification and object detection.
Structure of this paper is the following: in section 2 we formally define image scores and then examine some key properties of our approach; in section 3 we conduct experiments in a semisupervised learning setting to demonstrate how scoring the images could help with improving a pre-trained model, experiments are done in image classification and object detection tasks; section 4 concludes the entire work.

This paper proposes a new metric called the image score that compares the similarity of activation between a given image with a pool of groundtruth images. The paper finds it useful for semi-supervised learning with self-teaching, where the network picks the most confident sample and use the network prediction as the label. It finds that the proposed method is better than 1) not using the unlabeled data and 2) using softmax as an indicator for model prediction certainty.
Motivation: The introduction begins by motivating the interpretability story of deep learning, but I don’t see gaining any more interpretability by reading the rest of the paper. The paper proposes to improve interpretability by assigning a score to each individual example, but then the obtained scores are not properly analyzed in the paper, and only final classification accuracy is evaluated. What are the training samples that makes the model make certain decision at test time? How to measure the correlation between the usefulness of training samples and the proposed image score? These questions left unanswered in the paper. Figure 1 helps a little bit, but then the top row is not necessarily the bad images, but maybe hard examples that needs extra attention to learn. Therefore, I think the end results presented in the experiments do not align with the motivation. Rather than shooting for interpretability, this is just another semi-supervised learning paper.
Models: The major issue of this paper is the model formulation that is not well motivated. The intuition of how the authors come up with the equation for computing the image score is not well explained. Hence the formulation seems very ad-hoc, and it is unclear why this is the selected method.
Experiments: As a semi-supervised learning paper, a common setting for CIFAR-10 is to use 4k labeled images. Here, the method uses 30k, which is 7.5x the size of the usual setting. It also does not compare to prior semi-supervised learning work (e.g. one of the recent one is: https://arxiv.org/abs/1711.00258). The only two baselines discussed here are weak. Also the improvement from the baselines by using the proposed method is not very significant.
Comparison: Figure 2-4 shows some positive correlation between the accuracy and score, which is fair, but it doesn’t compare to any baselines--the only one we have is softmax baseline and it is not shown in the figure.
In conclusion, I couldn’t see how the paper improves interpretability as claimed in the introduction. The proposed method seems ad-hoc, without any justification. Being considered as a semi-supervised learning paper, it lack significant amount of comparison to prior work and adopting a common semi-supervised benchmark. Due to the above reasons, I recommend reject.
---
Minor points:
“...almost all of the existed works investigate only the models and ignore the relationship between models and samples”. This is over-exaggerated. I believe most of the visualization techniques are dependent on the actual input samples. It is true to say about “training samples” not “samples” in general.
“all correctly classified images should have similar chain of activation, while incorrectly classified images should have very different activations both within themselves and with correctly classified images”. This claim seems not backed up. How do you know it is the case for “all” correctly classified images? What defines similar/different?


BIG-LITTLE NET: AN EFFICIENT MULTI-SCALE FEATURE REPRESENTATION FOR VISUAL AND SPEECH RECOGNITION
In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy. This is achieved by using a multi-branch network, which has different computational complexity at different branches with different resolutions. Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation. The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks, using popular architectures including ResNet, ResNeXt and SEResNeXt. For object recognition, our approach reduces computation by 1/3 while improving accuracy significantly over 1% point than the baselines, and the computational savings can be higher up to 1/2 without compromising the accuracy. Our model also surpasses state-of-the-art CNN acceleration approaches by a large margin in terms of accuracy and FLOPs. On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains.

This paper presents a novel multi-scale architecture that achieves a better trade-off speed/accuracy than most of the previous models. The main idea is to decompose a convolution block into multiple resolutions and trade computation for resolution, i.e. low computation for high resolution representations and higher computation for low resolution representations. In this way the low resolution can focus on having more layers and channels, but coarsely, while the high resolution can keep all the image details, but with a smaller representation. The branches (normally two) are merged at the end of each block with linear combination at high resolution. Results for image classification on ImageNet with different network architectures and for speech recognition on Switchboard show the accuracy and speed of the proposed model.
Pros:
- The idea makes sense and it seems GPU friendly in the sense that the FLOPs reduction can be easily converted in a real speed-up
- Results show that the joint use of two resolution can provide better accuracy and lower computational cost, which is normally quite difficult to obtain
- The paper is well written and experiments are well presented.
- The appendix shows many interesting additional experiments
Cons:
- The improvement in performance and speed is not exceptional, but steady on all models.
- Alpha and beta seem to be two hyper-parameters that need to be tuned for each layer.
Overall evaluation:
Globally the paper seems well presented, with an interesting idea and many thorough experiments that show the validity of the approach. In my opinion this paper deserves to be published.
Additional comments:
- - In the introduction (top of pag. 2) and in the contributions, the advantages of this approach are explained in a different manner that can be confusing. More precisely in the introduction the authors say that bL-Net yeald 2x computational saving with better accuracy. In the contributions they say that the savings in computation can be up to 1/2 with no loss in accuracy.


There were many algorithms to substitute the back-propagation (BP) in the deep neural network (DNN) training. However, they could not become popular because their training accuracy and the computational efficiency were worse than BP. One of them was direct feedback alignment (DFA), but it showed low training performance especially for the convolutional neural network (CNN). In this paper, we overcome the limitation of the DFA algorithm by combining with the conventional BP during the CNN training. To improve the training stability, we also suggest the feedback weight initialization method by analyzing the patterns of the fixed random matrices in the DFA. Finally, we propose the new training algorithm, binary direct feedback alignment (BDFA) to minimize the computational cost while maintaining the training accuracy compared with the DFA. In our experiments, we use the CIFAR-10 and CIFAR-100 dataset to simulate the CNN learning from the scratch and apply the BDFA to the online learning based object tracking application to examine the training in the small dataset environment. Our proposed algorithms show better performance than conventional BP in both two different training tasks especially when the dataset is small.

The paper propses to use a combination of Direct Feedback Allignment (DFA) and BackPropagation (BP) to improve upon standard back propagation.
To understand what is done, consider the following: Feedback Alignment is +- equal to back propagation when using random but fixed weights in the backwards pass. Direct feedback alignment uses random backprojections directly to the layer of interest.
The advantage of DFA is that It bypasses the normal computational graph. The advantage of this is that if compute is infinite, all of these updates can be computed in parallel instead of pipelining them as is done in standard BP.
In the current paper, the use of DFA for dense layers and BP for conv layers which is named CDFA is proposed.
In addition the paper also proposes a binarized version of BDFA to limit memory consumption and communication. It is claimed that the proposed techniques improve upon standard back propagation.
Overall, the paper is easy to understand, but I lean towards rejecting this paper because I am not convinced by the experimental evidence. As outlined below, the key issue is that the baseline appears to be weak. Additionally, the main limitation of the proposed approach can only benefit a very limited set of architectures.
Positive points:
---------------------
The authors did an excellent job of introducing BP, FA and DFA in the paper. This makes the core concepts and ideas accessable without having to delve through prior work.
The own contributions and the key idea is easy to understand.
Limitations and possible improvements
-------------------------------------------------------
A core limitation is that recent networks do not have a combination of dense layers and convolutional layers. In many cases the networks are fully convolutional, this limits the applicability of the proposed combination of DFA and BP. The use of additional networks would benefit the paper. Currently only VGG 16 on Cifar 10 is used. Also, the data augmentation strategy is not discussed. Of course, it would be nice if additional datasets could be included as well, but this of course depends on the computational resources the authors have available.
The key issue to me is that performance improvements for CIFAR are reported, but I fear that the baseline accuracy for VGG16 might be a bit low. If I memory serves me well, it should be able to achieve around 90% at least on CIFAR 10 using VGG style networks. I did a quick search and found http://torch.ch/blog/2015/07/30/cifar.html corroborating this but I did not verify this directly.
Related to the previous point, since this is an empirical paper, describing the hyper-parameter optimizations and final settings in detail  can convince the reader that the study is exectued correctly. Much of the information is missing now.
Similarly, I have trouble understanding section 4.1 and section 4.2 since I do not know the exact details of the experiments. This can be fixed easily however.
Provide complexity estimates of the potential speedup or provide actual timing information. (Although this might not be that meaningful without much additional work given that gpu kernels are often heavily optimized).
Last year there was a submission to ICLR about fixing the final output layer and only learning the convolutional layers. If we consider that random projections work remarkably well and can be considered approximations of kernels, it could be interesting to add a baseline where the fully connected layers are fixed and only the convolutional layers are trained. The error signal can be propagated using standard BP, FA or DFA methods but it would shed light on whether learning in the higher layers is actually needed or BP in the conv layers is sufficient.
Minor possible improvements
------------------------------------------
Finally, I would strongly suggest that the authors perform some additional proofreading. There are quite a few strange formulations and spelling mistakes. That being said, it did not prevent me from understanding the manuscript so this remark DID NOT factor into my judgement.
In addition to remark above, I would suggest removing the second paragraph from the introduction. It feels out of place to me, and the vanishing gradient effects are not discussed in the remainder of the manuscript.
The list of possible optimizers before the selection for SGD+Momentum is not needed. Simply stating that SGD with momentum is used should be sufficient.
“Training from scratch” instead of “Training from the scratch”


GUIDING POLICIES WITH LANGUAGE VIA META-LEARNING
Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines. However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task. In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill. Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly. In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.

UPDATE: I've increased my rating based on the authors' thorough responses and the updates they've made to the paper. However, I still have a concern over the static nature of the experimental environments.
=====================
This paper proposes the use of iterative, linguistic corrections to guide (ie, condition and adjust) an RL policy. A major challenge in learning language-guided policies is grounding the language in environment states and agent actions. The authors tackle this challenge with a meta-learning approach.
The approach is fairly complex, blending imitation and supervised learning. It operates on a training set from a distribution of virtual pick-move-place tasks. The policy to be learned operates on this set and collects data, via something close to DAgger, for later supervised learning on the task distribution. The supervised-learning data comprises trajectories augmented with linguistic subgoal annotations, which are referred to as policy "corrections." By ingesting its past trajectories and the correction information, the policy is meant to learn to solve the task and to ground the corrections at the same time, end-to-end. Correction annotations are derived from an expert policy.
The idea of guiding a policy through natural language and the requisite grounding of language in environment states and policy actions have been investigated previously: for example, by supervised pretraining on a language corpus, as in the cited work of Andreas et al. (2018). The alternative meta-learning approach proposed here is both well-motivated and original.
Generally, I found the paper clear and easy to read. The authors explain convincingly the utility of guiding policies through language, especially with respect to the standard mechanisms of reward functions (sparse, engineered) and demonstrations (expertise required). The paper is also persuasive on the utility of iterative, interactive correction versus a fully-specified language instruction given a priori. The meta-learning algorithm and training/test setup are both explained well, despite their complexity. On the other hand, most architectural details necessary to reproduce the work are missing, at least from the main text. This includes various tensor dimensions, the structure of the network for perceiving the state, etc.
I like the proposed experimental setting. It enables meta-learning on sequential decision making problems in a partially observable environment, which seems useful to the research community at large. Ultimately, however, this paper's significance is not evident to me, mainly because the proposed method lacks thorough experimental validation. No standard baselines are evaluated on the task (with or without meta-learning), nor is a detailed analysis of the learned policies undertaken. The ablation study is useful, and a good start, but insufficient in my opinion. Unfortunately, the results are merely suggestive rather than convincing.
Some things I'd like to see in an expanded results section before recommending this paper include:
- Comparison to an RL baseline that attempts to learn the full task, without meta-training or language corrections.
- Comparison to a baseline that learns from intermediate rewards. Instead of annotating data with corrections, you could provide +/- scalar rewards throughout each trajectory based on progress towards the goal (since you know the optimal policy). How effective might this be compared to using the corrections?
- Comparison to a baseline that does some kind of pretraining on the language corrections, as in Andreas et al. (2018).
- Quantification of how much meta-training data is required. What is the sample complexity like with/without language corrections?
I also have concerns about the need for near-optimal agents on each task -- this seems very expensive and inefficient. The expert policy is obtained via RL on each individual task using "ground truth" rewards. It is not specified what these rewards are, nor is it stated how near to optimal the resulting policy is nor how this nearness affects the overall meta-learning process.
Its unclear to me how the "full information" baseline processes and conditions on the full set of subgoals/corrections. Are they read as a single concatenated string converted to one vector by the bi-LSTM?
There also might be an issue with the experimental setup, unless I've misunderstood it. The authors state that "the agent only needs 2 corrections where the first correction is the location of the goal object and the second is the location of the goal square." But if the specific rooms, indicated by colors, do not change location from task to task (and they appear not to from all the figures), then the agent can learn the room locations during meta-training and these two "corrections" tell it everything it needs to know to solve the task.
Pros:
- Appealing, well-motivated idea for training policies via language.
- Clear, pleasant writing and good communication of a complicated algorithm.
- Good experimental setup that should be useful in other research (except for possible issue with static room locations).
Cons:
- The need for a near-optimal policy for each task.
- Overall complexity of the training process.
- The so-called corrections are actually linguistic statements of subgoals computed from the optimal policy. There is much talk in the introduction of interactive policy correction by humans, which is an important goal and interesting problem, but the present paper does not actually investigate human interaction. This comes as a letdown after the loftiness of the introduction.
- Various details needed for reproduction are lacking. Maybe they're in the supplementary material; if so, please state that in the main text.
- Major lack of comparisons to alternative approaches.


PER-TENSOR FIXED-POINT QUANTIZATION OF THE BACK-PROPAGATION ALGORITHM
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suitable precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.

This paper proposes a FX (fixed point) framework to calculate the reduced bit numbers, which can (I) use float numbers with the reduced bits to represent each NN layer's weight values W_l, activation values A_l, gradients of weights G_l^W, gradients of nodes G_{l+1}^A, and the cumulated (updated) weight W_l^(acc); (II) with the reduced representations, the training loss and testing loss will not be sacrificed much comparing with the original FL framework, where each float number is 32-bit.
Some positive points:
(a) The proposed FX framework can reduce cost for both inference and training.
(b) The experimental results looks promising.
(c) The Criteria 1-5 seems systematic and the conditions in Claim 1 can be used to calculate the required bit numbers. The author proposed an implementation of Claim 1.
Some negative points / questions:
(a) The most important part of the paper is Criteria 1-5. Criteria 1 generalizes the idea from Sakr et al. 2017, to force the contributions of weight and activation almost at the same order, which seems reasonable to make the mismatch budget p_m smallest. But the other criteria (with their corresponding notions, e.g., Criterion 2 and the concept of clipping rate \beta) are introduced in a way which is not clear enough and make the audience confused. For example, why is clipping rate \beta and relative quantization bias \eta is needed here and what is their relationship with the usual weight gradient clip norm (5% target \beta and 1% \eta target correspond to what order of clip norm)? Criteria 4 & 5 are introduced in the same way with one sentence explained like heuristics. They seem to me are introduced just for reducing corresponding bit numbers. More motivation and explanation of introducing these criteria and notions are needed.
(b) For W_l and A_l, the necessary bit numbers are calculated using Criteria 1 & EFQN condition. But for gradients, their PDRs and Deltas are calculated using other criteria & conditions. Then how to calculate their bit numbers from PDRs and Deltas?
(c) The proof of Lemma 2 & 3 directly used CLT for mini-batch average gradient items. CLT is for asymptotic case and in finite sample case it is not true. So it is heuristic calculation rather than lemma with proof (If seeking proof then some finite-sample argument like Berry-Esseen theorem is needed to quantify the probability of the average is not Gaussian). And what is the mini-batch size used here? If it is too small then probably the error of taking the mini-batch SGD as Gaussian will be large.
(d) The importance/minimality of each individual bit number in C_o is not investigated, and the claim of the near minimality of C_o cannot hold according to the current experiments. The experiments of C_{+1} and C_{-1} do not exclude the possibility that some items (not the whole C_o) are minimal. More experiments (changing one or more items while fixing the others) are needed to show every item in C_o is minimal (or not). And which one of them is the most important for training/testing (how sensitive the training/testing performance is to each bit number)? Also in C_{-1} and C_{+1}, are target \beta, \eta changed? What are these changed values?
(e) From the Claim 1, it seems that these bit numbers are sufficient, and smaller than necessary to get the same training/testing performance (e.g., from the proof of Lemma 3, with the result \eta = 0.4% < 1%). So one question is what kind of \beta and \eta target are necessary for preserving the performance and what corresponding bit numbers are necessary to achieve the necessary \beta and \eta values?
(f) The computational cost definition looks different with Eq (3) in Sakr et al. 2017. Why?
Some typos:
1.Figure 2(a), B_{A_l} is 8 for Layer 1, but 9 in Appendix Table.
2. In the last third paragraph of Page 7, is it 2.6 = (148/56.5) instead of 2.6 * (148/56.5)? Same for the following numbers.
========================================
Revision: I have read the reply from the authors and it clarified several matters. I adjusted my rating of this paper (from 6 to 7).


Which generative model is the most suitable for Continual Learning? This paper aims at evaluating and comparing generative models on disjoint sequential image generation tasks. We investigate how several models learn and forget, considering various strategies: rehearsal, regularization, generative replay and fine-tuning. We used two quantitative metrics to estimate the generation quality and memory ability. We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10). We found that among all models, the original GAN performs best and among Continual Learning strategies, generative replay outperforms all other methods. Even if we found satisfactory combinations on MNIST and Fashion MNIST, training generative models sequentially on CIFAR10 is particularly instable, and remains a challenge. Our code is anonymously available online 1 .

This paper performs an empirical comparison of models and CL methods in a generative setting. The main motivation of the paper is to make statements about which model/method combinations are best to use for generative tasks in the CL setting. In short, the paper provides an empirical analysis and evaluation of the combination of CL methods and generative models.
The datasets used for comparison are MNIST, Fashion MNIST, and CIFAR10. For each dataset, sequential (class by class) generative tasks are introduced, aligning with the CL setting. The models investigated are VAEs, GANs, and WGANs, along with their (class) conditional counter-parts. The CL methods investigated are (i) fine-tuning (a simple baseline), (ii) rehearsal methods, (iii) elastic weight consolidation (EWC), and (iv) generative replay (GR). The authors propose to use two evaluation metrics: Fréchet Inception Distance (FID) measures the quality of the generated images, and fitting capacity (FC) measures the usefulness of the images to train classifiers.
Pros:
- The authors are correct in pointing out that most of the work on CL has been restricted to the discriminative case, and that there is value in exploring generative tasks in the CL setting.
- Empirical and experimental evaluation of this sort are useful, and help the community better understand the relationship between model, CL method, and task. Such an evaluation and in-depth analysis is welcomed in CL, especially in the generative setting.
- The authors draw a number of useful conclusions e.g., regarding the usefulness and dangers of employing the different CL methods.
Cons:
- My main concern with this paper regards the evaluation metrics used. The authors propose quality metrics for the generative model, both of which (directly or indirectly) measure the quality of the generated images. In this setting, it is unsurprising that GANs outperform VAEs, as they are known to generate higher-quality images. This however, does not necessarily mean that they are better at the continual learning task (i.e., avoiding catastrophic forgetting). It seems to me that one source from which to draw would be [1], which conducted a very rigorous and useful empirical evaluation of generative models, and the methodology followed there (i.e., evaluating marginal log-likelihoods via annealed importance sampling) would be more convincing evidence for empirical comparison of models, as it would somewhat detach the quality of the generated images from the ability of the model to avoid catastrophic forgetting.
Using their proposed image-quality metrics, the authors make statements such as: "Our results do not give a clear distinction between conditional and unconditional models. However, adversarial methods perform significantly better than variational methods. GANs variants are able to produce better, sharper quality and variety of samples, as observed in Fig. 13 and 14 in Appendix G. Hence, adversarial methods seem more viable for CL." My impression is that this statement on the viability of VAEs vs GANs for CL, which is a major point of the paper, does not follow from the empirical results on the quality of the generated images. It seems quite predictable that the GAN-based models would produce higher quality images, regardless of catastrophic forgetting.
Additional (minor) comments:
- Sec. 2 could consist of a more thorough review of the literature, with a more in-depth comparison of the different CL methods proposed and evaluated in the paper.
- Sec. 2 contains a number of statements of the form: "restricted to VAEs only". For many of the cases it is not immediately clear why this is true, and in my opinion the authors should either drop those comments, or make them rigorous.
- VCL "use specific weights for each task, which only works for the setting where the number of tasks is known in advance". Unclear what exactly this means or why this is true.
- "while the teacher retains knowledge" - how does it "retain knowledge", how is this then transferred to the student, and why is this restricted to VAEs?
Experimental protocol:
- Core-sets for the rehearsal as proposed by [1] could be an interesting extension. It is unclear how the samples were selected for rehearsal, and core-sets represent a principled way to do so, that would also be interesting to compare in this setting to a random baseline.
- For VAEs, a potentially better metric of their ability (other than the log-likelihood as suggested by [2]) would be fitting capacity (or other metric) over learned latent space rather than the reconstructed image-space.
Overall, my impression is that while an empirical analysis of CL methods in the generative setting is a useful concept, the submission in its current form requires some improvement. In particular, I am worried that the choice of evaluation metrics may lead to incorrect (or partially correct) conclusions, which could of course have a negative impact on the research into CL. It also seems that the paper could use some further polishing in both writing and presentation. As such, I encourage the authors to continue the work on this empirical analysis, and perhaps submit in again to future conferences.
[1] - Nguyen et al. Variational Continual Learning, ICLR 2018
[2] - Wu et al. On the Quantitative Analysis of Decoder-Based Generative Models, ICLR 2017