The principle of compositionality, which enables natural language to represent complex concepts via a structured combination of simpler ones, allows us to convey an open-ended set of messages using a limited vocabulary. If compositionality is indeed a natural property of language, we may expect it to appear in communication protocols that are created by neural agents in language games. In this paper, we propose an effective neural iterated learning (NIL) algorithm that, when applied to interacting neural agents, facilitates the emergence of a more structured type of language. Indeed, these languages provide learning speed advantages to neural agents during training, which can be incrementally amplified via NIL. We provide a probabilistic model of NIL and an explanation of why the advantage of compositional language exist. Our experiments confirm our analysis, and also demonstrate that the emerged languages largely improve the generalization of the neural agent communication.

This paper studies the emergence of compositional language in neural agents. They propose an iterated learning method that consists of three phases: a supervised learning phase for a randomly-initialized speaker and listener, a self-play phase (where both agents are updated together), and a phase where a new dataset is created based on the current speaker’s language. This dataset is then passed on to the next ‘generation’ of speaker and listener. The paper finds that this procedure, with the right hyperparameters, leads to the emergence of more compositional languages in a simple symbolic referential game.
The question of how to emerge a compositional language is indeed interesting. This paper does a good job of conducting a careful set of ablations and analyzing the results. In my mind, the main scientific contribution of this work is the empirical verification of the principle ‘compositional languages are easier to learn’. While this principle is intuitive, it’s good to see it confirmed via experiments. The paper’s description of the ‘interval of advantage’ --- the range of updates where a compositional language performs better on the task than a non-compositional language --- is insightful to me.
I do have concerns for this paper around utility and novelty. As the paper mentions, it has already been shown that iterated learning procedures give rise to more compositional languages in non-neural models. While there are some things to consider in adapting this to neural networks, to my eye they seem rather straightforward (i.e. tuning the number of updates of the speaker and listener, the values I_a and I_b), contrary to the paper’s assertion.  From a utility perspective, the paper doesn’t go into how this might be practically applied in general to train neural agents to learn compositional languages in more complex environments (where they might be simultaneously speakers and listeners), as they stick to a very simple symbolic referential game. The main contribution of this paper is really: “studying how neural networks behave when trained in an iterated learning setting in a simple referential game“. I think this is a nice contribution, but the main question for me is whether this is enough for an ICLR acceptance.
My other concern is around the length of the paper. In my opinion, while the paper is well-written, it’s quite bloated and there is a lot of repetition. I think the paper could easily be condensed to 8 pages and retain the same information. Alternatively, some of the graphs in the Appendix (which are quite nice) could be added to the main paper to give more insight about how neural networks behave in this iterated learning procedure.
Finally, the paper shows that compositional languages generalize better to the held-out validation set. While this is also an intuitive result, it’s nice to have in the paper. I’d encourage the authors to remove the ‘zero-shot’ terminology though (which usually refers to predictions on new samples outside of the training distribution), and just stick to what is actually being shown, which is improved generalization.
Overall, I like the paper, but due to the concerns mentioned above I think it’s borderline, with a slight lean towards rejection.


In this paper, we introduce MAMUS for constructing multilingual sparse word representations. Our algorithm operates by determining a shared set of semantic units which get reutilized across languages, providing it a competitive edge both in terms of speed and evaluation performance. We demonstrate that our proposed algorithm behaves competitively to strong baselines through a series of rigorous experiments performed towards downstream applications spanning over dependency parsing, document classification and natural language inference. Additionally, our experiments relying on the QVEC-CCA evaluation score suggests that the proposed sparse word representations convey an increased interpretability as opposed to alternative approaches. Finally, we are releasing our multilingual sparse word representations for the 27 typologically diverse set of languages that we conducted our various experiments on.

This paper describes a method to build sparse multilingual word vectors that is designed to scale easily to many languages. The key idea is to pick one language to be the source language, and then to build word embeddings and then a sparse dictionary + sparse coefficients  for that source language monolingually. Other target languages then first align their embedding to the source using a seed list of translations and standard techniques, and then determine their sparse coefficients based on the fixed source sparse dictionary. This latter process is a convex optimization, which improves efficiency and stability. The method is tested with an established correlation-based intrinsic metric (QVEC-CCA) as well as by using the multilingual embeddings to project systems for cross-lingual document classification, dependency parsing and natural language inference.
The core idea of this paper is substantially simpler than the method it compares to (BiSparse), which jointly optimizes dictionaries, coefficients and cross-lingual coefficient alignment. So, the question that immediately comes to mind for me is, how much accuracy am I giving up for improved scalability to multiple languages? This could be easily tested for the two language case, where BiSparse could be directly compared without modification to their proposed method. Instead, BiSparse is modified to fit scale to the multilingual setting (becoming MultiSparse), but since it shares the constraint that the source dictionary and coefficients are fixed, it has already lost a lot of the power of joint optimization. I think the paper would be stronger with a two-language experiment where we would expect the proposed method to lose to BiSparse, but we could begin to understand what has been given up for scalability.
I also wonder how relevant bilingual word embeddings are in a world of multilingual BERT and similar approaches. It would be interesting to know how cross-lingual embedding-in-context methods would do on the extrinsic evaluations in this paper, though I also acknowledge that this could be considered beyond the scope of the paper.
Otherwise, this is a fine paper. It is well written and easy to follow. The experiments look sane, and the inclusion of both intrinsic and extrinsic tasks makes them fairly convincing. I have only a few remaining nitpicks:
(1) As someone relatively unfamiliar with multilingual (as opposed to bilingual) word embedding research, it wasn’t clear to me how the experiments described here tested the multilinguality (as opposed to bilinguality) of the embeddings. It would be nice to provide an explanation for why (beyond the obvious efficiency gains) one couldn’t just do the necessary language pairs with bilingual methods for these experiments. And if one could perform the tests with bilingual methods, they should be included as baselines.
(2) The discussion of MultiSpare hyper-parameter tuning appears in the Monolingual Experiments section, leading me to wonder what target languages were used for this tuning process.
(3) In the second-last paragraph of 3.2.2, there is a sentence fragment that ends in “nonetheless their multiCluster and multiCCA embedding spaces contain no embeddings for”
(4) The last paragraph before the Conclusion also feels like a fragment, or like two sentences have been spliced together: “We have detailed the differences to Upadhyay et al. (2018) extends the previous work by incorporating dependency relations into sparse coding.”
There are also several places where periods seem to have been left out (such as immediately before the above sentence).


Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.

Summary
This paper proposes a model-based reinforcement learning algorithm suitable for high-dimensional visual environments like Atari. The algorithmic loop is conceptually simple and comprises 1) collecting real data with the current policy 2) updating an environment model with old and newly acquired data and 3) updating the policy "virtually" inside of the environment model using PPO. The approach is evaluated on 26 games from the Atari benchmark and compared against the model-free baselines Rainbow DQN and PPO. The newly proposed model-based method clearly outperforms both model-free baselines in low training regimes (100,000 steps). Further ablation studies are provided, e.g. similar results are obtained in a more stochastic setting of ALE with sticky actions.
Quality
This paper has a strong applied focus and needs to be judged based on its experiments. The quality of those are high. The method is evaluated on a suite of 26 games, compared to strong model-free baselines and results are averaged over 5 seeds. One concern I have is that the method is only evaluated in low training regimes. While I do understand that increasing the training horizon is computationally demanding, results in the appendix (Figure 11a) indicate that the proposed model-based method has worse asymptotic performance compared to the model-free baselines. After 500,000 training steps the effect of sample efficiency vanishes and the final performance results are far away from the final performance results of the model-free baselines after 50,000,000 training steps. Also, a plot similar to Figure 11a) from the appendix for Rainbow DQN would be good (but I do understand this might be difficult to obtain in the course of the review period should this require more experiments).
Clarity
The paper is clearly written and easy to follow. However, the authors could state in the main paper more clearly that their method excels in low training regimes and that the sample efficiency effect seems to vanish when increasing training iterations from 100,000 to 500,000 steps. In fact, Figure 11a) from the appendix should go into the main paper, and it should be also mentioned that there is a huge discrepancy between the maximum performance achieved by the proposed model-based method and the maximum performance achieved by the model-free baselines when training for 50,000,000 steps. Based on the experiments, it is not clear at all if the new method will eventually catch up with best model-free results from the literature when training time is increased, or stall in low-performance regimes indefinitely.
Originality
The originality of this paper is not very high since the proposed algorithm and its components are not novel (there might be some minor novelty in the environment model architecture). However, this paper should not be judged based on its originality but based on its significance.
Significance
A working model-based RL algorithm for Atari is clearly a huge gap in the current literature and this paper takes an important step towards this direction. Demonstrating improved sample efficiency compared to strong model-free baselines in low training regimes is a significant result. The significance is however decreased by the fact that the paper does not answer the question how to obtain good asymptotic performance that matches (or comes close to) model-free state-of-the-art results. I therefore vote for weak accept at this stage.
Minor details
On a side note, there are two citations missing related to model-based RL in visual domains:
- S. Alaniz. Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft. In the 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2017.
- F. Leibfried and P. Vrancx. Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks. In NIPS Deep Reinforcement Learning Workshop, 2018.
Update
After the author response, my review remains the same. I think this paper is worthwhile publishing at ICLR.


The family of feedback alignment (FA) algorithms aims to provide a more biologically motivated alternative to backpropagation (BP), by substituting the computations that are unrealistic to be implemented in physical brains. While FA algorithms have been shown to work well in practice, there is a lack of rigorous theory proofing their learning capabilities. Here we introduce the first feedback alignment algorithm with provable learning guarantees. In contrast to existing work, we do not require any assumption about the size or depth of the network except that it has a single output neuron, i.e., such as for binary classification tasks. We show that our FA algorithm can deliver its theoretical promises in practice, surpassing the learning performance of existing FA methods and matching backpropagation in binary classification tasks. Finally, we demonstrate the limits of our FA variant when the number of output neurons grows beyond a certain quantity.

This paper examines the question of learning in neural networks with random, fixed feedback weights, a technique known as “feedback alignment”. Feedback alignment was originally discovered by Lillicrap et al. (2016; Nature Communications, 7, 13276) when they were exploring potential means of solving the “weight transport problem” for neural networks. Essentially, the weight transport problem refers to the fact that the backpropagation-of-error algorithm requires feedback pathways for communicating errors that have synaptic weights that are symmetric to the feedforward pathway, which is biologically questionable. Feedback alignment is one approach to solving the weight transport problem, which as stated above, relies on the use of random, fixed weights for communicating the error backwards. It has been shown that in some cases, feedback alignment converges to weight updates that are reasonably well-aligned to the true gradient. Though initially considered a good potential solution for biologically realistic learning, feedback alignment both has not scaled up to difficult datasets and has no theoretical guarantees that it converges to the true gradient. This paper addresses both these issues.
To address these issues, the authors introduce two restrictions on the networks: (1) They enforce “monotone” networks, meaning that following the first layer, all synaptic weights are positive. This also holds for the feedback weights. (2) They require that the task in question be a binary classification task. The authors demonstrate analytically that with these restrictions, direct feedback alignment (where the errors are communicated directly to each hidden layer by separate feedback weights) is guaranteed to follow the sign of the gradient.  (Importantly, they also show that monotone nets are universal approximators.) Empirically, they back up their analysis by demonstrating that in fully connected networks that obey these two restrictions they can get nearly as good performance as back propagation on training sets, and even better performance on tests sets sometimes. However, they also demonstrate (empirically and analytically) that violating the second requirement (by introducing more classes) leads to divergence from the gradient and major impairments in performance relative to backpropagation.
Ultimately, I think this is a great paper, and I think it should be accepted at ICLR. It provides some of the first rigorous analysis of feedback alignment since the original paper came out, and unlike those original analyses, it is not restricted to linear networks (which are certainly not universal function approximators). I have looked over the proofs, and they seem to all be correct. As well, I found the paper easy to read, which was nice. However, there are a few things that could be done to clarify the contributions and situate the work within the field of biological learning algorithms better:
1) Though they do not include rigorous analyses, two previous papers have demonstrated empirically that feedback alignment works extremely well as long as the feedback weights share the same sign as the feedforward weights (see: Moskovitz, Theodore H., Ashok Litwin-Kumar, and L. F. Abbott. "Feedback alignment in deep convolutional networks." arXiv preprint arXiv:1812.06488 (2018) and Liao, Qianli, Joel Z. Leibo, and Tomaso Poggio. "How important is weight symmetry in backpropagation?." In Thirtieth AAAI Conference on Artificial Intelligence. 2016). Due to the requirement for monotone networks, this work is also providing a guarantee that the sign of feedforward and feedback weights are the same. That does not subtract substantially from the contributions of this paper, as the provision of the analytical guarantees is important. But, it is important for the authors to consider how their work relates to this past work. For example, could their analytical approach work equally well with nothing more than a sign symmetry guarantee? This should at least be discussed.
2) It should be admitted somewhere in the paper that the second requirement on the networks for binary tasks is deeply unbiological. As such, it should be recognized in discussion that this paper provides some important contributions to our understanding of feedback alignment, but does not ultimately move the question of biologically realistic learning forward all that much. Indeed, the discussion at the end about applications notably ignores biology. But, rather than just ignoring it, the biological mismatch should be openly admitted.
3) The results with the test sets are a little strange, at least for the tests with larger numbers of categories. In Bartunov et al. (2016), they reported not only better training set results with backprop, but also better test set results generally, than feedback alignment. Are the authors sure that their results, in say, Table 4, are not indicative of insufficient hyperparameter optimization?
Small notes:
- Lillicrap et al.’s paper was eventually published in Nature Communications (see citation above), and the reference should be changed to reflect this.
- The discussion on the impact of convolutions could be beefed up a little bit. In particular, it could be discussed relative to the results of Moskovitz et al. (above) who show that convnets work fine with nothing but guaranteed sign symmetry.


The penalized least squares (PLS) is a classic method to inverse problems, where a regularization term is added to stabilize the solution. Optimal transport (OT) is another mathematical framework for computer vision tasks that provides means to transport one measure to another at minimal cost. The cycle-consistent generative adversarial network (cycleGAN) is a recent extension of GAN to learn target distributions with less mode collapsing behavior. Although similar in that no supervised training is required, the algorithms look different, so the mathematical relationship between these approaches is not clear. In this article, we provide an important advance to unveil the missing link. Specifically, we reveal that a cycleGAN architecture can be derived as a dual formulation of the OT problem, if the consistency constraint of PLS is enforced as a regularization term of the OT problem. This suggests that cycleGAN can be considered stochastic generalization of classical PLS approaches. Our derivation is so general that various types of cycleGAN architectures can be easily derived by merely changing the transport cost. As proofs of concept, this paper provides a novel cycleGAN architecture for unsupervised learning in accelerated magnetic resonance imaging (MRI) and deconvolution microscopy problems, which confirm the efficacy and the flexibility of the theory.

In this paper, the authors present two contributions:
1)	The primary contribution is to show that CycleGAN can be formulated as a probabilistic version of a particular penalized-least squares problem (theory)
2)	As proof of concept, they apply their version of CycleGAN to accelerated MRI and deconvolution microscopy (application)
While I find the idea to be potentially interesting, the presentation of the theory is unclear and not well-motivated; after reading, I’m not convinced that the connection to CycleGAN is as significant as the authors claim. The experimental results are preliminary. My decision is to reject. Below are separate critiques on the sections.
Section 2-3: Hope the authors could clarify / strengthen these points in revision:
-	Since the discussion in Section 3 is based on the optimization problem in Equation (7), this problem should be well-motivated. Currently it is presented as a problem that has been explored previously by Zhang et al and Aggarwal et al. However, after taking a look at those papers, I don’t understand where this regularization term comes from. In these papers, the regularization term (i.e. equation 2 or 3 of Zhang et al) appears independent of y. Since this term is key to the paper, it should be well explained here. E.g. at the end of section 2.2: G_\theta(y) is a CNN pretrained on what task?
-	In the inverse problem, the objective is to estimate x from y. Therefore we care about \argmin x in Equation (7). In the probabilistic setting presented in Equation (8), analogously the objective is to estimate \pi^*, which is the solution to the primal problem. The theory shows that the primal formulation in Equation (8) is equivalent to the dual formulation in Equation (16), but does not show how the dual solution yields the primal solution, which is lacking as obtaining the primal solution seems to be the point of solving the PLS problem. (Interestingly, in Section 4, the authors are using the dual solution x = G_\theta(y) as if it is the mapping given by pi(x|y)… this needs to be explained.)
-	The authors claim that Proposition 1 shows that the cyclic loss term in their dual formulation is a more general version of the cycle-consistency loss in CycleGAN. But looking closely at Proposition 1 and its proof, it seems that the equivalence holds only for specific weights, not for arbitrary weights. Additionally, the specific weights are unknown (they depend on the solution \pi^* to the primal problem…). I do not understand the claim that this is a generalization of cycle-consistency loss, nor do I see how the authors implement their version of the cyclic loss as it depends on unknown weights.
-	The connection to CycleGAN seems to hold only when p=q=1?
-	End of section 3: The authors conclude “our cost formulation using (17) with (18) and (19) is more general compared to the standard CycleGAN, since a general form of measurement data generator Hx can be used”. I don’t see the connection between the theory and this claim. Even with CycleGAN, both generators can be arbitrary or fixed if one of them is known.
-	The proofs are easy to follow, though perhaps they could be moved to the Appendix in favor of providing more motivation and explanation in the main text.
Section 4:
-	The authors motivate the problem with the PLS setup but then they use the learned regularization term x = G_\theta(y) as if it is the mapping given by pi(x|y).  I am confused by this.
-	Putting aside the connection to the PLS problem, my interpretation of the experimental setup is that the authors use CycleGAN with Wasserstein GAN loss instead of the classic discriminator loss, where one of the generators is known (and hence only one generator/discriminator pair is needed). I might be missing something, but I’m not sure that this approach is different enough from CycleGAN.
-	Considering that the authors have the ground truth, they could provide quantitative evaluation of their method against other methods, rather than showing a few qualitative results where it is working.


It would be desirable for a reinforcement learning (RL) based agent to learn behaviour by merely watching a demonstration. However, defining rewards that facilitate this goal within the RL paradigm remains a challenge. Here we address this problem with Siamese networks, trained to compute distances between observed behaviours and the agent's behaviours. Given a desired motion such Siamese networks can be used to provide a reward signal to an RL agent via the distance between the desired motion and the agent's motion. We experiment with an RNNbased comparator model that can compute distances in space and time between motion clips while training an RL policy to minimize this distance. Through experimentation, we have had also found that the inclusion of multi-task data and an additional image encoding loss helps enforce the temporal consistency. These two components appear to balance reward for matching a specific instance of a behaviour versus that behaviour in general. Furthermore, we focus here on a particularly challenging form of this problem where only a single demonstration is provided for a given task -the one-shot learning setting. We demonstrate our approach on humanoid agents in both 2D with 10 degrees of freedom (DoF) and 3D with 38 DoF.

This paper presents visual imitation with reinforcement learning (VIRL), an algorithm for learning to imitate expert trajectories based solely on visual observations, and without access to the expert’s actions.  The algorithm is similar in form to GAIL and its extensions, learning a reward function which captures the similarity between an observed behavior and the expert's demonstrations, while simultaneously using reinforcement learning to find a policy maximizing this reward, such that the learned policy will replicate the demonstrated behavior as well as possible.  A key feature of this method is that the learned reward function is defined by a learned distance metric, which evaluates the similarity between the agent's current trajectory, and the nearest demonstrated expert trajectory.
The network describing the distance metric is recurrent, such that the distance is defined between trajectories rather than individual states.  The distance function network is trained via a negative sampling approach, where expert trajectories are randomly reordered to produce examples that dissimilar to the expert trajectories.   The distance network also defines a variational autoencoder, and the reconstruction of the target trajectories is treated as an auxiliary task to help train better representations of the trajectory space.
While previous work has considered the problem of visual imitation learning, the approach taken here is novel in its architecture and loss function, and significantly outperforms the baselines in terms of the similarity between the resulting behavior and the expert behavior.
The clarity of the technical presentation could be improved, however.  In particular, it would be helpful for the reader if the definitions of the negative sampling loss and the autoencoder losses were given before the combined loss, and if we saw the form of the loss for both positive and negative sequence pairs.  Equation 4 could also be made explicit, with the full summation term included.


Neural networks powered with external memory simulate computer behaviors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures. The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus resemble the Universal Turing Machine. A wide range of experiments demonstrate that the resulting machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks.

The authors discuss an interesting idea for memory augmented neural networks: storing a subset of the weights of the network in a key-value memory. The actual weights used are chosen dynamically. This is similar to having programs/subprograms in a classical computer. To the best of our knowledge, this idea was not previously studied, and it was a missing part of the study of memory augmented neural networks until now.
In the abstract authors mention a von Neumann architecture. However, the von Neumann architecture assumes that the program and the data are stored in the same memory, which is not the case for what the authors propose (a significant limitation is that the vector size of the memory doesn’t match the number of network weights). It is more reminiscent of the Harvard architecture. Please correct!
At the end of section 3.1 and in Eq 6 and 7 the authors claim that they use the regularization loss that makes the keys different to prevent programs from collapsing. However, this is only one way of collapsing. The other way is more reminiscent of what tends to happen in routing networks, where the controller network (here P_I) learns to attend to a single module (here memory slot) or attend to all of the modules with roughly equal weight. Are such collapses experienced during this work? If not, what could be the reason?
The main difference between routing networks and NSM is that in the former the selection is done in activation space, while in NSM it happens in weight space. Thus one should expect the same exploration/exploitation, transfer/inference tradeoffs, module collapse, etc to happen [1]. What makes it better/worse compared to these methods?
The method is mainly tested on methods used to test MANNs. However, because of its relation to the routing networks, it would also be interesting to compare them to [2] or [3] to see whether they suffer from the same problems.
Because the program distribution is rarely close to one-hot (all programs are contributing to some degree), could you please provide as a baseline a version where the program distribution is fixed and uniform?
In section 3.2, the last 2 sentences of the first paragraph, in one sentence the authors say “we store both into NSM to completely encode a MANN” and in the next they say “in this paper, we only use NSM to store W^C”. Please make this consistent.
In the last sentence of the first paragraph of section 3.2, the authors say “is equivalent to the Universal Turing Machine that can simulate any one-state Turing Machine”. Could you further clarify this?
In the second paragraph of section 3.2, the authors say “P_I simulates \delta_u of the UTM”. Shouldn’t \delta_u also include the state transition function, which is not included in P_I?
Several sentences mention “direct attention,” which is described as a neural network producing the weights directly. Why call this attention at all? Isn’t this a form of fast weights? Probably they should be called fast weights or dynamic links.
How does this relate to the original work on this? Note that the first end-to-end-differentiable systems that learn by gradient descent to quickly manipulate the fast weights of another net (or themselves) were published between 1991 and 1993 - see the references [FAST0-3a] [FAST5] [FASTMETA1-3] in section 8 of http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html
The authors also claim to “When the memory keys are slowly updated, the meta-network will shift its query key generation to match the new memory keys and possibly escape from the local-minima”. Why? What drives the network to use new memory keys in this case?
In section 3.3, “P_I is a meta learner”. This is not the correct term here, because P_I is not leveraging old knowledge to learn faster (nor in improving the learning algorithm itself, nor in the recently more popular transfer learning sense), but it learns to select which weights to use.
In figure 2b, could you show the mean with std?
“states of the generating automation” should be automaton instead.
“As NUTM requires fewer training samples to converge, it generalizes better to unseen sequences that are longer than training sequences” - Why this indication of a causal relationship here? Why would requiring fewer training samples make it generalize better?
In the sequencing tasks (section 4.3), how does the network know which task to perform? Is the task label one-hot encoded and concatenated with the input? From Fig 13 and 14, it seems to be a fixed concatenation of inputs, without any indication which task is which, and then a fixed concatenation of the outputs is requested from the network. Could you clarify, please?
We found Figure 5 very confusing. At first it seemed like the X-axis is “time” (where we assumed the network is trained on a sequence of tasks C, RC, AR, PS), and the title above the subplots indicates for which task the performance is plotted over “time” (which in this case would correspond to the time steps after completion of the training phase indicated on the X-axis). However, the second subplot shows perfect RC performance after having been trained only on C. We probably misunderstood the plots: the title plot is the “time”, and the X-axis shows just the datasets (so the order is not important) - so the plots don’t show the performance on the specific dataset over the course of training as assumed initially. But if so, why are they connected by lines and why not use a bar plot? It would be more interesting to see “time” on the X-axis, so one can see how performance degrades while training on new tasks.
In Figure 5, what is the average performance supposed to show? If you average the performance over each training phase, the dataset trained first will yield better performance than the ones trained later. This is because the last trained one will have a performance near chance for most of the time it is measured, while the first one will have the best performance on the first measurement and will degrade somewhat because of catastrophic forgetting - but hopefully it will still be better than chance.
In Figure 5, numbers on the Y-axis: the first 10 and 100 place digits are misaligned.
In the last paragraph of section 4.4, “- over 4 tasks”, the - is confusing because it is not part of the formula. Please rewrite the formula in a different style or remove the -.
In section 4.5, the authors write that the best hyperparameters they found are p=2 for 5 classes and p=3 for 10. What could be the reason for p and the number of classes being so unrelated?
In section 4.6, “DNC is more powerful and thus suitable for NSM integration” - this suggests that the reason why it is suitable is that it is more powerful, but that is not the case since it was integrated into NTM, too.
In section 5, “they impede modularity”. Why? Maybe they don’t increase it, but why impede? “which looses modularity” - Why? The original RNN is not modular either. Why is this even less modular?
In figure 7 etc. it would be nice to have a title for the topmost subplot, too.
What is a dynamic n-grams task? Could you clarify or include a reference?
Could you clarify how the white-black-yellow-orange input charts should be understood (Fig 3d, Fig 15, etc)?
How to understand the preservation plots (Fig 3d)?
Additional references besides [FAST0-3a] [FAST5] [FASTMETA1-3] mentioned above:
[1] Rosenbaum et al, Routing Networks and the Challenges of Modular and Compositional Computation
[2] Chang et al, Automatically Composing Representation Transformations as a Means for Generalization
[3] Kirsch et al, Modular Networks: Learning to Decompose Neural Computation
We think this paper is quite interesting, and might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.
Edit after rebuttal: score increased to 8. However, the authors should not forget to cite the original fast weight paper by Malsburg (1981). And one more thing: in the revised version they write "More recent works implement fast-weight using outer-products," citing papers from 2016-17, but this was actually first done in the 1993 paper they already cite.


The inference of Gaussian Processes concerns the distribution of the underlying function given observed data points. GP inference based on local ranges of data points is able to capture fine-scale correlations and allow fine-grained decomposition of the computation. Following this direction, we propose a new inference model that considers the correlations and observations of the K nearest neighbors for the inference at a data point. Compared with previous works, we also eliminate the data ordering prerequisite to simplify the inference process. Additionally, the inference task is decomposed to small subtasks with several technique innovations, making our model well suits the stochastic optimization. Since the decomposed small subtasks have the same structure, we further speed up the inference procedure with amortized inference. Our model runs efficiently and achieves good performances on several benchmark tasks.

PAPER SUMMARY:
This paper proposes a fast inference method for Gaussian processes (GPs) that imposes a sparse decomposition on the VI approximation of the posterior GP (for computational efficiency) using the KNN set of each data point. This is further coupled with armortized inference for better scalability.
NOVELTY & SIGNIFICANCE:
This paper adopts a different approach of characterizing the VI approximation of a GP posterior than original VI approximation that was developed in Titsias (2009): Instead of characterizing the surrogate q(f_I) of p(f_I | Y) for a small collection of inducing inputs, the proposed method characterize q(f) directly where q(f) = int_f_I q(f_I) p(f | f_I)df_I.
This is, however, a somewhat strange direction which, to me, seems to raise extra issues that could have been avoided if one follows the conventional VI approximation:
(1) As the posterior surrogate is now directly over f instead of f_I, the number of variational parameters is now proportional to the data size which requires several (redundant) extra approximations including armortized inference & the lower-bound on the entropy term that admits a sparse decomposition.
(2) This at least creates the armortized and entropy gaps that decrease the expressiveness of the original VI. While I understand that this is in exchange for the ability to encode local information (via KNN) within the surrogate posterior, it is not clear to me why do we need to incur all these computational issues to incorporate such local information.
(3) For example, instead of forcing such local information in the posterior surrogate q(f), we could alternatively let it be reflected in the test conditional p(f_* | f_I, Y_n(*)) such that the test output depends on both the inducing output and a local partition of data (e.g., via K-mean), which has been previously explored in [*] and later incorporated in the conventional VI paradigm of Titsias (2009) without incurring extra intractability [**].
(4) This maintains the dense correlation between data points within the same neighborhood while allowing the VI surrogate to be more concisely specified and independent of the no. of training data points. Furthermore, it also helps avoid the data-bound overhead of computing a KNN per test point.
[*] Local and global sparse Gaussian process approximations (AISTAT-07)
[**] A distributed variational inference framework for unifying parallel sparse gaussian process regression models (ICML-16)
To summarize, the practical significance of placing such a VI approximation directly on q(f) to encode such (indirectional) local information is, given the above, questionable to me.
Please note that I am not disputing the potential use of this VI form here, which could have been the only way to encode a different (directional) type of information.
For encoding KNN information, however, it only seems to create more troubles than it solves.
Minor point:
The above references, especially [*], should have been cited.
TECHNICAL SOUNDNESS:
[A] Optimization of the ELBO:
(1) The ordering of data (i.e., the directional information) was mentioned repeatedly in the paper but its importance to the fast approximation was neither explained nor discussed.
(2) The decomposition form of Eq. (6) also raises a question: How do we know that the term inside the log is positive? There seems to be missing information on the constraint of R.
[B] Amortized Inference:
(1) The choice of the GCN seems arbitrary here. I am in fact not sure why GCN is necessary for the inference network & furthermore, GCN also brings to the table another heuristic choice of A.
(2) How do we set the adjacency graph A?
(3) How do we know what is the right complexity for the GCN?
[C] Complexity:
The complexity analysis is too informal and lacking fine-grained information.
Please include a detailed complexity analysis of the training and inference cost in terms of the input dimension, the no. of data points, the size of the neighborhood and the batch size.
It is also necessary to factor in the KNN overhead (e.g., the cost of building the K-D tree for low-dimensional embedding of data & the approximation cost of projecting that information to high-dimensional data)
EXPERIMENT:
The experiment results only show marginal improvement over the baselines, and the size of the dataset for regression is too small. If I read correctly, both have fewer than 20000 data points.
SVGP in particular has been tested on a much larger datasets (AIRLINE, UK Housing) comprising millions of data points -- comparison on such dataset should have been reported.
Note that the largest dataset used to evaluate the efficiency of fast approximation of GP is on the scale of 6M data points [****]. On that note, eBird and precipitation should not even be considered mid-sized.
To demonstrate the efficiency of local information encoding, comparison with [*] should be reported. There is another class of inducing-point methods that use expectation propagation
that should have been discussed and/or compared with:
[***] A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (JMLR-18)
[****] Distributed Gaussian Processes (ICML-15)
CLARITY:
The paper is clearly written.
REVIEW SUMMARY:
This paper adopts a VI approximation that deviates from the conventional form of (Titsias, 2009) to encode the KNN information, which causes extra computational issues (that incurs extra approximations). I find this deviation redundant seeing that the same information could have also been accounted for using the old VI form, which is a lot more computational efficient. I also find the experiment lacking as comparison with fast approximation method such as [*] that incorporate local information is not included. There are also a few technical ambiguities that need to be clarified.
------ Post-Rebuttal Update ------
Thank you for the rebuttal & I have read it in detail. However, it still does not address my concern, which I re-summarize here:
I do not dispute the beneficial of exploiting neighborhood information but my point is we could still leverage the same amount of neighborhood information without going through the trouble of incurring extra steps of approximation due to approximating q(f) instead of q(f_I) -- I think I have elaborated this in points (1) - (4) in my original review -- which also creates the amortized gap. I am also not sure what model reuse means (in this context) and whether it is relevant since it appears somewhat noncentric to the objective of this paper.
Also, apparently, the experiment is still lacking since to me, comparing with [*] is important in substantiating the contribution claim of this paper.


In this work, we consider a new problem of training deep neural network on partially labeled data with label noise. As far as we know, there have been very few efforts to tackle such problems. We present a novel end-to-end deep generative pipeline for improving classifier performance when dealing with such data problems. We call it Uncertainty Mining Net (UMN). During the training stage, we utilize all the available data (labeled and unlabeled) to train the classifier via a semi-supervised generative framework. During training, UMN estimates the uncertainly of the labels' to focus on clean data for learning. More precisely, UMN applies the samplewise label uncertainty estimation scheme. Extensive experiments and comparisons against state-of-the-art methods on several popular benchmark datasets demonstrate that UMN can reduce the effects of label noise and significantly improve classifier performance.

The paper describes a method for learning in semi-supervised settings with label noise. This is an interesting topic with a relatively scarce literature. The proposed method works by first postulating a generative model for the labelled/unlabeled/label-corrupted data. The model is then fitted using a standard variational lower bound maximization.
The approach is elegant, and appears to work empirically well. Unfortunately, I do not seem to really understand the rationale behind the main novelty (even at an heuristic level) of the paper which is contained in Section 3.2.1, namely Equation (10) which describes the intensity of the label corruption. Why is it a sensible idea? There is only 3 lines of comments after equation 10, which seems a bit short since this is the crux of the paper. Also, the notations of equation (10) are very confusing (to me) since the function f(.) was used before to denote another quantity.
Minor remarks:
=============
1. page4 was a bit difficult to digest at first reading since the author did not describe in the main text he form (i.e. factorization structure) of the variational distribution q.
2. second line of Equation 14 seems to be wrong, while the 1st and last line seems correct -- the authors may want to double check
3. the term "labeled loss term" was not properly defined, which makes the reading a bit difficult even if one eventually gets what the authors mean
4. In equation (6) I do not understand why f(eps) = log[(C-1)(1-eps)/eps]. Why isnt it equal to log[p(hat(y)|y)] ?
5. notations z_a, z_1, etc.. do not seem to be consistent throughout the text
6. it is quite surprising to me that, in the low data-corruption regime, the method appears to be competitive/better with consistency-based method such as the MT approach.
7. naive question: why directly modeling \eps as a function of z_a or (z_a, y), possibly parametrized by a neural network, a bad idea?
In conclusion, it feels like the method has a lot of potential (in views of the numerical simulations) -- if the authors could clarify the exposition, this could be a very good contribution to the field. The method appears to be conceptually simple (i.e. postulate a generative model + fit it by maximizing the ELBO + one trick to estimate \eps), which is a good thing -- what is missing, I think, is a real discussion of why the proposed manner to estimate \epsilon is sensible.
Edit after reading [1]
======================
The proposed generative model is same -- the authors should make this very clear in the paper. Although is is acknowledged: "In this work, we implement the idea in (Langevin et al., 2018)" -- this is only in section "4.1 IMPLEMENTATION DETAILS". After reading (1), it is clear that the novelty of the paper us much less than what I had initially thought. The only difference is in Section 3.2.1, and this Section is far from being satisfying.
[1] "A Deep Generative Model for Semi-Supervised Classification with Noisy Labels", Langevin & al


Bayesian neural networks (BNNs) introduce uncertainty estimation to deep networks by performing Bayesian inference on network weights. However, such models bring the challenges of inference, and further BNNs with weight uncertainty rarely achieve superior performance to standard models. In this paper, we investigate a new line of Bayesian deep learning by performing Bayesian reasoning on the structure of deep neural networks. Drawing inspiration from the neural architecture search, we define the network structure as gating weights on the redundant operations between computational nodes, and apply stochastic variational inference techniques to learn the structure distributions of networks. Empirically, the proposed method substantially surpasses the advanced deep neural networks across a range of classification and segmentation tasks. More importantly, our approach also preserves benefits of Bayesian principles, producing improved uncertainty estimation than the strong baselines including MC dropout and variational BNNs algorithms (e.g. noisy EK-FAC).

This paper proposes to do approximate Bayesian inference in neural networks by treating the neural network structure as a random variable (RV), while inferring the parameters with point estimates.
While performing Bayesian inference for the neural network structure is sensible, I am not convinced by the approach taken in this work.
The biggest problem is that the model uses a point estimate of the same weights for different, random network structures.
Major problems:
- In the motivation the authors write “DBSN places distributions on the network structure, introducing more global randomness, thus is probable to yield more diverse predictions, and ensembling them brings more calibrated uncertainty”. What is “more global randomness”? This is used multiple times. Does it refer to the hierarchy in the graphical model? Please be precise here and point that out in your model by using an equation or graphical model. Or is it just an intuition?
- Generally, I would agree that integrating out multiple network structures provides better calibrated uncertainty. However, given that the authors use point estimates for the weights, it is not clear if that is still true, especially since the number of different architectures used in practice is small.
- What’s more, the approach uses *the same* point estimates for different structures. This leads to a graphical model, where the weights are not conditioned on the architecture/structure. This modeling choice could be a big limitation, because the weights now have to fit multiple different architectures; it may thus defeat the calibration completely. One can easily imagine that only a single random architecture works well with the learned point estimates, thus resulting in an (almost) deterministic model. I assume that this modeling choice was made for practical reasons, but could you expand on its implications / interpretation / limitations? Does the posterior of such a constrained model not quickly converge to an “almost” dirac, effectively just one network structure?
- Sec. 3.1. presents the above problem resulting from a modeling decision as a “training challenge”. To counter this problem, the authors propose to reduce the variance of the structure distribution. By doing so, the approach becomes even less Bayesian and the predictive uncertainty becomes even less reliable.
- “We only learn the connections between the B internal nodes, as shown in Appendix D”. All deterministic weights are learned, but the structure only for some parts of the model? If this is the case, then approach becomes again less probabilistic.
Regarding the experiments, the stddevs are calculated from 3(!) independent runs and thus completely misleading (imagine the stddev of the stddev estimate).
In summary, the model choice of point estimates for the weights, which are not conditioned on the architecture, leads to various problems. The authors have to introduce tricks such as reducing the variance of the random network structures or learning only a part of the whole structure to make the approach converge. The resulting probabilistic model and its predictive uncertainty is questionable. For this reason, this paper should be rejected.
Minor problems
- Sec. 3.2. “Improvements of the structure learning space”. What is a “structure learning space”?
- Section 3 introduces the ELBO in Eq. (4) before the complete model is specified. Please specify the whole model first. How do w and alpha depend on each other in your model?
- The prior for the weights is omitted; at the same time it is mentioned in the experiments (Sec.5.1.) that weight decay is applied. Why not just be explicit about it and say that a Gaussian prior is used?
- Background Sec. 2.2. is not clear.  what is a cell? some deterministic transformation in general? bunch of neural network layers? What are the operation (last term in Eq. (2)) doing? This is not detailed and abstract to me. Are the alphas probabilities? Is Eq. (2) consequently a mixture model of different architectures? Or is this here just a weighted sum, where the weights take arbitrary values? A small visualization (additionally) might help here, but can probably be rectified by better explanation.
- Bayesian reasoning on the structure. Inference?
- Writing that you propose a new “framework” is a bit grandiose for what is actually proposed. There has been previous work in which the architecture is inferred as well and these approaches would certainly be part of the same framework. Please just say model/algorithm/approach, whatever is applicable.
- new paragraph starting at “To empirically validate” in the intro.
- Before (4): “Then we rewrite the approximation error”. Eq. (4) is the ELBO, this is not an approximation error.


Many machine learning algorithms are trained and evaluated by splitting data from a single source into training and test sets. While such focus on in-distribution learning scenarios has led interesting advances, it has not been able to tell if models are relying on dataset biases as shortcuts for successful prediction (e.g., using snow cues for recognising snowmobiles). Such biased models fail to generalise when the bias shifts to a different class. The cross-bias generalisation problem has been addressed by de-biasing training data through augmentation or re-sampling, which are often prohibitive due to the data collection cost (e.g., collecting images of a snowmobile on a desert) and the difficulty of quantifying or expressing biases in the first place. In this work, we propose a novel framework to train a de-biased representation by encouraging it to be different from a set of representations that are biased by design. This tactic is feasible in many scenarios where it is much easier to define a set of biased representations than to define and quantify bias. Our experiments and analyses show that our method discourages models from taking bias shortcuts, resulting in improved performances on de-biased test data.

###  Summary
The paper proposes a method for regularizing neural networks to mitigate certain known biases from the representations learned by CNNs. The authors look at the setting in which the distribution of biases in the train and test set remains the same, but the distribution of targets given biases changes.
To remove the bias, they propose hand-designing a set of models that rely exclusively on these biases to make predictions. They then train a new unbiased model by making sure that this new model uses sufficiently different information than the biased models to make predictions (By adding a regularization term)
### Decision and reasons
I vote for a weak accept.
Strengths:
1: The paper is well written with a clearly defined problem-setting. The proposed method is sound and interesting, and the empirical results, thorough.
Weaknesses:
2: The solution just pushes the problem of 'learning a model that only uses the true signal to make predictions' to 'learning a set of models that only use the noise to make predictions.' It's not clear why the latter is easier than the former.
3: The paper does not have any baselines that directly try to remove the bias (Instead of using the two-step process). As a result, it's hard to judge how meaningful the improvements are.
### Supporting arguments for the reasons for the decision.
Strengths:
1: The paper does a really good job of defining the problem-setting. Contrasting cross-bias with cross-domain and in-distribution makes the goal of the paper very clear. The notation used to formalize the problem setting in Section 2.1 is also clear and concise. Moreover, the experiments on the toy dataset help clarify the proposed solution whereas experiments on Biased MNIST and Imagenet show that it successfully mitigates the bias. Finally, the authors show the importance of each component of the proposed solution by factor analysis.
Weaknesses:
2: In the most general case, it is not obvious why it is easier to define and learn a set of models that only use noise to make predictions (which is the required first step for their proposed solution) as opposed to learning a model that only uses signal (which is the goal of the problem). These two problems seem equally hard. The paper builds on the premise that in some cases former is easier (i.e. in some cases, it is easier to learn a set of models that use only noise as opposed to learning a debiased model directly). The authors give two such examples (They only explore the first experimentally.)
1. Learning a model that relies on local texture. They achieve this by limiting the receptive field of the features.
2. Learning a model that relies on static images to make predictions about actions in videos.
I feel that the two given examples are very narrow. It would be nice if the authors could identify a broader class of problems for which G is given or can easily be defined. Moreover, even in these two examples, I'm not convinced that the proposed biased models only use B for making predictions. For example, for some classification tasks, the local texture could be part of the signal and not just the bias. Similarly, static images from a video do contain important information for making the prediction.
3: The authors only compare their method to a baseline that does nothing to debias the representations. Even though this is an important comparison (as it shows that the proposed method can debias the representations), it does not tell the reader how effective the proposed method is compared to other possible solutions. The results would be more meaningful if the authors could include at-least a simple baseline that tries to remove the bias in other ways (For example, they could use the style-transfer baseline used by Geirhos et al., 2019).
I vote for accepting the paper as a poster. It introduces an interesting approach for debiasing representations. However, due to its narrow scope and missing baselines, I would not recommend the paper for an oral presentation.
### Questions
1. What are some broader class of problems for which defining G is easier than directly regularizing for debiased representations?
2. How well do Bagnets alone perform on the benchmarks in Table 3? I would expect to see that Bagnets alone do worse than vanilla Resnets on Unbiased and IN-A. Is that so?
3. How well do other methods do in these domains? (Such as methods that directly debias the training data against texture by applying style-transfer).
### Update after Author's response
The author's response has clarified the motivation behind the proposed approach to an extent. They have also added a comparison with a method that directly promotes learning the shape as opposed to the texture ( by training on stylized Imagenet)
I agree with R1 on all accounts (i.e. it is very hard to define the family of biased feature extractors, the proposed approach is ad-hoc, the authors need to compare to texture-shape disentanglement methods, etc), however at the same time, I can see that proposed approach can act as a useful heuristic for regularizing neural networks to pay attention to certain kind of information.
An interesting use-case of the proposed method (which the authors indirectly mentioned in their response to my review) is in a multi-modal setting. It's not trivial to enforce deep learning systems to utilize all data modalities in a multi-modal setting. By defining G to be models trained on individual modalities, it would be possible to nudge our models to pay attention to the information in all modalities.
Some important results in deep learning have been ad-hoc (For example skip connections in deep networks, ReLUs) and have nonetheless progressed the field. This work is not as widely applicable as skip connections or ReLUs, but it is, nonetheless, providing a heuristic for solving an important problem.


Deep learning models are often sensitive to adversarial attacks, where carefully-designed input samples can cause the system to produce incorrect decisions. Here we focus on the problem of detecting attacks, rather than robust classification, since detecting that an attack occurs may be even more important than avoiding misclassification. We build on advances in explainability, where activity-map-like explanations are used to justify and validate decisions, by highlighting features that are involved with a classification decision. The key observation is that it is hard to create explanations for incorrect decisions. We propose EXAID, a novel attack-detection approach, which uses model explainability to identify images whose explanations are inconsistent with the predicted class. Specifically, we use SHAP, which uses Shapley values in the space of the input image, to identify which input features contribute to a class decision. Interestingly, this approach does not require to modify the attacked model, and it can be applied without modelling a specific attack. It can therefore be applied successfully to detect unfamiliar attacks, that were unknown at the time the detection model was designed. We evaluate EXAID on two benchmark datasets CIFAR-10 and SVHN, and against three leading attack techniques, FGSM, PGD and C&W. We find that EXAID improves over the SoTA detection methods by a large margin across a wide range of noise levels, improving detection from ∼ 70% to over 90% for small perturbations.

A Simple method to detect adversarial examples, but needs more work.
#Summary:
The paper proposed a method that utilizes the model’s explainability to detect adversarial images whose explanations that are not consistent with the predicted class.  The explainability is generated by SHAP, which uses Shapley values to identify relative contributions of each input to a class decision. It designs two detection methods: EXAID familiar, which is aimed to detect the known attacks and EXAID unknown, which is against unknown attacks. Both of the two methods are evaluated on perturbed test data which are generated by FGSM, PGD and CW attack with perturbations of different magnitudes. Qualitative results also show that the proposed method can effectively detect adversaries, especially when the perturbation is relatively small.
#Strength
The method is easy to implement and using the idea of interpretation for detecting adversarial examples seems interesting.
Good results are demonstrated compared with other comparators.
#Weakness
The idea of this paper is based on the interpretation method of DNN. However, it has been shown that these interpretation methods are not reliable and easy to be manipulated [1][2]. Therefore, although the method is simple to design, it also brings other security concerns.
Unfortunately, the paper does not address these issues. In addition, the comparators listed in the experiments are not state-of-art or common baselines. It is either not clear why authors modified the existing method and develop their own “unsupervised” version.
In the experiments, many details are omitted. For example, how is the “noise level” defined? Are they based on L1, L2 or L-inf perturbation? For PGD attack, how many iterations does the generation run and what is the step size? How many effective adversarial examples are generated for training and testing? And all the experiments are conducted in a relatively small dataset, it is also suggested to do experiments on large datasets, e.g. Imagenet.
In the evaluation part, it looks strange to me why the EXAID familiar performs worse than EXAID unknown in evaluating FGSM attack on SVHN since the EXAID familiar is trained using FGSM attack.
#Presentation
I think the authors used a wrong template to generate the article. The font looks strange and the headnote indicates it is prepared for ICLR2020. The paper contains many typos and even the title contains a misspelling. Poor coverage of citations. There are more works for detecting adversarial examples that are published, e.g. [3][4][5]. On the other hand, the paper does not have the literature review for work related to the model interpretation.
Overall, I think the paper is not good enough for publication at ICLR.
[1] Dombrowski, Ann-Kathrin, et al. "Explanations can be manipulated and geometry is to blame." arXiv preprint arXiv:1906.07983 (2019).
[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. "Interpretation of neural networks is fragile." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
[3] Meng, Dongyu, and Hao Chen. "Magnet: a two-pronged defense against adversarial examples." In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147. ACM, 2017.
[4] Liao, Fangzhou, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. "Defense against adversarial attacks using high-level representation guided denoiser." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787. 2018.
[5] Ma, Shiqing, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. "NIC: Detecting Adversarial Samples with Neural Network Invariant Checking." In NDSS. 2019.


Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman (Elman, 1990), we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models at the time of model submission), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

This paper proposes to use additional structures within and between sentences for pre-training BERT. The basic idea is to shuffle either some n-grams within sentences or the sentences in texts, then train the model to predict the correct orders. Experiments in this work show that, with this additional training objective, the proposed pre-trained model, StructBERT, obtains good performance on the tasks including natural language understanding and question answering.
Overall, I think the experiments and results in this work are not sufficient enough to support the claim:
- It is necessary to show the performance of BERT only trained with the proposed word and sentence objectives. Otherwise, it is not clear how much benefit the model can get from them and the work is basically incremental.
- Some justification is needed about why choosing trigrams and why 5% is a good number of sampling trigrams from texts
Besides, there are some recent work on analyzing why BERT encodes any linguistic properties of texts, for example
- Goldberg. Assessing BERT's syntactic abilities. 2019
- Tenny et al. BERT Rediscovers the Classical NLP Pipeline. ACL 2019
- Tenny et al. What do you learn from context? ICLR 2019
All of them show positive results on BERT can capture some syntactic information from text automatically. Which makes me wonder why the simple additional training objective proposed in this work can still lead to performance improvement. Is there an explanation?


Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. In experiments on two datasets, restorative Brownian motion and a hand-drawn sketch dataset, we find that RNNs are sub-optimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. We show how constraining past information by injecting noise into the hidden state can improve the ability of RNNs to extract predictive information for both maximum likelihood and contrastive loss training.

This paper certainly poses an interesting question: How well do RNNs compress the past while retaining relevant information about the future. In order to quantitatively answer this question, the authors suggest to look at (the optimal solutions of) the Information Bottleneck Lagrangian (IBL). The investigated RNNs need to solve the task of next-step prediction, which can be used to evaluate the IBL. In the paper, the (deterministic) hidden state h is transformed through simple additive Gaussian noise into a stochastic representation which then is utilized to compute the IBL. In general the IBL is not tractable and hence the paper uses approximate computations.
I definitely like the underlying question of the paper. Yet, to me it seems not ready for publication. For one, the presented experimental results look interesting but the suggested method for improvement through adding noise to the latents (during training) is too much of handwaving for such a fundamental problem. Second, shouldn't the results on the BHC be quite surprising in terms of LSTMs performance? Why is that, usually LSTM (or GRU) deliver excellent performance in typical (supervised) sequential tasks. Third, the task itself seems not well described, it seems to be next-step prediction, but how are more future predictions generated -- these seem to be not considered in the equation, but probably should when talking about 'retaining relevant information for the future'? Fourth, recently some researchers started to question whether 'reconstruction' is a good idea in order to learn generative-like models, for example you cite van den Oord 2018. How would such models perform in your metric.
A final remark with respect to your citation for eq. 4, I think you meant Barber, Agakov, "The IM algorithm...", 2003?


Deep CNNs have achieved state-of-the-art performance on numerous machine learning and computer vision tasks in recent years, but as they have become increasingly deep, the number of parameters they use has also increased, making them hard to deploy in memory-constrained environments and difficult to interpret. Machine learning theory implies that such networks are highly overparameterised and that it should be possible to reduce their size without sacrificing accuracy, and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this. In this paper, we take a further step in this direction by proposing a filter-sharing approach that reformulates deep, complex CNNs as an iterative application of shallower modules (a single convolutional mapping in the simplest case). We show, via experiments on CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet that this allows us to reduce the parameter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional to their depth, whilst leaving their accuracy largely unaffected. At a broader level, our approach represents a way of rethinking neural network architectures so as to leverage the scale-space regularities found in visual signals, resulting in models that are both parsimonious and easier to interpret.

This paper proposes to modify a standard CNN by requiring all of its layers to share the same filter set, essentially allowing it to be expressed as an iterative (or recurrent) network.  This also has the effect of forcing the same number of feature channels to be used throughout the network.  For ResNet-like architectures with bottleneck blocks, sharing occurs at the level of the block (3 conv layers in series that are repeated).  Another variant of the sharing pattern inserts unshared 1x1 convolutional layers after shared layers or blocks; this adds some flexibility while still reducing parameters compared to standard CNNs.
On CIFAR-10, CIFAR-100, and Tiny ImageNet, experiments demonstrate the ability of the sharing scheme to reduce parameters without impacting accuracy (or more drastically reduce parameters at the cost of accuracy) (Tables 1ab, 2a).
However, results are less compelling on ImageNet (Table 2b), where SL-ResNet-50 and SL-ResNet-34 are both less accurate than the baseline standard ResNets as well as ShaResNet [Boulch, 2018].  The accuracy gap between SL-ResNet and ResNet on ImageNet (Table 2b) is significant (approx 5% Top-1 and 2% Top-5 accuracy) and might make it difficult to justify use of the proposed method in this setting.  As ImageNet is the most challenging of the datasets used, this is cause for concern.
There is also a major concern with respect to novelty and related work.  Unfortunately, the paper appears to have completely missed the following highly related publication from ICLR 2019:
Learning Implicitly Recurrent CNNs Through Parameter Sharing
Pedro Savarese, Michael Maire
ICLR 2019
This prior work proposes a network structure in which a set of L layers share a set of k parameter templates.  The templates and sharing coefficients are learned as part of the standard training procedure.  This prior work demonstrates both parameter savings and accuracy improvements when training networks in this manner.  Additionally, this prior work shows that some learned networks can be converted into explicitly recurrent forms as a post-processing step.
The paper under review appears be a special case of this prior work with the number of templates k = 1 (shared between all layers).  It is possible this is an important special case, worthy of significant attention on its own.  Notably, [Savarese and Maire, 2019] considered sharing across at most all layers within the same stage of a residual network, rather than all layers in the network.  However, arguing for the importance of this special case would require focused experimental comparison and analysis, which is not present in the current version of the paper.
Novelty is clearly limited in light of this overlooked prior work.  At minimum, citation, discussion, and experimental comparison to the above ICLR 2019 paper is necessary.


We propose a computationally efficient wrapper feature selection method -called Autoencoder and Model Based Elimination of features using Relevance and Redundancy scores (AMBER) -that uses a single ranker model along with autoencoders to perform greedy backward elimination of features. The ranker model is used to prioritize the removal of features that are not critical to the classification task, while the autoencoders are used to prioritize the elimination of correlated features. We demonstrate the superior feature selection ability of AMBER on 4 well known datasets corresponding to different domain applications via comparing the accuracies with other computationally efficient state-of-the-art feature selection techniques. Interestingly, we find that the ranker model that is used for feature selection does not necessarily have to be the same as the final classifier that is trained on the selected features. Finally, we hypothesize that overfitting the ranker model on the training set facilitates the selection of more salient features.

In this paper, the authors present an iterative approach for feature selection which selects features based both on the relevance and redundancy of each feature. The relevance of each feature is determined using a mild variant of the Feature Quality Index; essentially, the relevance is computed as the loss in model performance when setting each feature value to the mean and measuring the change in performance. Similarly, the redundancy of each feature is determined by comparing the reconstruction loss of an autoencoder when setting the feature value to its mean for all training samples. These two values are combined to give a single score for each feature at each iteration. The feature with the worst value is removed. A limited set of experiments suggests the proposed approach mildly outperforms other efficient feature selection methods.
Major comments
The paper does not include relevant, recent work on using autoencoders for feature selection, such as [Han et al., ICASSP 2018; Balın et al., ICML 2019], among others. Thus, it is difficult to discern how this paper either theoretically or empirically advances the state of the art.
I found the proposed approach to efficient feature selection reasonable. However, there is no theoretical justification for the approach. Thus, I would expect a thorough empirical analysis. Only a few limited experiments on toy datasets (and one slightly more challenging one) are given.
The paper is not well-written. For example, it seems as though the proposed approach is not applicable to datasets with categorical features. It is not obvious (and, presumably, would need to be shown empirically) if the mode could be used to replace categorical values analogously to how the mean is used for real-valued features. Alternatively, one could imagine one-hot encoding the categorical variables and grouping them in some manner similar to that used for the RadioML pairs (since the one-hot values are obviously highly correlated). However, the authors do not address these issues.
Similarly, the entire discussion in Section 3 seems to assume the ranker model will be some sort of neural network. However, as far as I can tell, the ranker model is treated as a black box, so it could easily be some random forest model, etc. If there are some implicit assumptions that the ranker model is a neural network, this should be made explicit; if not, the discussion should be revised (and, of course, non-neural models should be used in the experiments).
The approach seems to heavily depend on the ability of the autoencoder to reconstruct the input; however, it is unclear how the structure/capacity of the autoencoder affects the performance of the algorithm. For example, the authors propose a relatively simple structure, presumably to maintain computational efficiency. It would be interesting to explore more deeply how autoencoders with more capacity impact the results.
It is unclear why the autoencoder is retrained at each step compared to just setting the removed feature values to the respective means, as is done with the ranker model.
Clearly, the relevance and redundancy scores could be weighted unequally when selecting the feature to remove. It would be interesting to explore how different combinations affect the results.
It seems that the experiments only consider backward feature selection approaches. Including forward feature selection approaches would add useful context for how the proposed approach compares to other strategies.
Minor comments
The cross-validation scheme used is not clear. While the authors mention that three runs are used to estimate performance variance, they do not describe if this is 3-fold cross validation, some Monte Carlo cross validation, or if the same splits are used all three times and just the random seeds are different.
While methods like RFE have significantly higher computational cost than the methods considered here, it would be helpful to include it for at least one of the datasets to provide context on how much the less costly methods “lose”.
What is the overlap in the selected features? both among the different methods and among the different folds for the same method.
How were the hyperparameters for the various models chosen?
Typos, etc.
The references are not consistently formatted.
The Section 2 headers all have an unnecessary “0” in them (e.g., “2.0.1”).
Table 1 should include the standard deviations.


We study covariate-shifted data augmentation where the augmented targets are drawn from the true predictive distribution but the inputs are shifted. Empirically, some forms of data augmentation such as adversarial training improve robustness, but increase test error. We provide precise conditions under which data augmentation can increase test error for minimum norm interpolation estimators in linear regression. As a fix, we propose X-regularization which uses unlabeled data to regularize the parameters towards the non-augmented estimate. We prove that augmentation with X-regularization never increases test error in linear regression. Empirically, X-regularization consistently improves both robustness and standard test error across different adversarial training algorithms and perturbations on CIFAR-10.

This paper studies data augmentation in the regime where labels for the augmented datapoints are known. Special emphasis is put on the study of overparametrised linear models with minimum Euclidean norm of the regression weights (a.k.a. ridgeless regression). The results of this study are then used to motivate their “X-regularization” method, a semi-supervised learning algorithm which they test on the task of improving accuracy of adversarially trained models. The authors report improvement in accuracy of adversarially trained classifiers on CIFAR-10 when X-regularization is applied.
While I think the paper could be potentially interesting to the ICLR community, I am currently leaning towards rejection for the following two reasons: (i) I am confused by significant chunks of the theoretical derivations;  (ii) the paper is hard  follow at several places, seems to be hastily written, and not well placed in the existing literature. Finally, I would like to state that since the paper is longer than 8 pages, I am holding it to a higher standard than an 8 page one as instructed by the guidelines.
Major comments:
- I am very confused by the way you compute the (uncentred) sample covariance for the augmented dataset. On the penultimate line before eq.5, you state that Sigma_data = 1 / n X^T X if I interpret n as the total number of rows of X (which is IMHO correct). But then a couple of lines below you say that Sigma_aug = 1 / n (X_std^T X_std + alpha X_ext^T X_ext) which by the above logic should have been Sigma_aug = 1 / ((1 + alpha) n) (X_std^T X_std + X_ext^T X_ext) which is different from your definition. Inspecting appendix A.1, you also use Sigma_data = Sigma_std + alpha Sigma_ext whereas the above would suggest Sigma_data = 1 / ((1 + alpha)n) (n Sigma_data + alpha n Sigma_ext) = 1 / (1 + alpha) (Sigma_data + alpha Sigma_ext)?! Can you please clarify? Note that I did not go through the rest of the proof in the appendix.
- On a related note, I am also rather confused by your example in sect.3.1.1. In particular, if you observe e_1, e_2, and e_3, and you are in the scenario where you can get the true labels for the X_ext data points + there is no noise on y (sigma = 0), then you will get theta_aug = (X^T X)^dagger X^T X theta* = Id^(-1) Id theta* = theta* (where Id stands for the identity matrix), i.e., theta_aug = theta* and so there will be zero generalisation error since y = x^T theta* is the correct data generating process in this scenario. However, in the first display on p.4, you claim the bias (and thus generalisation error) of theta_aug is non-zero. The rest of the section similarly seems to contradict the theta_aug = theta*. Can you please clarify?
- Since the rest of sect.3 also uses the Sigma_aug = Sigma_data + alpha Sigma_ext instead of scaling this expression by Sigma_aug = 1 / (1 + alpha) (Sigma_data + alpha Sigma_ext), I will withhold my judgement until you could please clarify this confusion.
- Can you please point me to where I can find more detail about fig.1? In particular, how was the right plot produced (incl. how many repetitions of the experiment were executed)? Also, is it true that the green points in the middle plot were not chosen at random? If so, this should be clarified in the caption and the text.
- Since all of your non-toy experiments are focused on adversarial learning, it might benefit the paper to advertise this focus more clearly from page one. Furthermore, have you tested X-regularization with non-adversarial data augmentation (on CIFAR-10 or other standard benchmark) please?
- While you briefly mention the existence of alternatives to your algorithm, you don’t include any of these in your experiments section. I would suggest including some benchmark results (even if this only means restating numbers from other papers with citation).
Minor comments:
- In the abstract, you claim about X-regularization: “We prove that our new estimator never increases test error.” while sect.4 you say that this guarantee only holds for linear models. I think this should be clarified in the abstract.
- p.1 in par.1, you say “but in practice, we often work with models which can fit the augmented training data perfectly” (similar statements also appear throughout the paper, e.g., the last sentence of par.3 on p.1). I am confused by this statement because many augmentation schemes have full support in the input space, i.e., you will eventually have to be able to classify correctly any point in the input space. Are you assuming that the conditional distribution of y given x is almost surely deterministic, and using some universal approximation theorem for neural networks (even those tend to have some restrictions on the y = f(x) function!)? Please clarify.
- p.1 in par.3, “... may increase increase error …” -> “may increase error”
- Second to last paragraph on p.2, “Y = X theta + sigma N(0, I)” is mixing random variables and distributions. Please consider replacing “N(0, I)” with “epsilon, epsilon ~ N(0, I)“. Also, comparing with eq.1, did you mean “sigma^2” in eq.1?
- Last paragraph on p.2, “compare the performance two estimators” -> “ compare the performance of two estimators”.
- Top of p.2 in eq.3: the definition of the augmented estimator is somewhat confusing. If the augmentation causes the total number of points (i.e., n + \alpha n) to be higher than d (the number of parameters), then the set on the right hand side can be empty (and will be unless there is some perfect multicollinearity). Can you please clarify how is the augmented estimator defined in that case?
- Please reconsider use of contractions (e.g., “doesn’t” -> “does not” in par.2 of sect.3) throughout the paper.
- In eq.5, in the expression for bias, I think you are missing star superscript for the 2nd theta.
- Under eq.5, “... we say that augmentation is safe if the predictive risk or bias does not increase, and hurtful if it does.” Can you please clarify whether the use of “or” is in the strictly logical sense, i.e., arbitrarily large increase in variance is fine as long as bias does not increase (in which case the second part of the statement and thus the full statement is still true)?
- Beginning of “Large Sample Regime” paragraph of sect.3, you claim  “Common intuition from a statistical standpoint would suggest that more data is always good.” I am not sure I agree: there is a plethora of literature on data poisoning, robustness, differential privacy, adversarial examples, etc., which shows that adding data can significantly worsen performance of models. Even without reading these papers, it seems quite natural that adding **arbitrary** data into the dataset may not be the best idea (even in the large sample regime).
- Unresolved equation reference on p.11, 2nd line. Also, in the very next sentence, “yields zero and variance …” -> “yields zero bias and variance ...“?! Broken references also on p.14 (twice).


For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.

This paper describes an approach to learn word embedding functions more efficiently and with fewer parameters. This is done by replacing the embedding lookup function which is typical in NLP tasks such as language modeling and machine translation with a hierarchical embedding model. This allows for a low dimensional embedding layer, reducing total parameters and training time. A novel skip-connections architecture is introduced as a part of the "embedding generation model". Experiments are conducted for language modeling and machine translation tasks and performance improvements are observed with a reduction in parameters and lesser training time.
The direction of this work is nice, the problem that is being tackled is indeed important.
The obtained results are nice (though this can be improved) and there is indeed some potential value in this work.
However, I have the following concern. The paper completely ignores a lot of previous and concurrent work in reducing the size of the embedding layer. These works are in most cases not even cited and no empirical comparisons are provided. For example, please see below works in matrix factorization approaches, sparse word representation learning, codebook learning and other quantization approaches for compressing word embeddings:
https://www.aclweb.org/anthology/P16-1022/
https://aaai.org/ojs/index.php/AAAI/article/download/4578/4456
http://web.cs.ucla.edu/~chohsieh/papers/Mulcode_Compressor.pdf
https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17042/16071
https://storage.googleapis.com/pub-tools-public-publication-data/pdf/f158f7c81ed8e985fd51a20d193103ce427cad51.pdf
https://arxiv.org/pdf/1711.01068.pdf
https://arxiv.org/abs/1510.00149
I would appreciate if comparisons with some of these approaches is provided in the next iteration of this work.
Other suggestions:
1. I think the paper would benefit from some analysis of the differences in the word embeddings learnt by a general lookup table learning model in comparison with the word embeddings learnt by this model. How are the embeddings compressed? How do the decompressed embeddings compare to the embeddings learnt by the lookup approach? More insights in the machinery via some visualizations would help.
2. How do the gains of this method change as more or less training data is provided. For example, are the gains lesser on Gigaword? This would be interesting to know.
3. GLT is mentioned twice in this paper. Perhaps a slightly more detailed explanation of the same would help improve the readability of this paper.
Based on the presentation improvements and new experiments added to the paper in the rebuttal time period, I am updating my evaluation of this work.


Deep Neural Networks (DNNs) have achieved high accuracy in various machine learning applications in recent years. As the recognition accuracy of deep learning applications increases, reducing the complexity of these neural networks and performing the DNN computation on embedded systems or mobile devices become an emerging and crucial challenge. Quantization has been presented to reduce the utilization of computational resources by compressing the input data and weights from floating-point numbers to integers with shorter bit-width. For practical power reduction, it is necessary to operate these DNNs with quantized parameters on appropriate hardware. Therefore, systolic arrays are adopted to be the major computation units for matrix multiplication in DNN accelerators. To obtain a better tradeoff between the precision/accuracy and power consumption, using parameters with various bit-widths among different layers within a DNN is an advanced quantization method. In this paper, we propose a novel decomposition strategy to construct a low-power decomposable multiplier-accumulator (MAC) for the energy efficiency of quantized DNNs. In the experiments, when 65% multiplication operations of VGG-16 are operated in shorter bit-width with at most 1% accuracy loss on the CIFAR-10 dataset, our decomposable MAC has 50% energy reduction compared with a non-decomposable MAC.

Summary:
This paper proposes a novel decomposition strategy for matrix multiplication to aim for less energy consumption. They demonstrate that energy consumption is reduced on several CNNs. It is an interesting work but it is not sure that there is a difference in contribution when compared to previous work.
The main argument to impact to the score:
    1. A similar idea is addressed in [1]. It is highly recommended to show the difference in contribution.
    2. In Figure 7,  the performance and energy consumption of systolic arrays of decomposable MAC  is shown. However, the only energy consumption is shown in section 4.2 when the method is applied to DNNs. It is better to show both performance and energy consumption.
    3. It is inappropriate to quantize VGG16, VGG19, DenseNet-121, DenseNet-169 and MobileNet based on CIFAR10 dataset since all models are designed for ImageNet dataset. It is appropriate to quantize these models on ImageNet dataset.
[1]     Sungju Ryu, Hyungjun Kim, Wooseok Yi and Jae-Joon Kim. BitBlade: Area and Energy-Efficient Precision-Scalable Neural Network Accelerator with Bitwise Summation. DAC. 2019.
Minor comments not to impact the score:
    1. "Very Deep Convolutional Networks for Large-Scale Image Recognition" has been accepted by ICLR 2014. It is better not to use arXiv preprint on citations unless there is a reason.
    2. It is recommended to put citations on CIFAR-10 dataset.