Dual learning has been successfully applied in many machine learning applications including machine translation, image-to-image transformation, etc. The high-level idea of dual learning is very intuitive: if we map an x from one domain to another and then map it back, we should recover the original x. Although its effectiveness has been empirically verified, theoretical understanding of dual learning is still missing. In this paper, we conduct a theoretical study to understand why and when dual learning can improve a mapping function. Based on the theoretical discoveries, we extend dual learning by introducing more related mappings and propose highly symmetric frameworks, cycle dual learning and multipath dual learning, in both of which we can leverage the feedback signals from additional domains to improve the qualities of the mappings. We prove that both cycle dual learning and multipath dual learning can boost the performance of standard dual learning under mild conditions. Experiments on WMT 14 English↔German and MultiUN English↔French translations verify our theoretical findings on dual learning, and the results on the translations among English, French, and Spanish of MultiUN demonstrate the efficacy of cycle dual learning and multipath dual learning.

The paper gives theorems concerning "dual learning" - that is, making
use of round-trip consistency in learning of translation and other
tasks.
There are some interesting ideas here. Unfortunately, I think there
are issues with clarity/choice of notation and correctness (errors
resulting from problems with the notation - or at least it's very
hard to figure out if things are correct under some intepretation).
More specifically, I'm uneasy about the use of x^i and x^j as defined
in section 2. In some cases x^j is a deterministic function of x^i, in
some cases it's a random variable, these cases are mixed. Section 2
becomes tangled up in this issue. It would be much better I think to
define a function f_ij(x) for each (i,j) pair that maps a sentence
x \in S^i to its correct translation f_ij(x) \in S^j.
A critical problem with the paper is that Eq. 2 is I think incorrect.
Clearly,
Pr(T_ij(x_i) = f_ij(x_i), T_ji(f_ij(x_i)) = x_i)      [1]
=
Pr(T_ij(x_i) = f_ij(x_i))                             [2]
*
Pr(T_ji(f_ij(x_i)) = x_i) | T_ij(x_i) = f_ij(x_i))    [3]
I think [1] is what is meant by the left-hand-side of Eq 2 in the paper -
though the use of x^j is ambiguous (this ambiguity is a real issue).
It can be verified that
Pr(T_ij(x_i) = f_ij(x_i)) = p_ij
however
Pr(T_ji(f_ij(x_i)) = x_i) | T_ij(x_i) = f_ij(x_i)) \neq p^r_ji
The definition of p^r_ji is that of a different quantity.
This problem unfortunately permeates the statement of theorem 1, and
the proof of theorem 1. It is probably fixable but without a
significantly revised version of the paper a reader/reviewer is
basically guessing what a corrected version of the paper would
be. Unfortunately I think publishing the paper with errors such as
this would be a problem.
Some other points:
[1] The theorem really does have to assume that there is a unique
correct translation f_ij(x^i) for each sentence x^i. Having multiple
possible translations breaks things. The authors say in section 2 "In
practice, we may select a threshold BLEU (Papineni et al., 2002a)
score, above which the translation is considered correct": this seems
to imply that the results apply when multiple translations (above
a certain BLEU score) are possible. But my understanding is that this
will completely break the results (or at least require a significant
modification of the theory).
[2] A further problem with ambiguity/notation is that T^d is never
explicitly defined. Presumably we always have T_ij^s(x^i) = T_ij(x^i) if
T_ji(T_ij(x^i)) = x^i? That needs to be explicitly stated.
[3] There may be something interesting in theorem 1 - putting aside point
[1] above - but I am just really uneasy with this theorem and its proof
given that it uses p^r_ji, and the issue with Eq. 2.
[4] Another issue with the definition of p^r_ij: the notation
P_{X^(j, r) ~ \mu}(...) where the expression ... does not refer
to X^{j, r} (instead it refers to x^j) is just really odd,
and confusing.


Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration. Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges. How to ensure visually realistic restoration while avoiding hallucination or modecollapse? How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification? Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network. With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task. Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features. Additionally, the trained task network show potentials to achieve superhuman level performance in identifying pathology and diagnosis. Further validation on super-resolved face identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks.

This paper proposed a new method for image restoration based a task-discriminator in addition to the GAN network. It shows superior performance than the baseline methods without such task-discriminator on medical image restoration and image super-resolution. While the results are better, the idea seems straightforward and has limited novelty. Please see the following comments:
1. Adding an task-discriminator in a GAN network seems straightforward to improve the specific task. And this idea has already used in existing papers, e.g. Cycada.
Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A.A. and Darrell, T., 2017. Cycada: Cycle-consistent adversarial domain adaptation. ICML, 2018
2. On the application side, the results are not very convincing because the baseline methods were not selected properly. For medical image reconstruction and image super-resolution, the proposed method was not compared with any of the state-of-the-art methods, but only with the same method without a task-discriminator as a baseline. For those tasks, there are many traditional methods and deep nets with different losses. For example, a simple L1/L2 or perceptual loss probably leads to better PSNR than the GAN loss, which is not compared at all. See the attached references.
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z. and Shi, W., Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR 2017.
Johnson, J., Alahi, A. and Fei-Fei, L., Perceptual losses for real-time style transfer and super-resolution. In ECCV 2016.
Kim, J., Kwon Lee, J. and Mu Lee, K., Accurate image super-resolution using very deep convolutional networks. In CVPR 2016.
3. Some questions about medical image datasets. For the low-dose PET dataset, the input was randomly undersampled by a factor of 100. What is the random pattern? Is it uniform? In addition, why not acquire real low-dose data and show the quality results using the proposed model? For the multi-constast MRI data, how is the input generated and what is the ground-truth?


Learning at small or large scales of data is addressed by two strong but divided frontiers: few-shot learning and standard supervised learning. Few-shot learning focuses on sample efficiency at small scale, while supervised learning focuses on accuracy at large scale. Ideally they could be reconciled for effective learning at any number of data points (shot) and number of classes (way). To span the full spectrum of shot and way, we frame the variadic learning regime of learning from any number of inputs. We approach variadic learning by meta-learning a novel multi-modal clustering model that connects bayesian nonparametrics and deep metric learning. Our bayesian nonparametric deep embedding (BANDE) method is optimized end-to-end with a single objective, and adaptively adjusts capacity to learn from variable amounts of supervision. We show that multi-modality is critical for learning complex classes such as Omniglot alphabets and carrying out unsupervised clustering. We explore variadic learning by measuring generalization across shot and way between meta-train and meta-test, show the first results for scaling from few-way, few-shot tasks to 1692-way Omniglot classification and 5k-shot CIFAR-10 classification, and find that nonparametric methods generalize better than parametric methods. On the standard few-shot learning benchmarks of Omniglot and mini-ImageNet, BANDE equals or improves on the state-of-the-art for semi-supervised classification.

Update after Author Rebuttal
--------------
After reading the rebuttal, I'm pleased that the authors have made significant revisions, but I still think more work is needed. The "hard/soft" hybrid approach still lacks justification and perhaps wasn't compared to a soft/soft approach in a fair and fully-correct way (see detailed reply to authors). I also appreciate the efforts on revising clarity, but still find many clarity issues in the newest version that make the method hard to understand let alone reproduce. I thus stand by my rating of "borderline rejection" and urge the authors to prepare significant revisions for a future venue that avoid hybrids of hard/soft probabilities without justification.
(Original review text below. Detailed replies to authors are in posts below their responses).
Review Summary
--------------
While the focus on variadic learning is interesting, I think the present version of the paper needs far more presentational polish as well as algorithmic improvements before it is ready for ICLR. I think there is the potential for some neat ideas here and I hope the authors prepare stronger versions in the future. However, the current version is unfortunately not comprehensible or reproducible.
Paper Summary
-------------
The paper investigates developing an effective ML method for the "variadic" regime, where the method might be required to perform learning from few or many examples (shots) and few or many classes (ways). The term "variadic" comes from use in computer science for functions that can a flexible number of arguments. There may also be unlabeled data available in the few shot case, creating semi-supervised learning opportunities.
The specific method proposed is called BANDE: Bayesian Nonparametric Deep Embedding. The idea is that each data point's feature vector x_i is transformed into an embedding vector h(x_i) using a neural network, and then clustering occurs in the embedding space via a single-pass of the DP-means algorithm (Kulis & Jordan 2012). Each cluster is assumed to correspond to one "class" in the eventual classification problem, though each class might have multiple clusters (and thus be multi-modal).
Learning occurs in an episodic manner. After each episode (single-pass of DP-means), each point in a query set is embedded to its feature vector, then fed into each cluster's Gaussian likelihoods to produce a normalized cluster-assignment-probability vector that sums to one. This vector is then fed into a cross-entropy loss, where the true class's nearest cluster (largest probability value) is taken to be the true cluster. This loss is used to perform gradient updates of the embedding neural network.
There is also a "cumulative" version of the method called BANDE-C. This version keeps track of cluster means from previous episodes and allows new episodes to be initialized with these.
Experiments examine the proposed approach across image categorization tasks on Omniglot, mini-ImageNet, and CIFAR datasets.
Strengths
---------
* I like that many clusters are used for each true class label, which is better than rigid one-to-one assumptions.
Limitations
-----------
* Can only be used for classification, not regression
* The DP-means procedure does not account for the cluster-specific variance information that is used at other steps of the algorithm
Significance and Originality
----------------------------
To me, the method appears original. Any method that could really succeed across various variadic settings would be significant.
Presentation Concerns
---------------------
I have serious concerns about the presentation quality of this paper. Each section needs careful reorganization as well as rewording.
## P1: Algo. 1 contains numerous omissions that make it as written not correct.
* the number of clusters count variable "n" is not updated anywhere. As writting this algo can only update one extra cluster beyond the original n.
* the variable "c" is unbound in the else clause. You need a line that clarifies that c = argmin_{c in 1 ... n} d_ic
Would be careful about saying that "a single pass is sufficient"... you have *chosen* to do only one pass. When doing k-means, we could also make this choice. Certainly the DP-means objective could keep improving with multiple passes.
## P2: Many figures and tables lack appropriate captions/labels
Table 1: What metric is reported? Accuracy percentage? Not obvious from title/caption. Should also make very clear here how much labeled data was used.
Table 2: What metric is reported? Accuracy percentage? Not obvious from title/caption. Should also make how many labeled and unlabeled examples were used easier to find.
## P3: Descriptions of episodic learning and overall algorithm clarity
Readers unfamiliar with episodic learning are not helped with the limited coverage provided here in 3.1 and 3.2. When exactly is the "support" set used and the "query" set used? How do unlabeled points get used (both support and query appear fully labeled)? What is n? What is k? What is T? Why are some points in Q denoted with apostrophes but not others? Providing a more formal step-by-step description (perhaps with pseudocode) will be crucial.
In Sec. 3.2, the paragraph that starts with "The loss is defined" is very hard to read and parse. I suggest adding math to formally define the loss with equations. What parameters are being optimized? Which ones are fixed?
Additionally, in Sec. 3.2: "computed in the same way as standard prototypical networks"... what is the procedure exactly? If your method relies on a procedure, you should specify it in this paper and not make readers guess or lookup a procedure elsewhere.
## P4: Many steps of the algorithm are not detailed
The paper claims to set \lambda using a technique from another paper, but does not summarize this technique. This makes things nearly impossible to reproduce. Please add such details in the appendix.
Major Technical Concerns
------------------------
## Alg. 1 concerns: Requires two (not one) passes and mixes hard and soft assingments and different variance assumptions awkwardly
The BANDE algorithm (Alg. 1) has some unjustified properties. Hard assignment decisions which assume vanishing variances are used to find a closest cluster, but then later soft assignments with non-zero variances are used. This is a bit heuristic and lacks justification... why not use soft assignment throughout? The DP means procedure is derived from a specific objective function that assumes hard assignment. Seems weird to use it for convenience and then discard instead of coming up with the small fix that would make soft assignment consistent throughout.
Furthermore, The authors claim it is a one pass algorithm, but in fact as written in Alg. 1 it seems to require two passes: the first pass keeps an original set of cluster centers fixed and then creates new centers whenever an example's distance to the closest center exceeds \lambda. But then, the *soft* assignment step that updates "z" requires again the distance from each point to all centers be computed, which requires another pass (since some new clusters may exist which did not when the point was first visited). While the new soft values will be close to zero, they will not be *exactly* zero, and thus they matter.
## Unclear if/how cluster-specific variance parameters learned
From the text on top of page 4, it seems that the paper assumes that there exist cluster-specific variances \sigma_c. However, these are not mentioned elsewhere, only a general (not cluster-specific) label variance \sigma and fixed unlabeled variance sigma_u are used.
## Experiments lack comparison to internal baselines
The paper doesn't evaluate sensitivity to key fixed hyperparameters (e.g. \alpha, \lambda) or compare variants of their approach (with and without soft clustering step, with and without multimodality via DP-means). It is difficult to tell which design choices of the method are most crucial.


TOWARDS GAN BENCHMARKS WHICH REQUIRE GENERALIZATION
For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be "won" by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.

Summary:
The paper looks at the problem of benchmarking models that unconditionally generate images. In particular they focus on GAN models and discuss the Inception Score (IS) and Fréchet Inception Distance (FID) metrics. The authors argue that a good benchmark should not have a trivial solution (e.g. memorising the dataset) and find that a necessary condition for such a metric is a large number of samples. They also find that for IS and FID , a GAN is outperformed by a model that memorises the dataset, while a method based on neural network divergences (NND) does not show the same behaviour. NND works by training a discriminative model to discriminate between samples of the generative model and samples from a held out test set. The poorer the discriminative model performs, the better the generative model is.
The authors show a range of results using a CNN based divergence: on PixelCNN++, GANs, overfitted GANs, WGAN-GP and conclude that it’s a better metric than IS/FID at the expense of requiring much more computation to evaluate.  They also perform a test with limited compute and show that the results correlate well with a bigger dataset, but show some bias.
Review:
The paper is well written, with a clear description of the properties a good benchmark should have, an analysis of the current solutions and their shortcomings and an extensive experimental evaluation of the CNN divergence metric. The authors also compared with non GAN methods and experimented with small datasets, both are not necessarily within scope but a welcome addition. The authors also open source their code.
In the section “Outperforming Memorization”, the authors mention a way to tune capacity of the “critic” network and influence its ability to overfit on the sample. This means that if someone wants to compare the generalisation and diversity of samples between GANs, they would need to train the exact same critic CNN to be able to make a comparison. However the authors do not provide any principled way to determine the right size of the "critic" network. In general, given evaluating the metric requires training a network from scratch, it will be very difficult to make this consistent. This makes the proposed benchmark more impractical to use than its alternatives.
In the section “training against the metric”, the authors mention that a main criticism is the fact that a GAN directly optimises for the NND loss. In table 3 we indeed see that this is the case, however the authors argue that perhaps the GAN is simply the better model. I am worried by the fact that both PixelCNN++ and IAF-VAE perform worse than the training set on this benchmark. It seems like this particular benchmark would then work well specifically for GANs, but would (still) not allow us to compare with models trained using maximum likelihood.
In conclusion, I think the paper is well written and the authors clearly make progress towards a dependable benchmark for GANs. The paper does not introduce any new method, but instead has a thorough analysis and discussion of current methods which is worthwhile by itself.
Nits:
Page 7, second paragraph, fifth line, spurious “q”
########
Revision
I would like to thank the authors for a thoughtful revision and response. I have updated my score to a 7 and think this paper is a worthy contribution to ICLR. The new drawback section is well written and informative.


Recently, there have been some attempts to use non-recurrent neural models for language modeling. However, a noticeable performance gap still remains. We propose a nonrecurrent neural language model, dubbed graph temporal convolutional network (GTCN), that relies on graph neural network blocks and convolution operations. While the standard recurrent neural network language models encode sentences sequentially without modeling higher-level structural information, our model regards sentences as graphs and processes input words within a message propagation framework, aiming to learn better syntactic information by inferring skip-word connections. Specifically, the graph network blocks operate in parallel and learn the underlying graph structures in sentences without any additional annotation pertaining to structure knowledge. Experiments demonstrate that the model without recurrence can achieve comparable perplexity results in language modeling tasks and successfully learn syntactic information.

This paper draws inspiration from recent works on graph convolutional networks and proposes GTCN, a convolutional architecture for language modeling. The key intuition is to treat sentences as (potentially densely-connected) graphs over tokens, instead of sequences as in many RNN-based language models. The model then, when predicting a token, summarizes previous tokens using attention mechanism as context. Empirical evaluation on word-level language modeling on Penn Treebank shows competitive performance.
The idea of this work appears reasonable and well-motivated to me. But the connections to previous works, especially those based on self-attention, should be clearly addressed. Further, writing can be improved, and I would encourage a thorough revision since there are typos making the paper a bit hard to follow.
Last but not least, I find several of the claims not very-well supported. Please see details below.
Pros:
- Well-motivated intuition treating language as structured.
Cons:
- Writing can be improved.
- Missing discussion of existing works.
Details:
- Based on my understanding of Eqs. 6--11, the proposed GTCN seems to be a gated version (also equipped with window-2 convolutions) of the self-attention mechanism. Could the authors comment on how GTCN relates to Vaswani et al. (2017), Salton et al. (2017), among others? Also, empirical comparisons to self-attention based language models might be necessary.
- I was confused by Eqs. 13--14 and the text around it. Doesn't one need some kind of classifier (e.g., an MLP) to predict x_{t+1}? Why are these two equations predicting word embedding?
- The start of Section 4.1. There seems to be a typo here. I'm assuming the two vectors are `$\mathbf{v}$ and $\mathbf{q}$` here, as in Eqs. 6 and 7.
- More clarification on Eq. 9 might be necessary. Is \mathbf{W}^p part of the parameters? I'm guessing \mathbf{W}_{i-j}^p selects a row from the matrix, since there is a dot product outside.
- Can the authors clarify Eq. 5? I'm not sure how to interpret it, and it seems not used anywhere else.
- Eq. 2 is a bit misleading: it might give the impression that f_{t+1} does not depend on f_t (and so forth), which is not the case for LSTM.
- It would be interesting to be how GTCN compare to other models in efficiency, since the paper mentions parallel computation many times.
- Contribution.2: GTCN is not really the state-of-the-art model on LM.
- Comparison to RNNG: RNNG treats each sentence as a separate sequence, in contrast to most cited works in Table 1, where the whole training (eval) set is treated as a single sequence, and truncate the length when applying BPTT. And according to the second paragraph of Section 5.1, this work follows the latter. To the best of my knowledge, such a difference does have an effect on the perplexity metric. In this sense, RNNG is not comparable to the rest in Table 1. It is perhaps fine to still put it in the table, but please clarify it in the text.
Minors:
- Why is the margins above equations seem larger. Can the authors make sure the template is right?
- Around Eq.5: why is \mathbf{X} is capitalized in the eq, but not in the text? Are they the same thing?
- Section 4.3: the dependence of attention weights $a$ is not reflected in the notation.
- Section 5.1: I think what it means here is a `10K` vocabulary, instead of a 10K word tiny corpora.
References
Vaswani et al.. 2017. Attention is All You Need. In Proc. of NIPS.
Salton et al.. 2017. Attentive Language Models.


We propose an energy-based framework for correcting mislabelled training examples in the context of binary classification. While existing work addresses random and class-dependent label noise, we focus on feature dependent label noise, which is ubiquitous in real-world data and difficult to model. Two elements distinguish our approach from others: 1) instead of relying on the original feature space, we employ an autoencoder to learn a discriminative representation and 2) we introduce an energy-based formalism for the label correction problem. We prove that a discriminative representation can be learned by training a generative model using a loss function comprised of the difference of energies corresponding to each class. The learned energy value for each training instance is compared to the original training labels and contradictions between energy assignment and training label are used to correct labels. We validate our method across eight datasets, spanning synthetic and realistic settings, and demonstrate the technique's state-of-the-art label correction performance. Furthermore, we derive analytical expressions to show the effect of label noise on the gradients of empirical risk.

This submission proposes an energy-based method to correct mislabelled examples. Intuitively, the authors claim that contradictions between energy and noisy labels can be used to identify label noise. To make the idea reliable, the authors propose to compute the energy by using learned (commonly shared) features. Experiment results look good. The presentation is also clear. My concerns are as follows:
(1) By learning discriminative features and then correcting the label noise, the authors have implicitly assumed that the label noise strongly dependent on the discriminative features. This assumption may be strong as most labels are provided according to the original instance (features). In the experiment section, it is very unclear about how the noise is generated, e.g., how to select x_i according to variance? How to set the threshold? What is "col" in Tables 1 and 2? What are the overall label noise rates? Note that if the threshold for the variance is large. It means that the noise is only added to the discriminative features, making the experiment too ad-hoc.
(2) The theory of why the residual energy can be used to identify label noise is elusive. How to set the threshold for identifying label noise with a theoretical guarantee is also unclear. Two recent papers on learning with instance-dependent label noise are surprisingly missed, e.g., Menon, Aditya Krishna, Brendan Van Rooyen, and Nagarajan Natarajan. "Learning from binary labels with instance-dependent corruption." arXiv preprint arXiv:1605.00751 (2016). and Cheng, Jiacheng, et al. "Learning with Bounded Instance-and Label-dependent Label Noise." arXiv preprint arXiv:1709.03768 (2017). The latter one proposes algorithms to identify label noise with theoretical guarantees. The authors should compare the proposed method with them.
(3) There are methods provided for choose the values of hyperparameters. Most of them are empirically set, which is not convincing.
(4) The authors reported that with discriminative features learned by employing noisy data, the proposed method also provide good performance. It would be interesting to see how corrected labels will recursively help better learn the discriminative features. Illustrating figures are preferred.


We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent's actions. Causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent's reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents' actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. We therefore employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.

INTRINSIC SOCIAL MOTIVATION VIA CAUSAL INFLUENCE IN MULTI-AGENT RL
Main Idea: The authors consider adding a reward term to standard MARL which is the mutual information between its actions and the actions of others. They show that adding this intrinsic social motivation can lead to increased cooperation in several social dilemmas.
Strong Points:
-	This paper is a novel extension of ideas from single agent RL to multi agent RL, there are clear benefits from doing reward shaping in the right way to make deep RL work better.
-	The paper focuses on cooperative environments which is an underfocused area in RL right now

Weak Points:
-	There is missing discussion of a lot of literature. The causal influence term can be thought of as a form of reward shaping. There is little discussion on the (large) literature on reward shaping to get MARL to exhibit good behavior.
-	The results feel quite thin. Related to the point above: the theory of different types of reward shaping (e.g. optimistic Q-learning, prosociality, etc…) are well understood. It is not clear to me under what conditions the authors’ proposed augmentation to the reward function will lead to better or worse outcomes. The experiments in this paper are quite simple and only span a small set of environments so it would be good to have at least some formal theory.
-	Social dilemmas don’t seem like the best application. The authors define the social dilemma as: “For each individual agent, ‘defecting’ i.e. non-cooperative behavior has the highest payoff.” With the intrinsic motivation, agents learn to cooperate. This is good, however, if we’re thinking about situations where agents aren’t trained together and have their own rewards (the authors’ example: “autonomous vehicles are likely to be produced by a wide variety of organizations and institutions with mixed motivations”) then won’t these agents be exploited by rational agents? Other solutions to this problem (e.g. recent papers on tit-for-tat by Lerer & Peysakhovich or LOLA by Foerster et al. construct agents where defectors get explicitly punished and so don’t want to try exploiting). Is there something I am missing here? Do the agents learn to punish non-cooperators (if no, isn’t it rational at that point to just not cooperate and won’t self-driving cars trained via this method get exploited by others)?
-	Relate to the point(s) above: a better environment for application here seems to be coordination games/”Stag Hunt” games where it is known that MARL converges to poor equilibria and many other methods e.g. optimistic Q-learning or prosociality have been invented to make things work better. Perhaps the method proposed here will work better than these (and it has the appealing property that it does not require the ability to observe the other agents' rewards as e.g. prosociality does)
-	This paper contains some quite grandiose language connecting the proposed reward shaping to “how humans learn” (example: It may also have correlates in human cognition; experiments show that newborn infants are sensitive to correspondences between their own actions and the actions of other people, and use this to coordinate their behavior with others) it’s unclear to me that humans experience extra reward for their actions having high mutual information (and/or causal information with others). While it’s fine to argue some of these points at a high level I would suggest scrubbing the text of the gratuitous references to this.
Nits:
“Crawford & Sobel (1982) find that once agents’ interests diverge by a finite amount, no communication is to be expected.” – this is an awkward phrasing of the Crawford and Sobel result (it can be read as “if interests diverge by any epsilon there can be no communication”). The CS result is that information revealed in communication (in equilibrium) is proportional to amount of common interest.


MAE: MUTUAL POSTERIOR-DIVERGENCE REGULAR- IZATION FOR VARIATIONAL AUTOENCODERS
Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suffers from learning uninformative latent representations with the observation called KL Varnishing, in which case VAE collapses into an unconditional generative model. In this work, we introduce mutual posterior-divergence regularization, a novel regularization that is able to control the geometry of the latent space to accomplish meaningful representation learning, while achieving comparable or superior capability of density estimation. Experiments on three image benchmark datasets demonstrate that, when equipped with powerful decoders, our model performs well both on density estimation and representation learning.

In this paper the authors present mutual posterior divergence regularization, a data-dependent regularization for the ELBO that enforces diversity and smoothness of the variational posteriors. The experiments show the effectiveness of the model for density estimation and representation learning.
This is an interesting paper dealing with the important issues of fully exploiting the stochastic part of VAE models and avoiding inactive latent units in the presence of very expressive decoders. The paper reads well and is well motivated.
The authors claim that their method is "encouraging the learned variational posteriors to be diverse". While it is important to have models that can use well the latent space, the constraints that are encoded seem too strong. If two data points are very similar, why should there be a term encouraging their posterior approximation to be different? In this case, their true posteriors will be in fact be similar, so it seems counter-intuitive to force their approximations to be different.
The numerical results seem promising, but I think they could be further improved and made more convincing.
- For the density estimation experiments, while there is an improvement in terms of NLL thanks to the new regularizer, it is not clear which is the additional computational burden. How much longer does it takes to train the model when computing all the regularization terms in the experiments with batch size 100?
- I am not completely convinced by the claims on the ability of the regularizer to improve the learned representations. K-means implicitly assumes that the data manifold is Euclidean. However, as shown for example by [Arvanitidis et al. Latent space oddity: on the curvature of deep generative models, ICLR 2018] and other authors, the latent manifold of VAEs is not Euclidean, and curved riemannian manifolds should be used when computing distances and performing clustering. Applying k-means in the high dimensional latent spaces of ResNet VAE and VLAE does not seem therefore a good idea.
One possible reason why your MAE model may perform better in the unsupervised clustering of table 2 is that the terms added to the elbo by the regularizer may force the space to be more Euclidean (e.g. the squared difference term in the Gaussian KL) and therefore more suitable for k-means.
- The semi-supervised classification experiment is definitely better to assess the representation learning capabilities, but KNN suffers with the same issues with the Euclidean distance as in the k-means experiments, and the linear classifier may not be flexible enough for non-euclidean and non-linear manifolds. Have you tried any other non-linear classifiers?
- Comparisons with other methods that aim at making the model learn better representation (such as the kl-annealing of the beta-vae) would be useful.
- The lack of improvements on the natural image task is a bit concerning for the generalizability of the results.
Typos and minor comments:
- devergence -> divergence in introduction
- assistant -> assistance in 2.3
- the items (1) and (2) in 3.1 are not very clear
- set -> sets in 3.2
- achieving -> achieve below theorem 1
- cluatering -> clustering in table 2


LEARNING LATENT SUPERSTRUCTURES IN VARIA- TIONAL AUTOENCODERS FOR DEEP MULTIDIMEN- SIONAL CLUSTERING
We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features. In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data. When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model. We call our model the latent tree variational autoencoder (LTVAE). Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable. This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways.

This paper introduces a new VAE model, the latent tree VAE (LTVAE), which aims to learn models with multifaceted clustering, that is separate clusterings are enforced on different subsets of the latent features.  This is achieved using a tree-structured prior on a set of discrete "super latent variables" (Y_1,...,Y_L) that identify which cluster the datapoint falls into for each separate facet (i.e. there is a separate clustering associated with each Y_n).  The subset of the standard latent variables z then form a Gaussian mixture model (GMM) for each Y_n.   Both the structure of this setup (i.e. the associated graphical model) and the parameters (i.e. means and variances of the clusters) are learned during training.  This introduces a number of computational challenges not usually seen in for VAE training, for which, seemingly well thought through, novel schemes are introduced, most notably a message passing scheme for calculating gradients of the log marginal p(z).
Overall I think this is a very good paper.  The exposition of the work is, for the most part, very good - the paper was a pleasure to read.  I think that the key idea is novel and adds something unique and useful to the literature, I thus think it is work which will be of substantial interest to the ICLR community.  The quality of the paper is also very good: algorithmic details seem to have been well thought through and the experimental evaluation is above average, both in terms of apparent performance and in the breadth of experiments considered.  I would very much like to see this work accepted to ICLR and I think that the extra use of space over 8 pages in the submission is justified.  However, I do have some questions and concerns that I would like to see addressed in the rebuttal period and I may lower my score if they are not.
The key issues I would like to see addressed further discussion on are:
a) There is no discussion about what is done for the encoder in the paper.  This is surely a very important consideration here as if the encoder is not expressive enough, this will impact the learned models.  For example, the dependency structures of the latent space induce particular dependencies in the posterior that must be carefully handled to avoid harming the learning (see e.g. https://arxiv.org/abs/1712.00287).
b) I would like to see some numerical results for the similarity between the different clusterings that are learned.  A lot of the novelty of the work rests on being able to pick up different clusterings with the different facets.  However, the results suggest that the clusterings may actually have very significant overlap and so this should be quantified.
c) The approach is presuming substantially slower than a setup where the structure is pre-fixed.  I think it is fine even if there is a big slow down, but I would like to see timing information so that the reader can assess how much higher the time cost is.
d) As far as I can tell (sorry if I have made a mistake), the presented results are from single runs.  I would like to see information about the variability across different runs so that the fragility of the approach can be assessed.
e) I would like to see more justification for having a dependency structure between the Y's, ideally both in motivating this choice and in experimental evaluation to check it (more generally ablation studies for different components of the algorithm would improve the paper).  Might it be possible to use this in a way the encourages the different clusterings to be distinct from one another?
Other comments:
1) Though the writing is generally very good, there are a few exceptions:
- The second paragraph in the intro becomes a list of related work from the point where DEC is introduced.  This should be moved to the related work to improve the flow (just cite those papers at the end of the first sentence in the third paragraph) and it would be good for it to be less of a list of separate things and more something that puts the current work in the context of other approaches.
- The paragraph after Eq 3 needs some rewriting
- The explanations around and including equations 5 and 6 were quite poor: \pi is referred to but not used, it is not made clear that that g is the gradient of log p(z) instead of p(z), use brackets for the log in Eq 6 to avoid ambiguity
2) The reference formatting is wrong (i.e. cite is used everywhere instead of citep)
3) I thought the motivation for the approach in the intro was very good
4) As the seemingly most related work, it would be good to elaborate more on the Goyal et al paper and the differences of your approach to theirs.  Is there a reason this is not used as a baseline in the experiments?
5) I could not understand the step from the gradient to the gradient of the log in Eq 6.  Is this because p(y_b|z) = f(y_b) Norm(..)?
6) The text in figures 2 and 3 is too small and difficult to make out.
7) I think it is misleading to talk about p(z) as being a marginal likelihood and would use the term marginal prior, or just marginal, instead.
8) I thought Figure 4b provided a nice demonstration.
9) Is there a reason that log likelihood / ELBO scores are only provided on MNIST and only for the LTVAE / VAE?  I might be wrong, but I thought at least some of the other baselines provide this and those results presumably already exist as a side effect from calculating the clustering scores?  Relatedly, I'm aware that a previous version of this work included estimates of the normalized mutual information -- is there any reason these are no longer included?
10) Did the larger dimensional latent spaced used for the qualitative results improve or worsen the performance of previous metrics?
Minor points / typos
- mehod -> method
- of generation network -> of the generation network
- brackets in eq 7
- MoG not defined in section 4.5


VARIATIONAL AUTOENCODER WITH ARBITRARY CONDITIONING
We propose a single neural probabilistic model based on variational autoencoder that can be conditioned on an arbitrary subset of observed features and then sample the remaining features in "one shot". The features may be both real-valued and categorical. Training of the model is performed by stochastic variational Bayes. The experimental evaluation on synthetic data, as well as feature imputation and image inpainting problems, shows the effectiveness of the proposed approach and diversity of the generated samples.

The goal of this paper is to use deep generative models for missing data imputation. This paper proposes learning a latent variable deep generative model over every randomly sampled subset of observed features. First, a masking variable is sampled from a chosen prior distribution. The mask determines which features are observed. Then, the likelihood of the observed features is maximized via a lower bound. Inference in this latent variable model is achieved through the use of an inference network which conditions on the set of "missing" (to the generative model) features.
Novelty:
Generative models have a long history of being used to impute missing data. e.g. http://www.cs.toronto.edu/~fritz/absps/ranzato_cvpr2011.pdf, https://arxiv.org/pdf/1610.04167.pdf,
https://arxiv.org/pdf/1808.01684.pdf, https://arxiv.org/pdf/1401.4082.pdf [Appendix F]
It is a little difficult to guage what the novelty of this work is.
Clarity
This is a poorly written paper. Distilling the proposed methodology down to one paragraph was challenging since the text meanders through several concepts whose relevance to the overarching goal is questionable. For example, it is not clear what Section 3.2 adds to the discussion. The text describes a heuristic used in learning GSNNs only to say that the loss function used by GSNNs is not used in the experimental section for this paper -- this renders most of 4.3.2 redundant. There are issues like awkward grammar, sloppy notation, and spelling mistakes (please run spell check!) throughout the manuscript. Please use a different notation when referring to the variational distributions (do not re-use "p").
Experimental Results
The model is evaluated against MICE and MissForest on UCI datasets. RMSE and accuracy of classification (from imputed data is compared). The complexity of data considered is simplistic (and may not make use of the expressivity of the deep generative model). Why not run these experiments on datasets like MNIST and Omniglot?
Beyond that:
(a) was there any comparison to how classification performance behaves when using another neural network based imputation baseline (e.g. the method in Yoon et. al)?
(b) the *kind* of missingness considered here appears to be MCAR (the easiest kind to tackle) -- did you consider experiments with other kinds of missingess?
The qualitative results presented in this work are interesting. The method does appear to produce more diverse in-paintings than the method from Yeh et. al (though the examples considered are not aligned).
Table 5 claims negative log-likelihood numbers on MNIST as low as 61 and 41 (I assume nats...). These numbers do not make sense. How were they computed?
Priors on b:
What kind of priors on b did you experiment with?


GENERATING MULTIPLE OBJECTS AT SPATIALLY DISTINCT LOCATIONS
Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image background and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image background.

The authors are proposing a method for allowing the generation of multiple objects in generated images given simple supervision such as bounding boxes and their associated labels. They control the spatial location of generated objects by the mean of an object pathway added to the architecture of both Generator and Discriminator within a GAN framework. They show generated results on Multi-MNIST, CLEVR with discussions of their model's abilities and properties. they also provide quantitative results on MSCOCO (IS and FID) using StackGAN and AttGAN models with the object pathway modifications and show some improvements compared to the original models. However it must be noted (as commented by the authors) that these models are using image captions only and do not have explicit supervision of bounding box and object labels.
This paper proposes a simple approach to generating requested objects in GAN-based image generation task, The method is supervised and requests (in its current form) the Bounding Boxes and Labels of the objects to integrate into the image generation. This task of controlling the nature (identity) and size of objects to integrate in a generated image is an important one and is significant to the GAN-base image generation community. In terms of originality, the approach is a nice simple architecture that takes care of the spatial location problem head-on. It seems like an obvious step but this does not take away from the merits of the proposed method.
The generator Global path is given a noise component. From the text, it does not seem that the Object path is given a noise component. Do you generate always the same object given the same label and Bounding Box then? Why not integrate some noise in this pipeline too?
Multi-MNIST:
The authors present results on Multi-MNIST 50K customed data to present the ability of the model to accurately put request images in the correct bounding box (BB) and do some ablation study. This is an interesting test as it shows that indeed the method proposed generates digits where it is expected to. Could you provide the ground truth labels for each/some image/s? For the failure cases it is often not clear what digit is what. For the Row E and F, 1s could be 7s and vice versa. Since it is a qualitative study, it would be nice to have the Ground Truth (GT) (which you provide to G at for generation). For the failure case of Row D (right) an interesting results would have been to have example of a digit bounding box from top to bottom with few pixel vertical shift to visualize when the model starts to mess-up the generation. This seems to point that your model (exposed to the location from BB for the object paths) is sensitive to what locations it has seen in training. How would you make the object path more robust to unseen location (overall you need to design an object of a given size, then locate it in your empty canvas prior to the CNN for generation)?
CLEVR:
The images resolution make it hard to really see the shape of the images (here too, the GT would be great). The bounding boxes make the images even harder to parse. I know the colors change but "We can confirm that the model can control both location and object's shape". For the location, it is true, for the shape is hard to completely tell at this resolution without GT.
MS-COCO:
 Just a comment in passing on the fact that resizing images from COCO to 256x256 will inherently distort quite a bit of images, the median size (for each dim) for COCO is 640x480, if I am not mistaken. Most, if not all images in COCO are not 1.0 size ratio.
The quantitative results on COCO seem to confirm that the proposed method is generating "better" images according to IS and FID. This is a good thing, however the technique is strongly supervised (Bounding Box and Object Labels, caption compared to solely captions for StackGAN and AttGAN) so this result should be expected and really put into perspective as your are not comparing models w/ the same supervision (which you mention in the Discussion).
Discussion:
I appreciate that the authors addressed the limitations of their approach in this section. The overlapping BBs seems to be an interesting challenge. Did you try to normalize the embeddings in overlapping area? A simple sum does not seem to be a good solution. In Figure 7 w/ overlapping zebras, the generation seems completely lost.
In terms of clarity, the paper is well-written but would benefit *greatly* from using variables names when discussing 'layout embedding', 'generated local labels', etc. Variable names and equations, while not necessary, can go a long way to clearly express a model's internal blocks (most of the papers you referenced are using this approach). The paper employs none of this commonly used standard and suffers from it. I myself had to write down on the margin the different variables used at each step described in text to have an understanding of what was done (with help of Figure 1). You should reference Figure 1 in the Introduction, as you cover your approach there and the Figure is useful to grasp your contributions.
Another comment concerning clarity is, while it is fine to rely on previously published papers for description of our own work, you should not assume full knowledge from the reader and your paper should stand on its own without having the reader lookup for several papers to have an understanding of your training procedures. If one uses GAN training, it should be expected to cover/formulate quickly the min max game and the various losses you are trying to minimize. I am afraid that "using common GAN procedures" is not enough. When describing your experimental setup, pointing to another paper as "hyperparameters remain the same as in the original training procedure" should not be a substitution for covering it too, even if lightly in the Appendix. For instance: in the Appendix, it is mentioned that training was stopped at 20 epochs for Multi-MNIST, 40 for CLEVR... How did you decide on the epoch (early stopping, stopped before instabillity of GAN training, etc.) Did you use SGD? ADAM? Did you adjust the learning rate, which schedule? etc. for your GAN training. This information in the Appendix would make the paper overall stronger.
Last comment: In terms of generation multiple objects. Have you had the chance to run an object detector on your generated image (you can build one on MSCOCO given the bounding box and label, finetune an ImageNet pretrained model). It would be interesting to see if the generated images are good enough for object detection.
Post-Rebuttal: Given the work from the authors on improving the clarity of the paper as well as investigating the use of object detection metrics to compare their methods, I decided to move my rating upward to 7


Driver's cognitive state of mental fatigue significantly affects driving performance and more importantly public safety. Previous studies leverage the response time (RT) as the metric for mental fatigue and aim at estimating the exact value of RT using electroencephalogram (EEG) signals within a regression model. However, due to the easily corrupted EEG signals and also non-smooth RTs during data collection, regular regression methods generally suffer from poor generalization performance. Considering that human response time is the reflection of brain dynamics preference rather than a single value, a novel model called Brain Dynamics ranking (BDrank) has been proposed. BDrank could learn from brain dynamics preferences using EEG data robustly and preserve the ordering corresponding to RTs. BDrank model is based on the regularized alternative ordinal classification comparing to regular regression based practices. Furthermore, a transition matrix is introduced to characterize the reliability of each channel used in EEG data, which helps in learning brain dynamics preferences only from informative EEG channels. In order to handle large-scale EEG signals and obtain higher generalization, an online-generalized Expectation Maximum (OnlineGEM) algorithm also has been proposed to update BDrank in an online fashion. Comprehensive empirical analysis on EEG signals from 44 participants shows that BDrank together with OnlineGEM achieves substantial improvements in reliability while simultaneously detecting possible less informative and noisy EEG channels.

The paper proposes an algorithm for mental fatigue monitoring, relating a subjects' EEG signals to their reaction time (RT) during a simulated driving task, as an ordinal regression problem. The authors argue that RTs could be heavily skewed and/or non-smooth, making traditional regression approaches unstable due to outlier values. They propose a brain dynamic ranking algorithm,  BDrank, using a generalized EM algorithm to estimate its parameters, and compare it to support vector regression and Logistic Ordinal Regression, where they show improved performance by accuracy and root mean squared error (RMSE) over a database of 44 subjects.
General comments, in no particular order:
1. There are some minor grammatical errors throughout. The paper could benefit from another read-through to correct these errors.
2. It is unclear to me how the model works at test time; as the model is essentially building a relational structure in the data, does the user have to provide multiple EEG trials at time of prediction?
3. There is notation early on in the paper that doesn't appear to be appropriately defined. For example, Equation (1) describes two sets of propositions, with M1 and M2 elements, respectively. How is M1 and M2, the total set of propositions, calculated? It appears to be all pair-wise comparisons of RTs but then it's unclear why there are two indexes associated with them. Also, what does it mean for the orderings to be significant? (i.e.: that the "type-1 preference propositions that the orderings between the RTs are significant"). The authors then switch to a new notation x^1 and x^2 without defining them. Notational problems also persist throughout the paper, making it hard to gauge what is being done at each step.
4. The authors describe using an FFT to transform the EEG data into the frequency domain. I'm assuming they are doing the FFT on the entire 10-s interval but the paper does not make this clear. Also, the authors state using EEG power between 0-30Hz for their analysis; do they further sub-divide this range (for example, to the standard theta/alpha/beta power ranges) or just use the power across the entire 0-30Hz band?
5. I am concerned with the relatively sparse set of comparison algorithms the authors use. The authors only compare to relatively simple approaches (support vector regression and logistic ordinal regression), yet they cite many previous works in this area but do not compare against them, instead just leaving a pretty generic statement of "The regression assumption of this method between EEG signals and RT is not correct"; they do not elaborate on this aspect.

Overall I think there is limited novelty in the approach; the idea to learn the structure of the data relationally instead of absolutely is pretty straight-forward, and is a standard practice for example in non-parametric statistical modeling. I am also not positive that ICLR is the best venue for this work; perhaps a better avenue for this would be in a more BCI/neural engineering-focused venue.


COST-SENSITIVE ROBUSTNESS AGAINST ADVERSARIAL EXAMPLES
Several recent works have developed methods for training classifiers that are certifiably robust against norm-bounded adversarial perturbations. These methods assume that all the adversarial transformations are equally important, which is seldom the case in real-world applications. We advocate for cost-sensitive robustness as the criteria for measuring the classifier's performance for tasks where some adversarial transformation are more important than others. We encode the potential harm of each adversarial transformation in a cost matrix, and propose a general objective function to adapt the robust training method of  to optimize for cost-sensitive robustness. Our experiments on simple MNIST and CIFAR10 models with a variety of cost matrices show that the proposed approach can produce models with substantially reduced cost-sensitive robust error, while maintaining classification accuracy.

** review score incremented following discussion below **
Strengths:
Well written and clear paper
Intuition is strong: not all source-target class pairs are as beneficial to find adversarial examples for
Weaknesses:
Cost matrices choices feel a bit arbitrary in experiments
CIFAR experiments still use very small norm-balls
The submission builds on seminal work by Dalvi et al. (2004), which studied cost-sensitive adversaries in the context of spam detection. In particular, it extends the approach to certifiable robustness introduced by Wong and Kolter with a cost matrix that specifies for each pair of source-target classes whether the model should be robust to adversarial examples that are able to take an input from the source class to the target (or conversely whether these adversarial examples are of interest to an adversary).
While the presentation of the paper is overall of great quality, some elements from the certified robustness literature could be reminded in order to ensure that the paper is self-contained. For instance, it is unclear how the guaranteed lower bound is derived without reading prior work. Adding this information in the present submission would make it easier for the reader to follow not only Sections 3.1 and 3.2 but also the computations behind Figure 1.b.
The experiments results are clearly presented but some of the details of the experimental setup are not always justified. If you are able to clarify the following choices in your rebuttal, this would help revise my review. First, the choice of cost matrices feels a bit arbitrary and somewhat cyclical. For instance, binary cost matrices for MNIST are chosen according to results found in Figure 1.b, but then later the same bounds are used to evaluate the performance of the approach. Yet, adversarial incentives may not be directly correlated with the “hardness” of a source-target class pair as measured in Figure 1.b. The real-valued cost matrices are better justified in that respect. Second, would you be able to provide additional justification or analysis of the choice of the epsilon parameter for CIFAR-10? For MNIST, you were able to improve the epsilon parameter from epsilon=0.1 to epsilon=0.2 but for CIFAR-10 the epsilon parameter is identical to Wong et al. Does that indicate that the results presented in this paper do not scale beyond simple datasets like MNIST?
Minor comments:
P2: The definition of adversarial examples given in Section 2.2 is a bit too restrictive, and in particular only applies to the vision domain. Adversarial examples are usually described as any test input manipulated by an adversary to force a model to mispredict.
P3: typo in “optimzation”
P5: trade off -> trade-off
P8: the font used in Figure 2 is small and hard to read when printed.


Despite its omnipresence in robotics application, the nature of spatial knowledge and the mechanisms that underlie its emergence in autonomous agents are still poorly understood. Recent theoretical work suggests that the concept of space can be grounded by capturing invariants induced by the structure of space in an agent's raw sensorimotor experience. Moreover, it is hypothesized that capturing these invariants is beneficial for a naive agent trying to predict its sensorimotor experience. Under certain exploratory conditions, spatial representations should thus emerge as a byproduct of learning to predict. We propose a simple sensorimotor predictive scheme, apply it to different agents and types of exploration, and evaluate the pertinence of this hypothesis. We show that a naive agent can capture the topology and metric regularity of its spatial configuration without any a priori knowledge, nor extraneous supervision.

This very interesting paper is based on the sensorimotor contingency theory, which grounds the perception of the agent (motor perception and sensory perception) in the capacity to learn to predict future sensory experience and to build a compact internal representation of proprioception and motor states. They show that through active experience (exploration) of an environment, an agent can encode its motor state in a way that captures both the topology and the relative distances. The authors show that in the case of an environment consistent with sensorimotor transitions and with some changes in the environment that are not inconsistent with the sensorimotor transitions (resets?), the network can learn the representation h of motor that project to the actual position of the agent in the environment (albeit in very contrived 2D toy tasks).
The model does not rely on RL at all; rather, it uses the representation of motor states (proprioception) to make predictions about the sensory observations of the environment. If such an agent were to act, I presume that there would be a search procedure across motor states to make the agent reach a desired state. The paper merits publication at ICLR provided very extensive revisions are made, and I am listing here the improvements to be made.
It is cool to cite Kant, Poincaré and Nicod, whose philosophical work underlies subsequent work on representing space and sensory experience. When citing Kant, please cite the primary source, not the 1998 re-edition.
There are missing references to Wayne et al (2018) "Unsupervised Predictive Memory in a Goal-Directed Agent" arXiv:1803.10760 and Ha & Schmidhuber "World Models" arXiv:1803.10122, where an agent is shown to build a representation of the world that can be decoded into spatial position and even a map of obstacles, on previously unseen environments, using only prediction of images, rewards and actions. Please include them in your work as they considerably change the narrative of section 2 (related work). Essentially, while the claims of the paper are interesting and relevant for the representation learning community, similar work has already been done, at much larger scale, from visual observations, using RL and self-supervised learning.
Section 2 is also somewhat overly critical of previous work: in (Banino et al, 2018; Cueva & Wei, 2018) "rely[ing] on extraneous spatial supervision signals [do not] counter any claim of autonomy", first because these signals can come from sensory perception (e.g., smell) and also because the agent is still autonomous at test time. Similarly, the depth prediction task in (Mirowski et al, 2017) is rather intuitive (stereo-vision).
Part 3 is difficult to parse: it would help to use the word proprioception (or explain why it is inappropriate) when talking about motor states, and exteroception (sensing the environment). I understood the first assumption, which is local continuity in sensory and motor space as well as the ambiguity of redundant motor systems that can generate the same sensory states, but not the second assumption. From what I understand, there are two invariants in building the model of proprioception: invariance to the topology of the environment and to the distances between objects, but then this is hard to reconcile with the setup of (in)consistent transitions in a moving environment and consistent transitions in a static environment. Please rewrite this section in a way that is easier to parse for people who know state-space models and RL for navigation and grasping (who may be your audience). Moreover, all the references point to a single work, which suggests that it is a very peculiar way of approaching a much more general problem of sensorimotor prediction, and therefore begs for a clear and simple explanation.
The architecture of the model is interesting: typically deep RL papers encode the sensory observations s into a hidden representation h, to take actions and produce a motor state m. Here, the current and future motor states m_t and m_{t+1} are embedded into h_t and h_{t+1} using a siamese MLP and used, in combination with the current sensory observation s_t, to make a prediction of s_{t+1}. This is somewhat related to learning the dynamics in model-based RL; please look into and cite Pathak et al (2017) "Curiosity-driven exploration by self-supervised prediction", ICML and other work on intrinsic curiosity.
The experiments are in very simplistic 2D grid world environments, but it makes the analysis and understanding of the 3D representations h much more simpler to follow. On the other hand, the discrete world task is very contrived (especially the weird mapping from m to h and from p to s) and hard to relate to existing work. One difficult problem that is solved by (Banino et al, 2018) is that of path integration in 2D from egocentric velocity. Could the authors present results on such a nonlinear case?
Revision: Score updated from 6 to 7.


Training neural networks is traditionally done by sequentially providing random mini-batches sampled uniformly from the entire dataset. In our work, we show that sampling mini-batches non-uniformly can both enhance the speed of learning and improve the final accuracy of the trained network. Specifically, we decompose the problem using the principles of curriculum learning: first, we sort the data by some difficulty measure; second, we sample mini-batches with a gradually increasing level of difficulty. We focus on CNNs trained on image recognition. Initially, we define the difficulty of a training image using transfer learning from some competitive "teacher" network trained on the Imagenet database, showing improvement in learning speed and final performance for both small and competitive networks, using the CIFAR-10 and the CIFAR-100 datasets. We then suggest a bootstrap alternative to evaluate the difficulty of points using the same network without relying on a "teacher" network, thus increasing the applicability of our suggested method. We compare this approach to a related version of Self-Paced Learning, showing that our method benefits learning while SPL impairs it.

This problem of interest in this paper is Curriculum Learning (CL), in the context of deep learning in particular. CL refers to learning a non-random order of presenting the training examples to the learner, typically with easier examples presented before difficult ones, to guide learning more effectively. This has been shown to both speed up learning and lead to better generalization, especially for more challenging problems. In this paper, they claim that their contribution is to decompose the problem of CL into learning two functions: the scoring function and the pacing function, with the role of the former being to estimate the difficulty of each training example and the latter to moderate the schedule of presenting increasingly more challenging examples throughout training.
Overall, I found it hard to understand from reading the paper what exactly is new versus what is borrowed from previous work. In particular, after reading Weinshall et al, I realized that they have already proposed a number of things that are experimented with here: 1) they proposed the approach of transfer learning from a previously-trained network as a means of estimating the ‘scoring function’. 2) they also distinguish between learning to estimate the difficulty of examples, and learning the schedule of decreasing difficulty throughout learning, which is actually stated here as the contribution of this paper. In particular, in Section 3 of Weinshall et al, there is a sub-section named “scheduling the appearance of training examples” where they describe what in the terminology of this paper would be called their pacing function. They experiment with two variants: fixed, and adaptive, which are very similar to two of the pacing functions proposed here.
Bootstrapping:
A component of this work that didn’t appear in Weinshall et al, is the bootstrapping approach to estimating the scoring function. In general, this involves using the same network that is being trained on the task to estimate the difficulty of the training examples. The authors explain that there are two ways to do this: estimate how easy each training example is with respect to the ‘current hypothesis’ (the weights of the network at the current step), and with respect to the ‘final hypothesis’, which they estimate if I understand correctly as the network at the end of training. The latter would necessitate first training the network in the standard way, and then using it to estimate how easy or hard each example is, and using those estimates to re-train the network from scratch using that curriculum. They refer to the former as self-paced learning and to the latter as self-taught learning. I find these names confusing in that they don’t really convey what the difference is between the two. Further, while self-paced learning has been studied before (e.g. Kuman et al), I’m not sure about self-taught learning. Is this a term that the authors here coined? If not, it would be useful to add a reference.
Using easy / hard examples as judged by the current / final hypothesis:
When using the current hypothesis, under some conditions, Weinshall et al showed that choosing harder examples is actually more beneficial than easy examples, similar in spirit to hard negative mining. On the other hand, when using the final hypothesis to estimate examples’ difficulty, using a schedule of increasing difficulty is beneficial. Based on this, I have two comments: 1) It would therefore be useful to implement a version that uses the current hypothesis to estimate how easy each example is (like the self-paced scoring function) but then invert these estimates, in effect choosing the most challenging instead of the easiest ones as is done for anti-curriculum learning. This would be a hybrid between the current self-paced scoring function and anti-curriculum scoring function that would essentially implement the hard negative mining technique in this context. 2) It would be useful to comment on the differences between the self-paced scoring function used here, and that in Kumar et al. In particular, in this case using a curriculum based on this scoring function seems to harm training but in Kumar et al, they showed it actually increased performance in a number of different cases. Why does one work but the other doesn’t?
Experiments:
The experiments are presented in a subset of 5 classes from CIFAR-10 (also used by Weinshall et al.), but also in the full CIFAR-10 and CIFAR-100 datasets. They used both a small CNN (same as in Weinshall et al) as well as a VGG architecture. Overall, their results are comparable to what was previously known: using a curriculum computed by transfer leads to improved learning speed and final performance (though sometimes very slightly) compared to the standard training, and the training with a random curriculum. Further, the benefit is larger when the task is harder (as measured by the final vanilla-trained performance). By computing the distances between the gradients obtained from using a curriculum (via the transfer scoring function) and no curriculum confirms that these two training setups indeed drive the learning in different directions; an analysis similar to Weinshall et al. Also, since, as was previously known and they also observe, the benefit of CL is larger at the beginning of training, they propose a single-step pacing function that performs similarly to other pacing functions while is simpler and more computationally effective. The idea is to decrease only once the proportion of easy examples used in mini-batches, via a step function. Therefore at the start many easy examples are used, and after this threshold is surpassed, few easy examples are used.

Overall, I don’t feel the contribution of this paper is large enough to recommend acceptance. The main points that guided this decision are:
1) The relationship with previous work is not clear. In particular, Weinshall et al seems to have already proposed a few components that are claimed to be the contribution of this paper, as elaborated on above. The authors should mention that the transfer scoring function was borrowed from Weinshall et al, clarify the differences between their pacing functions from those in Weinshall et al., etc.
2) The usefulness of using easy or hard experiments when consulting the current or final hypothesis is discussed but not explored sufficiently. An additional experiment is proposed above to add another ‘data point’ to this discussion.
3) self-paced learning is presented as something that doesn’t work and wasn’t expected to work. However, in the past successes were shown with this method, so it would be useful to clarify the difference in setup, and justify this difference.
4) It seems that the experiments resulted to similar conclusions to what was already known. While it’s useful to confirm these findings on additional datasets, I didn’t feel that there was a significant insight gained from them.


PREVENTING POSTERIOR COLLAPSE WITH δ-VAES
Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering the training objective. We develop an alternative that utilizes the most powerful generative models as decoders, optimize the variational lower bound, and ensures that the latent variables preserve and encode useful information. Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate our method's efficacy at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32.

General:
The paper attacks a problem of the posterior collapse that is one of the main issues encountered in deep generative models like VAEs. The idea of the paper relies on introducing a constraint on the family of variational posteriors in such a way that the KL term could be controlled.
The authors propose to use a linear autoregressive process (AR(1)) as the prior. Alternatively, they trained a single-layer LSTM network with conditional-Gaussian outputs as the prior (the auxiliary prior). Additionally, the authors claim that the encoder should contain anti-causal dependencies in order to introduce additional bias that may diminish the posterior collapse.
The experiments present various results on image and text datasets. Interestingly, the proposed techniques allowed to perform on a par with purely autoregressive models, however, the latent variables were utilized (i.e., no posterior collapse). For instance, in Figure 3(a) we can notice that a decoder is capable of generating similar images for given latent variable. A similar situation is obtained for text data (e.g., Figure 12).
In general, I find the paper interesting and I believe it should be discussed during ICLR.
Pros:
+ The paper is well-written and all ideas are clearly presented.
+ The idea of “hard-coded” constraints is interesting and constitutes an alternative approach to utilizing either quantized values in the VAE (VQ-VAE) or a constrained family of variational posteriors (e.g., Hyperspherical VAE).
+ The obtained results are convincing. Additionally, I would like to highlight that at the first glance it might seem that there is no improvement over the autoregressive models. However, the proposed approach allows to encode an image or a document and then decode it. This is not a case for purely autoregressive models.
+ The introduction of the Slow Features into the VAE framework constitutes an interesting direction for future research.
Cons:
- The quality of Figure 4 is too low.
- I am not fully convinced that the auxiliary prior is significantly better than the AR(1) prior. Indeed, the samples seem to be a bit better for the aux. prior but it is rather hard to notice by inspecting quantitative metrics.
- In general, the proposed approach is a specific solution rather than a general framework. Nevertheless, I find it very interesting with a potential for future work.


Learning world dynamics has recently been investigated as a way to make reinforcement learning (RL) algorithms to be more sample efficient and interpretable. In this paper, we propose to capture an environment dynamics with a novel forward model that leverages recent works on adversarial learning and visual control. Such a model estimates future observations conditioned on the current ones and other input variables such as actions taken by an RL-agent. We focus on image generation which is a particularly challenging topic but our method can be adapted to other modalities. More precisely, our forward model is trained to produce realistic observations of the future while a discriminator model is trained to distinguish between real images and the model's prediction of the future. This approach overcomes the need to define an explicit loss function for the forward model which is currently used for solving such a class of problem. As a consequence, our learning protocol does not have to rely on an explicit distance such as Euclidean distance which tends to produce unsatisfactory predictions. To illustrate our method, empirical qualitative and quantitative results are presented on a real driving scenario, along with qualitative results on Atari game Frostbite.

This paper describes an approach for training conditional future frame prediction models, where the conditioning is with respect to the current frame and additional inputs - specifically actions performed in a reinforcement learning (RL) setting.
The authors suggest that one can predict future frames from a vector comprised of an observation encoding and an action. To train the model, they suggest using a linear combination of three different losses: (1) an adversarial loss that encourages the generated sample to look similarly to training data, (2) an InfoGAN-inspired loss that is supposed to maximise mutual information between the conditioning (e.g. action) and the generated sample, and (3) a content loss, taken to be the mean-squared error of the prediction and ground-truth in the VGG feature space.
The major contribution of this work seems to be using these three losses in conjunction, while doing conditional frame prediction at the same time. While interesting, there exist very similar approaches that also use adversarial losses [1] as well as approaches using different means to reach the same goal [2, 3]. None of these are mentioned in the text, nor evaluated against. It is true that [1] is not action-conditional, but adding actions as conditioning could be a simple extension.
Experimental section consists of an ablation study, which evaluates importance of different components of the loss, and a qualitative study of model predictions. With no comparison to state of the art (e.g. [1, 3]), it is hard to gauge how valuable this particular approach is.
The qualitative evaluation starts with §4.4¶1 “we follow the customary GAN literature to include some qualitative results for illustration”, as if there was no other reason for including samples than to follow the custom. Since the paper is about action-conditional prediction, it would be interesting to see predictions conditioned on the same initial sequence but different actions, which are not present, however. Moreover, this work is developed in the context of RL applications, and since prior art [4] has shown that better predictive models do not necessarily lead to better RL results, it would be interesting to evaluate the proposed approach against baselines in an RL setting.
The paper is clearly written, but some claims in the text are not supported by any citations (e.g. §1¶2 “More recently, several papers have shown that forward modelling…” without a citation).  Some claims are misleading (e.g. §1¶3 says that by using adversarial training we don’t need to use task-specific losses and it does not put constraints on input modality. While true, using MSE loss is equally general). Some other claims are not supported at all or may not be true (e.g. §3.2¶1 “ResNet … aims at compressing the information in the raw observation” - to the best of my knowledge, there is no evidence for this).
To conclude, the suggested approach is not novel, the experimental evaluation is lacking, and the text contains a number of unsupported statements. I recommend to reject this paper.
[1] Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., & Levine, S. (2018). Stochastic Adversarial Video Prediction. CoRR, abs/1804.01523.
[2] Eslami, S.M., Rezende, D.J., Besse, F., Viola, F., Morcos, A.S., Garnelo, M., Ruderman, A., Rusu, A.A., Danihelka, I., Gregor, K., Reichert, D.P., Buesing, L., Weber, T., Vinyals, O., Rosenbaum, D., Rabinowitz, N.C., King, H., Hillier, C., Botvinick, M.M., Wierstra, D., Kavukcuoglu, K., & Hassabis, D. (2018). Neural scene representation and rendering. Science, 360, 1204-1210.
[3] Denton, E.L., & Fergus, R. (2018). Stochastic Video Generation with a Learned Prior. ICML.
[4] Buesing, L., Weber, T., Racanière, S., Eslami, S.M., Rezende, D.J., Reichert, D.P., Viola, F., Besse, F., Gregor, K., Hassabis, D., & Wierstra, D. (2018). Learning and Querying Fast Generative Models for Reinforcement Learning. CoRR, abs/1802.03006.


Deep Neural Networks (DNNs) are increasingly deployed in cloud servers and autonomous agents due to their superior performance. The deployed DNN is either leveraged in a white-box setting (model internals are publicly known) or a black-box setting (only model outputs are known) depending on the application. A practical concern in the rush to adopt DNNs is protecting the models against Intellectual Property (IP) infringement. We propose BlackMarks, the first endto-end multi-bit watermarking framework that is applicable in the black-box scenario. BlackMarks takes the pre-trained unmarked model and the owner's binary signature as inputs. The output is the corresponding marked model with specific keys that can be later used to trigger the embedded watermark. To do so, BlackMarks first designs a model-dependent encoding scheme that maps all possible classes in the task to bit '0' and bit '1'. Given the owner's watermark signature (a binary string), a set of key image and label pairs is designed using targeted adversarial attacks. The watermark (WM) is then encoded in the distribution of output activations of the DNN by fine-tuning the model with a WM-specific regularized loss. To extract the WM, BlackMarks queries the model with the WM key images and decodes the owner's signature from the corresponding predictions using the designed encoding scheme. We perform a comprehensive evaluation of BlackMarks' performance on MNIST, CIFAR-10, ImageNet datasets and corroborate its effectiveness and robustness. BlackMarks preserves the functionality of the original DNN and incurs negligible WM embedding overhead as low as 2.054%.

summary:
The paper proposes an approach for model watermarking (i.e., watermarking a trained neural neteowrk). The watermark is a bit string, which is embedded in the model as part of a fine-tuning procedure, and can be decoded from the network from the model's specific predictions for a specific set of inputs (called keys) chosen during the fine-tuning step. The process generates a watermark when we can be confident that a model that didn't go through the exact same fine-tuning procedure gives significantly different predictions on the set of keys. The application scenario is when a company A wants to deploy a model for which A has IP ownership, and A wants to assess whether a competitor is (illegaly) re-using A's model. The approach presented in the paper works in the black-box setting, meaning that whether a model posesses the watermark can be assessed only by querying the model (i.e., without access to the internals of the model).
The overall approach closely follows Merrer et al. (2017), but extends this previous work to multi-bit watermarking. The similarity with Merrer et al. is that keys are generated with a procesdure to generate adversarial examples, watermarking is performed by specifically training the network to give the source label (i.e., the label of the image from which the adversarial example has been generated). The differences with Merrer et al. lie in the fact that each key encoded a specific bit (0 or 1), and the multi-bit watermark is encoded in the predictions for all keys (in case of a multi-class classifier, the labels are first partitionned into two clusters to map each class to either 0 or 1). In contrast, Merrer et al. focused on "zero-bit" watermarking, meaning that all keys together are only used to perform a test of whether the model has been watermarked (not encode the watermark). Another noticeable difference with Merrer et al. is in step 4 of the algorithm, in which several unmarked models are generated to select better key images.
comments:
While overall the approach makes sense and most of the design decisions seem appropriate, many questions are only partly addressed. My main concerns are:
1- the watermarks are encoded in adversarial examples for which the trained model gives the "true" label (i.e., the watermark is embedded in adversarial examples on which the model is robust). The evaluation does not address the concerns of false alarms on models trained to be robust to adversarial examples. Previous work (e.g., Merrer et al.) study at least the effect of fine-tuning with adversarial examples..
2- A watermark of length K is encoded in K images, and the test for watermarking is "The owner can prove the authorship of the model if the BER is zero.". This leaves little room to model manipulation. For instance, the competitor could randomize its predictions once in a while (typically output a random label for one out of K inputs), with very small decrease in accuracy and yet would have a non-negligible probability of having a non-zero BER.
other comments:
1- overhead section: in step 4 of the algorithm, there is a mention of "construct T unmarked models": why aren't they considered in the overhead? This seems to be an extremely significant part of the cost (the overall cost seems to be more T times the cost of building a single unmarked model rather than a few percent)
2- step 2 page 4: "The intuition here is that we want to filter out the highly transferable WM keys": I must have misunderstood something here. Why are highly transferable adversarial examples a problem? That would be the opposite: if we want the key to generate few false alarms (i.e., we do not want to claim ownership of a non-watermarked model), then we need the adversarial examples to "transfer" (i.e., be adversarial for non-watermarked models), since the watermarked model predicts the source class for the key. Merrer et al. (2017) on the contrary claim " As such adversaries seem to generalize across models [...] , this frontier tweaking should resist model manipulation and yield only few false positives (wrong identification of non marked model).", which means that transferability of adversarial examples is a fundamental assumption underlying the approach.
3- under Eq. 1: "Note that without the additional regularization loss (LWM), this retraining procedure resembles ‘adversarial training’ (Kurakin et al., 2016).": I do not understand that sentence. Without L_{WM}, the loss is the usual classification loss (L_0), and has nothing to do with adversarial training.
4- more generally, the contribution of the paper is on multi-bit watermarking, but there is no clear application scenario/experiment  where the multi-bit is more useful than the zero-bit watermarking.


BACKPROPAMINE: TRAINING SELF-MODIFYING NEU- RAL NETWORKS WITH DIFFERENTIABLE NEUROMODU- LATED PLASTICITY
The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.

This work presents Backpropamine, a neuromodulated plastic LSTM training regime. It extends previous research on differentiable Hebbian plasticity by introducing a neuromodulatory term to help gate information into the Hebbian synapse. The neuromodulatory term is placed under network control, allowing it to be time varying (and hence to be sensitive to the input, for example). Another variant proposes updating the Hebbian synapse with modulated exponential average of the Hebbian product. This average is linked to the notion of an eligibility trace, and ties into some recent biological work that shows the role of dopamine in retroactively modulating synaptic plasticity.
Overall the work is nicely motivated and clearly presented. There are some interesting ties to biological work -- in particular, to retroactive plasticity phenomena. There should be sufficient details for a reader to implement this model, thought there are some minor details missing regarding the experimental setup, which will be addressed below.
The authors test their model on three tasks: cue-award association, maze learning, and Penn Treebank (PTB). In the cue-award association task the retroactive and simple modulation networks perform well, while the non-modulated and non-plastics fail. For the maze navigation task the modulated networks perform better than the non-modulated networks, though the effect is less pronounced. Finally, on PTB the authors report improvements over baseline LSTMs.
One of the main claims of this paper is that neuromodulated plastic LSTMs...outperform standard LSTMs on a benchmark language modeling task”, and that therefore “differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks”. This claim is unfortunately unfounded for a very important reason: the LSTM performance is not at all close to that which can be achieved by LSTMs in general. The authors cite such models in the appendix (Melor et al), but claim that “much larger models” are needed, potentially with other mechanisms, such as dropout. Though this may be true, these models still undermine the claim that “neuromodulated plastic LSTMs...outperform standard LSTMs on a benchmark language modeling task”. This claim is simply not true, and more care is needed in reporting the results here in the wider context of the literature. Also, I am left wondering what are considered the parameters of the models -- are only the neuromodulatory terms considered as the additional trainable parameters compared to baseline LSTMs? How are the Hebbian synapses themselves considered in this calculation? If the Hebbian synapses are not considered, then the authors need a control with matched memory-capacities to account for the extra capacity afforded by the Hebbian synapses. Given the ties between Hebbian synapses and attention (see Ba et al), an important control here could be an LSTM with Bahdanau (2014) style attention.
Finally, the style (font) of the paper does not adhere to the ICLR style template, and must be changed.
Overall, the ideas presented in the paper are intriguing, and further research down this line is encouraged. However, in its current state the work lacks sufficiently strong baselines to support the paper’s claims; thus, the merits of this approach cannot yet be properly assessed.