Conventional reinforcement learning rarely considers how the physical variations in the environment (eg. mass, drag, etc.) affect the policy learned by the agent. In this paper, we explore how changes in the environment affect policy generalization. We observe experimentally that, for each task we considered, there exists an optimal environment setting that results in the most robust policy that generalizes well to future environments. We propose a novel method to exploit this observation to develop robust actor policies, by automatically developing a sampling curriculum over environment settings to use in training. Ours is a model-free approach and experiments demonstrate that the performance of our method is on par with the best policies found by an exhaustive grid search, while bearing a significantly lower computational cost.

The paper looks at the problem of generalization across physical parameter varaition in learning for continuous control. The paper presents a method to develop a sampling based curriculum over env. settings for training robust agents.
* The paper makes an interesting observation on inadvertent generalization in robust policy learning.
However, the examples in both the cartpole and the pendulum cases seem not to be watertight.
For instance, the authors claim that
But from a dynamical system perspective in both cases, the controller is operating near limits.
The solution and subsequent generalization depend more on the topology of the solution space.
A heavy Pendulum is an overdamped system and required the policy to operate at the limits of action to generate momentum for swing up. Hence a solution for a lighter pendulum in implicitly included. Similarly, the rolling ball is an underdamped system, and where the policy operates near zero limits in light ball case to prevent the system from going unstable. Adding mass results in damping which makes it easier. In this case, as well the solution space is implicitly contained.
But this is not a novel observation. Similar observations have been made for Robust control and Model-Reference Adaptive Control.
The paper also overlooks a number of related works in model-free randomization [4], adaptive randomization [3], adversarial randomization [5,6]. The method also does not compare with model-based methods for adaptive policy learning and iLQR based methods to handle this problem [2, 7].
The argument that the method is model-free is perhaps not as acceptable since the model parameters need to be known apriori for adaptation. The policy itself may be model-free but that is a design choice.
A good experimental evaluation for this is generalization across known unknowns and unknown unknowns.
* The algorithm itself is reasonable but the problem setup and choice of a discrete dynamics parameter choices are questionable. The bandit style method operates over a discrete decision set.
It also assumes in the multi-parameter setting that they are independent, which may not be true very often.
The algorithm proposed itself isnt novel, but would have been justified if the results supported the use of such a method.
* Experiments are quite weak.
Both the experimental domains are rather simplistic with smooth nonlinear dynamics. There are more sophisticated and interesting continuous control environments such as control suite [1] or manipulation suite [2].
It would be useful to see how tis method works in more complicated domains and how the performance compares with simpler methods such as joint brute-force randomization both in performance and in computation.
Questions:
1. Please provide details of Algorithm 1. How are the quantities K and M related?
2. What is the process of task initialization? What information is required and what priors are used. Uniform prior over what range?
In summary, the authors explore an interesting adaptive curriculum design method. However, in its current form, the work needs more thought and empirical evaluation for the sake of completeness.
References:
1. Model Reference Adaptive Control [https://doi.org/10.1007/978-1-4471-5102-9_116-1
]
2. ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems [https://arxiv.org/abs/1707.04674]
3. EPOpt: Learning Robust Neural Network Policies Using Model Ensembles [https://arxiv.org/abs/1610.01283]
4. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
[https://arxiv.org/abs/1610.01283]
5. Certifying Some Distributional Robustness with Principled Adversarial Training [https://arxiv.org/pdf/1710.10571.pdf]
6. Adversarially Robust Policy Learning: Active Construction of Physically-Plausible Perturbations [http://vision.stanford.edu/pdf/mandlekar2017iros.pdf]
7. Synthesis and Stabilization of Complex Behaviors through Online Trajectory Optimization [https://homes.cs.washington.edu/~todorov/papers/TassaIROS12.pdf]


THE LIMITATIONS OF ADVERSARIAL TRAINING AND THE BLIND-SPOT ATTACK
The adversarial training procedure proposed by  is one of the most effective methods to defend against adversarial examples in deep neural networks (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the "blind-spot attack", where the input images reside in "blind-spots" (low density regions) of the empirical distribution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including (Kolter & Wong, 2018) and (Sinha et al., 2018) because these trainable robustness certificates can only be practically optimized on a limited set of training data.

This paper provides some insights on influence of data distribution on robustness of adversarial training. The paper demonstrates through a number of analysis that the distance between the training an test data sets plays an important role on the effectiveness of adversarial training. To show the latter, the paper proposes an approach to measure the distance between the two data sets using combination of nonlinear projection (e.g. t-SNE), KDE, and K-L divergence. The paper also shows that under simple transformation to the test dataset (e.g. scaling), performance of adversarial training reduces significantly due to the large gap between training and test data set. This tends to impact high dimensional data sets more than low dimensional data sets since it is much harder to cover the whole ground truth data distribution in the training dataset.
Pros:
- Provides insights on why adversarial training is less effective on some datasets.
- Proposes a metric that seems to strongly correlate with the effectiveness of adversarial training.
Cons:
- Lack of theoretical analysis. It could have been nice if the authors could show the observed phenomenon analytically on some simple distribution.
- The marketing phrase "the blind-spot attach" falls short in delivering what one may expect from the paper after reading it. The paper would read much better if the authors better describe the phenomena based on the gap between the two distribution than using bling-spot. For some dataset, this is beyond a spot, it could actually be huge portion of the input space!
Minor comments:
- I believe one should not compare the distance shown between the left and right columns of Figure 3 as they are obtained from two different models. Though the paper is not suggesting that, it would help to clarify it in the paper. Furthermore, it would help if the paper elaborates why the distance between the test and training dataset is smaller in an adversarially trained network compared to a naturally trained network.
- Are the results in Table 1 for an adversarially trained network or a naturally trained network? Either way, it could be also interesting to see the average K-L divergence between an adversarially and a naturally trained network on the same dataset.
- Please provide more visualization similarly to those shown in Fig 4.


Object recognition in real-world image scenes is still an open problem. With the growing number of classes, the similarity structures between them become complex and the distinction between classes blurs, which makes the classification problem particularly challenging. Standard N-way discrete classifiers treat all classes as disconnected and unrelated, and therefore unable to learn from their semantic relationships. In this work, we present a hierarchical inter-class relationship model and train it using a newly proposed probability-based loss function. Our hierarchical model provides significantly better semantic generalization ability compared to a regular N-way classifier. We further proposed an algorithm where given a probabilistic classification model it can return the input corresponding super-group based on classes hierarchy without any further learning. We deploy it in two scenarios in which super-group retrieval can be useful. The first one, selective classification, deals with the problem of low-confidence classification, wherein a model is unable to make a successful exact classification. The second, zero-shot learning problem deals with making reasonable inferences on novel classes. Extensive experiments with the two scenarios show that our proposed hierarchical model yields more accurate and meaningful super-class predictions compared to a regular N-way classifier because of its significantly better semantic generalization ability.

SUMMARY
The paper presents a method for classification which takes into account the semantic hierarchy of output labels, rather than treating them as independent categories. In a typical classification setup, the loss penalizes the KL-divergence between the model’s predicted label distribution and a one-hot distribution placing all probability mass on the single ground-truth label for each example. The proposed method instead constructs a target distribution which places probability mass not only on leaf category nodes but also on their neighbors in a known semantic hierarchy of labels, then penalizes the KL-divergence between a model’s predicted distribution and this target distribution. This model is used for classification on ImageNet-1k, and for zero-shot classification on ImageNet-21k where a model must predict superclasses seen during training for images of leaf categories not seen during training.
Pros:
- Method is fairly straightforward
- Modeling relationships between labels is an important problem
Cons:
- Missing references to key prior work in this space
- Minimal comparison to prior work
- Confusing experimental setup
- Paper is difficult to read
MISSING REFERENCES
This paper is far from the first to consider the use of a semantic hierarchy to improve classification systems; see for example:
Deng et al, “Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition”, CVPR 2012
Deng et al, “Large-scale object classification using label relation graphs”, ECCV 2014 (Best Paper)
Jiang et al, “Exploiting feature and class relationships in video categorization with regularized deep neural networks”, TPAMI 2017
None of these are cited in the submission. [Deng et al, 2014] is particularly relevant, as it considers not just “is-a” relationships as in this submission, but also mutual exclusion relationships between categories. Without citation, discussion, and comparison with some of these key pieces of prior work, the current submission is incomplete.
COMPARISON TO PRIOR WORK
The only direct comparison to prior work in the paper is the comparison to DeViSE on ILSVRC12 classification performance in Table 3. However since DeViSE was intended to be used for zero-shot learning and not traditional supervised classification, this comparison seems unfair.
Instead the authors should compare their method against DeViSE and ConSE for zero-shot learning. Indeed, in Section 4.3 the authors construct a test set “in a [sic] same manner defined in Frome et al” but do not actually compare against this prior work.
I suspect that the authors chose not to perform this comparison since unlike DeViSE and ConSE their method cannot predict category labels not seen during training; instead it is constrained to predicting a known supercategory when presented with an image of a novel leaf category. As such, the proposed method is not really “zero-shot” in the sense of DeViSE and ConSE.
EXPERIMENTAL SETUP
From Section 3.1, “we adopt a subset of ImageNet the ILSVRC12 dataset which gather [sic] 1K classes [...]”. The 1000 category labels in ILSVRC12 are mutually exclusive leaf nodes; when placed in the context of the WordNet hierarchy there are 820 internal nodes between these leaves and the WordNet root. As a result, for the method to make sense I assume that all models must be trained to output classification scores for all 1820 categories rather than the 1K leaf categories. This should be made more explicit in the paper, as it means that none of the performance metrics reported in the paper are comparable to other results on ILSVRC12 which only measure performance on the 1K leaf categories.
The experiments on zero-shot learning are also confusing. Rather than following the existing experimental protocol for evaluating zero-shot learning from [Frome et al, 2013] and [Norouzi et al, 2013] the authors evaluate zero-shot learning by plotting SG-hit vs SG-specificity; while these are reasonable metrics, they make it difficult to compare with prior work.
POOR WRITING
The paper is difficult to follow, with confusing notation and many spelling and grammatical errors.
OVERALL
On the whole, the paper addresses an important problem and presents a reasonable method. However due to the omission of key references and incomplete comparison to prior work, the paper is not suitable for publication in its current form.


We present Generative Adversarial Privacy and Fairness (GAPF), a data-driven framework for learning private and fair representations of the data. GAPF leverages recent advances in adversarial learning to allow a data holder to learn "universal" representations that decouple a set of sensitive attributes from the rest of the dataset. Under GAPF, finding the optimal decorrelation scheme is formulated as a constrained minimax game between a generative decorrelator and an adversary. We show that for appropriately chosen adversarial loss functions, GAPF provides privacy guarantees against strong information-theoretic adversaries and enforces demographic parity. We also evaluate the performance of GAPF on multi-dimensional Gaussian mixture models and real datasets, and show how a designer can certify that representations learned under an adversary with a fixed architecture perform well against more complex adversaries.

The paper authors provide a good overview of the related work to Private/Fair Representation Learning (PRL). Well written, The theoretical approach is extensively explained and the first sections of the paper are easy to follow. The authors demonstrate the model performance on or the GMM, the comparison between theoretical and data driven performance is a good case study to understand the PRL.
We usually expect to see related work in the first sections, in this case it's has been put just before the conclusion. It can be still justified by the need o introduce the  PRL concepts before comparing with other works.
The GMM study case is interesting, but incorporates strong assumptions. Moreover, for a 4 or 8 dimensional GM, 20K data points are more than enough to infer the correct parameter. It would have been more useful if it was used to comapre between the mentioned methods in "Related Work".
There seems to be important parts of the paper that has been put in the appendices: how to solve the constrained problem, Algorithm.... Similarly, some technical details were expanded in the paper body (Network structure).
The authors mentioned the similarities with other works and their model choices that set theirs apart from other. Yet, the paper doesn't provide performance ( accuracy, MI) comparison to other works. There seems to be a strong similarity with Censoring representations with an adversary, Harrison Edwards and Amos Storke (link: https://arxiv.org/abs/1511.05897). Difference : distortion instead of H divergence, non-generative autoencoders.
Consequently, I question the novelty of the paper's contribution. Without extensive comparison with other methods and especially to similar ones mentioned in the related work, there is little to say about the "state-of-the-artness". Yet, it is important to acknowledge the visible effort behind the paper and how the author(s) managed to leverage the simplicity and power of GANs.
On a lighter note:
A)- the paper mention "state-of-the-art CNNs, state-of-the-art entropy estimators, MI, generative models", for the Machine Learning community, many of these elements have been around for a while now.
B)- "Observe that the hard constraint in equation 2 makes our minimax problem different from what is extensively studied in the machine learning community": I would argue it's not an objective statement.


The goal of the paper is to propose an algorithm for learning the most generalizable solution from given training data. It is shown that Bayesian approach leads to a solution that dependent on statistics of training data and not on particular samples. The solution is stable under perturbations of training data because it is defined by an integral contribution of multiple maxima of the likelihood and not by a single global maximum. Specifically, the Bayesian probability distribution of parameters (weights) of a probabilistic model given by a neural network is estimated via recurrent variational approximations. Derived recurrent update rules correspond to SGD-type rules for finding a minimum of an effective loss that is an average of an original negative log-likelihood over the Gaussian distributions of weights, which makes it a function of means and variances. The effective loss is convex for large variances and non-convex in the limit of small variances. Among stationary solutions of the update rules there are trivial solutions with zero variances at local minima of the original loss and a single non-trivial solution with finite variances that is a critical point at the end of convexity of the effective loss in the mean-variance space. At the critical point both first-and second-order gradients of the effective loss w.r.t. means are zero. The empirical study confirms that the critical point represents the most generalizable solution. While the location of the critical point in the weight space depends on specifics of the used probabilistic model some properties at the critical point are universal and model independent.

Summary: Non-convex learning problems can have multiple solutions with different generalization properties, thus it is important to find solutions that generalize well. The goal of this paper is to derive an algorithm for finding a solution to the learning problem with the best possible generalization properties. This is achieved by using a Bayesian approach in which the parameters w (e.g., weights of a network) are random variables and the effective loss (integral wrt to w) is minimized in lieu of the usual loss. The paper assumes that each component of the weight vector w is Gaussian and derives a formula for updating the mean and covariance of said Gaussians (this is an SGD method). The paper claims that the resulting effective loss is convex for large variances (sigma > threshold), nonconvex for small variances (sigma < threshold), and converges to original loss as sigma goes to zero. The paper also claims that when sigma=0 there are trivial solutions that are unstable as data changes, but that when sigma=threshold (assuming this is what is meant by end of convexity) there are non-trivial solutions that are less sensitive to data changes and hence the most generalizable.
Comments: the goal of the paper (finding minima that generalize well) is an excellent one. But the paper is not clearly written and appears to oversell the contribution. In particular, the title speaks about SGD, dropout, generalization and critical points “at the end of convexity”. Naturally, a reader is inclined to think that the paper will study SGD and dropout for deep learning and analyze generalization properties of the solutions found by those methods. In reality, there is very little in the paper about SGD, dropout, and generalization. The connection with SGD is merely because the method for updating mu and sigma is an SGD method. The connection with dropout is mentioned in passing in one paragraph and it is not very clear. The connection with generalization is claimed but never quite explained (there are no generalization bounds in the paper). As far as understand, the paper considers the minimization of the effective loss, uses a Gaussian approximation for computing the effective loss, and focuses primarily on the characterization of convexity as a function of sigma as well as a characterization of the critical points depending on whether the effective loss is convex (sigma above a threshold) or not (sigma below the threshold). The main claim appears to be that critical points at the critical threshold lead to solutions that generalize well, but a detailed explanation of why this is the case isn't given. If my digest of the paper is the correct one, then modifying the title, abstract and intro to make this clear would have helped a lot.
Beyond the high-level lack of clarity about the contribution of the paper, the writing lacks precision and rigor, and many things are undefined (though one can figure them out after reading many times back and forth). Specifically:
1) It is not explained why the probability of each training sample can be expressed as a product of factors close to 1, with the product taken over the epochs.
2) It is not explained why each factor can be modeled as a product of Gaussians
3) At nearly the top of page 3, a product over n is substituted by a product over t, with x_n replaced by x_t and so forth, but the total number of products goes up from N to TxN. What is the value of y_{NT} and x_{NxT}? Do the authors mean that y_n should have been replaced by y_{n,t} and we now have two indices? Or do the authors mean that the same mini batch of N samples is reused, and so indices should be corrected accordingly?
4) It is not clear why replacing R_t/Q_{t+1} by 1 is an adequate approximation.
5) At the top of equation 4, there is a product, but no index wrt which the product is taken. Right after it says the index is t, but there is no t in the expression. Should mu_0 be mu_t and similarly for sigma?
In short, a promising direction, but the contribution of the paper appears to be over claimed and the writing of the paper needs significant improvement before the paper can be accepted for publication.


A key feature of neural networks, particularly deep convolutional neural networks, is their ability to "learn" useful representations from data. The very last layer of a neural network is then simply a linear model trained on these "learned" representations. Despite their numerous applications in other tasks such as classification, retrieval, clustering etc., a.k.a. transfer learning, not much work has been published that investigates the structure of these representations or indeed whether structure can be imposed on them during the training process. In this paper, we study the effective dimensionality of the learned representations by models that have proved highly successful for image classification. We focus on ResNet-18, ResNet-50 and VGG-19 and observe that when trained on CIFAR10 or CIFAR100, the learned representations exhibit a fairly low rank structure. We propose a modification to the training procedure, which further encourages low rank structure on learned activations. Empirically, we show that this has implications for robustness to adversarial examples and compression.

This paper introduces a new way to have more compressed (lower rank) representation of the data in a supervised fashion. The authors motivates their work by saying that such representation are more useful for transfer learning and are more robust to adversarial examples! In order to achieve this, the authors introduce a virtual LR layer and utilize Nystrom technique to make the process more efficient.  The idea introduced in this paper is interesting but the paper is poorly written and organized which makes its through evaluation difficult. Below, I provide more detailed comments.
I don't understand at what frequency the low-rank optimization as the  subproblem to Equation (OPT) is being done. Given that the DNN training  is being done using batched of examples, where do you put the low-rank  optimization, in the end of each epoch? It seems it is in the end of each epoch but it should be clearly stated.
I don't understand motivation for L_N(A). The authors justify it by  saying that a trivial solution for the optimization problem would be setting A+b=0 and they introduce this term to avoid it. However, this is not correct. Note that there are n examples in A and we can only make one of them  zero at a time. (Note that talking about A+b is not also accurate  because they are not of the same size).
In order to use Nystrom method for low-rank approximation, W needs to be  a symmetric postive semi-definite matrix. I am not sure if the heuristic  procedure introduced in Page 4 is well-justifed. and whether we are  still optimizing the objective function introduced in Page 3. Do we  still have a stable training?
What is ResNet N-LR in experiment? The authors introduced ResNet 1-LR and ResNet 2-LR but not ResNet N-LR! I found the description N-LR later but the naming is rather confusing. I would use LR instead of N-LR because it seems it has N LR layer. I would also explain this next to other methods.
Not sure if I understand Bottle-LR. The description in the text is not clear and I don’t understand the motivation for this baseline! Again, this should be described next to other methods.
The authors do not mention what is their setting for r in the experiments (Table 1).
Page 5, Paragraph after Table 1: First CIFAR-100 should be CIFAR-10.
One way the authors defend their framework is to have a representation that can be used in transfer learning. Nonetheless, the results in Table 1.b shows that their framework is not doing good for transfer learning.
Not sure if I understand Figure 1. How do you change the number of singular values? I understand this is a hyper parameter for your framework but I am confused how it is being set for N-LR method. Similarly, I don’t understand Table 2 and how you change the embedding dimension.
Unlike the claim made by the authors, it seems that VGG19 N-LR does better compared to VGG19 2-LR in Figure 2.


To be effective in sequential data processing, Recurrent Neural Networks (RNNs) are required to keep track of past events by creating memories. Consequently RNNs are harder to train than their feedforward counterparts, prompting the developments of both dedicated units such as LSTM and GRU and of a handful of training tricks. In this paper, we investigate the effect of different training protocols on the representation of memories in RNN. While reaching similar performance for different protocols, RNNs are shown to exhibit substantial differences in their ability to generalize for unforeseen tasks or conditions. We analyze the dynamics of the network's hidden state, and uncover the reasons for this difference. Each memory is found to be associated with a nearly steady state of the dynamics whose speed predicts performance on unforeseen tasks and which we refer to as a 'slow point'. By tracing the formation of the slow points we are able to understand the origin of differences between training protocols. Our results show that multiple solutions to the same task exist but may rely on different dynamical mechanisms, and that training protocols can bias the choice of such solutions in an interpretable way.

Post-rebuttal update:
The authors have clarified their main messages, and the paper is now less vague about what is being investigated and the conclusions of the experiments. The same experimental setup has been extended to use CIFAR-10 as an additional, more realistic dataset, the use of potentially more powerful LSTMs as well as GRUs, and several runs to have more statistically significant results - which addresses my main concerns with this paper originally (I would have liked to see a different experimental setup as well to see how generalisable these findings are, but the current level is satisfying). Indeed, these different settings have turned up a bit of an anomaly with the GRU on CIFAR-10, which the authors claim that they will leave for future work, but I would very much like to see addressed in the final version of this paper. In addition some of the later analysis has only been applied under one setting, and it would make sense to replicate this for the other settings (extra results would have to fit into the supplementary material).
I did spot one typo on page 4 - "exterme", but overall the paper is also better written, which helps a lot. I commend the authors on their work revising this paper and will be upgrading my rating to accept.
---
The authors investigate the hidden state dynamics of RNNs trained on a single task that mixes (but clearly separates) pattern recognition and memorisation. The authors then introduce two curricula specific to the task, and study how the trained RNNs behave under different deviations from the training protocol (generalisation). They show that under the curriculum that exhibited the best generalisation, there exist more robust (persisting for long time periods) fixed/slow points in the hidden state dynamics. They then extend the optimisation procedure developed by Sussillo & Barak for continuous-time RNNs in order to find these points. Finally, they use this method to track the speed of these points during the course of training, and link spikes in speed to one of the curricula which introduces new classes over time.
Understanding RNNs - and in particular how they might "generalise" - is an important topic of research. As done previously, studying RNNs as dynamical systems is a principled way to do so. In this line of work some natural objects to look into are fixed points and even slow points (Sussillo & Barak) - how long they can persist, and how large the basins of attraction are. While I believe the authors did a reasonable job following this through, I have some concerns about the experimental setup. Firstly, only one task is used - based on object classification with images - so it is unclear how generalisable these findings are, given that the authors' setup could be extended to cover at least another task, or at least another dataset. MNIST is a sanity check, and many ideas may fail to hold when extended to slightly more challenging datasets like CIFAR-10.
Secondly, as far as I can tell, the results are analysed on one network per setting, so it is hard to tell how significant the differences are. While some analyses may only make sense for single networks, e.g. Figure 3, a proper quantification of some of the results over several training runs would be appropriate.
Finally, it is worth investigating LSTMs on this task. This is not merely because they are more commonly used than GRUs, but they are strictly more powerful - see "On the Practical Computational Power of Finite Precision RNNs for Language Recognition", published at ACL 2018. Given the results in this paper and actually the paper that first introduces the forget gate for LSTMs, it seems that performing these experiments solely with GRUs might lead to wrong conclusions about RNNs in general.
There are also more minor spelling and grammatical errors throughout the text that should be addressed. For example, there is a typo on the task definition on page 2 - "the network should *output* a null label."


Adversarial examples, referred to as augmented data points generated by imperceptible perturbation of input samples, have recently drawn much attention. Wellcrafted adversarial examples may even mislead state-of-the-art deep models to make wrong predictions easily. To alleviate this problem, many studies focus on investigating how adversarial examples can be generated and/or resisted. All the existing work handles this problem in the Euclidean space, which may however be unable to describe data geometry. In this paper, we propose a generalized framework that addresses the learning problem of adversarial examples with Riemannian geometry. Specifically, we define the local coordinate systems on Riemannian manifold, develop a novel model called Adversarial Training with Riemannian Manifold, and design a series of theory that manages to learn the adversarial examples in the Riemannian space feasibly and efficiently. The proposed work is important in that (1) it is a generalized learning methodology since Riemmanian manifold space would be degraded to the Euclidean space in a special case; (2) it is the first work to tackle the adversarial example problem tractably through the perspective of geometry; (3) from the perspective of geometry, our method leads to the steepest direction of the loss function. We also provide a series of theory showing that our proposed method can truly find the decent direction for the loss function with a comparable computational time against traditional adversarial methods. Finally, the proposed framework demonstrates superior performance to the traditional counterpart methods on benchmark data including MNIST, CIFAR-10 and SVHN.

The authors argue that they propose a method to find adversarial examples, when the data lie on a Riemannian manifold. In particular, they derive a perturbation, which is argued to be the worst perturbation for generating an adversarial example, compared to the classical Euclidean derivation.
I strongly disagree with the proposed (Riemannian) geometric analysis, because there are several technical mistakes, a lot of arbitrary considerations, and flawed assumptions. In particular, my understanding is that the proposed method is not related at all with Riemannian geometry. For justification I will comment some parts:
#1) In Section 1 paragraph 4, and in Section 2.3 after Eq. 14, the sentences about the gradient of a function that is defined on a manifold are strange and unclear. In general, the gradient of a function defined on a manifold, points to the ascent direction. Thus, if I understood correctly the sentences in the paper support that the gradient of such a function is meaningless, so I think that they are wrong.
#2) How the $\ell_2$-ball on a manifold is defined? Usually, we consider a ball on the tangent space, since this is the only Euclidean space related to the manifold. Here, my understanding is that the authors consider the ball directly on the manifold. This is clearly wrong and undefined.
#3) To find the geodesic you have to solve a system of 2nd order non-linear ODEs, and there are additional details which I will not include  here, but can be easily found in the Riemannian geometry literature. Also, I think that the Lemma 2.2. is wrong, since the correct quantity of Eq. 3 is $ds^2 = g_ij(t)d\theta^i d\theta^j dt^2$, where $dt\rightarrow 0$ based on the included proof. This is clearly not a sensible geodesic, it is just the infinitesimal length of a line segment when $t\rightarrow 0$, which means that the two points are infinitesimally close.
#4) If $x, y$ is on a Riemannian manifold then the $x+y$ operator does not make sense, so Eq. 7 is wrong. In particular, for operations on Riemannian manifolds you need to use the exponential and the logarithmic map.
#5) Continuing from #4). Even if we consider the perturbation to be sufficiently small, still the $x+\epsilon$ is not defined. In addition, the constraint in Eq. 8 is wrong, because the inner product related to the Riemannian metric has to be between tangent vectors. Here the $\epsilon$ is an arbitrary quantity, since it is not defined where it actually lies. In general, the derivation here is particularly confusing and not clear at all. In my understanding the constraint of Eq. 8 is a purely linear term, and $d$ is not the geodesic distance on a manifold. It just represents the Mahanalobis distance between the points $x$ and $x+\epsilon$, for a matrix $G$ defined for each $x$, so it is a linear quantity. So Eq. 9 just utilizes a precondiner matrix for the classical linear gradient.
#6) The Eq. 12 is very flawed, since it equalizes a distance with the Taylor approximation error. I think that this is an unrealistic assumption, since these terms measure totally different quantities. Especially, if $d$ is the geodesic distance.
#7) The upper bound in inequality Eq. 13 comes from Eq. 12, and it is basically an assumption for the largest absolute value of the Hessian's eigenvalues. However, this is not discussed in the text, which are the implications?
#8) I find the paper poorly written, and in general, it lacks of clarity. In addition, the technical inconsistencies makes the paper really hard to follow and to be understood. I mentioned above only some of them. Also, there are several places where the sentences do not make sense (see #1), and the assumptions made are really arbitrary (see #6). The algorithms are not easy to follow. Minor comments, you can reduce the white spaces by putting inline Eq. 3, 4, 5, 6, 9, 10, 14, and Figure 2. The notation is very inconsistent, since it is very unclear in which domain/space each quantity/variable lies. Also, in Section 2.5. the authors even change their notation.
In my opinion, the geometrical analysis and the interpretation as a Riemannian manifold is obviously misleading. Apart from the previously mentioned mistakes/comments, I think that the proposed approach is purely linear. Since actually the Eq. 14 implies that the linear gradient of the loss function, is just preconditioned with the Hessian matrix of the loss function with respect to the input $x$. Of course, if this function is convex around $x$, then this quantity is the steepest ascent direction of the loss function, simply on the Euclidean space where $x$ lies. However, when this function is not convex, I am not sure what is the behavior when all the eigenvalues of the Hessian are set to their absolute value. Also, the (arbitrary) constraint in Eq. 13 implicitly sets a bound to the eigenvalues of the Hessian, which in some sense regularizes the curvature of the loss function. To put it simple, I think that the proposed method, is just a way to find a preconditioner matrix for the linear gradient of the loss function, which points to the steepest direction. This preconditioner is based on the Hessian of the loss function, where the absolute values of the eigenvalues are used, and also, are constrained to be bounded from above based on a given value.
Generally, in my opinion the authors should definitely avoid the Riemannian manifold consideration. I believe that they should change their perspective, and consider the method simply as a local preconditioner, which is based on the Hessian and a bound to its (absolute) eigenvalues. They should also discuss what is the implication by setting the eigenvalues to their absolute values. However, since I am not an expert in the field of adversarial examples, I am not sure how novel is this approach.


DECOUPLED WEIGHT DECAY REGULARIZATION
L 2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L 2 regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

This review has been somewhat challenging to complete. As the authors write, this work has already been impactful and motivated a great deal of further research. The empirical evaluation is convincing and the results have been reproduced and further studied by others. A moderate amount of space in the paper (Section 3, Section 4.5) is used to refer to work motivated by the paper itself. While I do not take issue with this I believe it should be considered for the final decision (in the sense that disentangling the contributions of the authors and related work becomes tricky). With this said, I continue with my review.
Paper summary: The authors observe that L2 regularization is not effective when using the Adam optimizer. By replacing L2 regularization with decoupled weight decay the authors are able to close the generalization gap between SGD and Adam and make Adam more robust to hyperparameter settings. The empirical evaluation is comprehensive and convincing.
Detailed comments:
1) The authors emphasize the fact that L2 regularization and weight decay are not the same for different optimizers and claim that this goes against the belief of some practitioners. In my experience, most practitioners would not be surprised by this observation itself. The second observation made by the authors, that L2 regularization is not effective in Adam, is the more interesting (and perhaps surprising) observation.
2) I am not convinced of the importance of Proposition 3. In practice, adaptive methods will have a preconditioner which depends locally on the parameters. I understand the motivation from the previous paragraph but felt that the formal result added little.
3) Section 3 introduced the Bayesian filtering perspective of stochastic optimization. The authors share the observation of Aitchison, 2018 that decoupled weight decay can be recovered in this framework. My interpretation is that this observation is important _because_ of the empirical observations in this paper and does not necessarily provide theoretical support for the approach. However, the last paragraph of Section 3 seems to utilize the Bernstein-von Mises theorem to promote the idea that with large datasets the prior distribution is unimportant (and is ignored). I am not sure that I follow this argument. For example, this claim seems to be completely independent of the optimization algorithm used and moreover Propositions 1,2, and 3 are independent of the data distribution. I suspect that this confusion is due to a misunderstanding on my part and would appreciate clarification from the authors.
4) The empirical evaluation in this paper is very strong and these practical techniques have already been adopted by the community in addition to spurring novel research. The empirical observation broadly explores two directions: decoupled weight decay leads to separable hyperparameter search spaces (meaning optimization is less sensitive to hyperparameters), and decoupled weight decay gives improved generalization (and training performance). Both claims are explored throughly with strong evidence given for the improvement due to AdamW.
Overall, I find this paper to be presented well and with convincing empirical results. I feel that the theoretical justification for decoupling weight decay are a little weak, and believe that other work is moving towards better explanations then the ones presented in this paper [1,2,3]. Despite this, I believe that this paper should be accepted.
Minor comments:
- I find the notation in the paper confusing in general. x is used to denote weights, and w to denote hyperparameters (e.g. w' for L2 regularization scale and w for weight decay scale). I don't see why it wouldn't be preferable to use the more standard W for weights, x for inputs, and lambda for hparams.
- Figure 4: it is difficult to distinguish between Adam and SGDWR (especially left).
Clarity: The paper is well written and clear. I find the notation confusing in places, but is consistent throughout.
Originality: This paper presents original findings but occasionally relies on work motivated by itself to convince the reader of its importance. I do not think that this subtracts from the value of the work.
Significance: The work is clearly significant. Even without knowing that practitioners have adopted the techniques presented in this work, the paper clearly distinguishes itself with strong empirical results.


While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is challenging and often requires substantial interactions with the environment. Further, a wide variety of domains have dynamics that share common foundations like the laws of physics, which are rarely exploited by these algorithms. Humans often acquire such physics priors that allow us to easily adapt to the dynamics of any environment. In this work, we propose an approach to learn such physics priors and incorporate them into an RL agent. Our method involves pre-training a frame predictor on raw videos and then using it to initialize the dynamics prediction model on a target task. Our prediction model, SpatialNet, is designed to implicitly capture localized physical phenomena and interactions. We show the value of incorporating this prior through empirical experiments on two different domainsa newly created PhysWorld and games from the Atari benchmark, outperforming competitive approaches and demonstrating effective transfer learning.

Quality: The paper proposed a new method to learn some physics prior in an environment along with a new SpatialNetwork Architecture. Instead of learning a specific dynamics model, they propose to learn a dynamics model that is action-free, purely learning the extrinsic dynamics.  They formulate this problem as a video prediction problem. A series of experiments are conducted on PhysWorld (a new physics based simulator) and a subset of Atari games.
Clarity: The writing is good.
Originality: This work is original as most of the model-based RL works are focusing on learning one environment instead of common rules of physics.
significance of this work: This work propose an interesting direction to pursue.
cons:
1. In Figure 4, the authors show that a pretrained model can learn faster than random initialization. However, it is hard to ablate the factor that causes this effect.  Does the dynamics predictor learn the physics priors or is it just because it learn the visual prior of the shape of the objects, etc?
2. The baseline for atari games is quite limited. First of all, 3 out of 5  atari games  in the original PPO paper show that ACER performs better than PPO. (asteroid, breakout, DemonAttack). I think it is better to make some improvement upon state-of-the-art methods.
3. All the experiments are shown with only 3 random seeds, without error bar in the main paper. Although the reward plots are shown in Figure 11.
4. 5 out of 10 atari games are similar to PPO (according to Figure 11). It's hard to be conclusive when half of the experiments are positive and the rest are not.
5. Lack of discussion about ego-dynamics. There are physics priors for both the environment and the controller. Usually the controller/agent  requires an action to predict its dynamics. Then why should we omit the ego-dynamics and only model the outer world.
6. Physics prior usually happen in physical environment. The proposed method works well in the physworld environments. But is there some task that are more realistic than atari games that can leverage the power of physics priors more? It's good that this method works in some atari games. But isn't learning the dynamics of atari games a bit off the topic?
7. The transfer learning experiments should contain a baseline -- maml/reptile. Since you are learning physics prior, it is fair to add meta-learning baselines for comparison.
I think the direction is interesting and the effort is made well. But the experiments are less convincing than the abstract/introduction.


Imitation learning provides an appealing framework for autonomous control: in many tasks, demonstrations of preferred behavior can be readily obtained from human experts, removing the need for costly and potentially dangerous online data collection in the real world. However, policies learned with imitation learning have limited flexibility to accommodate varied goals at test time. Model-based reinforcement learning (MBRL) offers considerably more flexibility, since a predictive model learned from data can be used to achieve various goals at test time. However, MBRL suffers from two shortcomings. First, the model does not help to choose desired or safe outcomes -its dynamics estimate only what is possible, not what is preferred. Second, MBRL typically requires additional online data collection to ensure that the model is accurate in those situations that are actually encountered when attempting to achieve test time goals. Collecting this data with a partially trained model can be dangerous and time-consuming. In this paper, we aim to combine the benefits of imitation learning and MBRL, and propose imitative models: probabilistic predictive models able to plan expert-like trajectories to achieve arbitrary goals. We find this method substantially outperforms both direct imitation and MBRL in a simulated autonomous driving task, and can be learned efficiently from a fixed set of expert demonstrations without additional online data collection. We also show our model can flexibly incorporate user-supplied costs at test-time, can plan to sequences of goals, and can even perform well with imprecise goals, including goals on the wrong side of the road.

# Summary
This submission proposes a method to combine the benefits of model-based RL and Imitation Learning (IL) for navigation tasks. The key idea is to i) learn a prior over trajectory distributions from a fixed dataset of demonstrations, and ii) use this learned dynamical model for path planning via probabilistic inference. Reaching target waypoints is done by maximizing the trajectory likelihood conditioned on the planning goal. The prior is learned using R2P2 on LIDAR features and past positions. Experiments using the CARLA driving simulator show that this method can outperform standard control, IL, and model-based RL baselines, while flexibly incorporating test-time goals and costs thanks to its probabilistic formulation.
# Strengths
The method is an elegant way to get the best of both worlds in RL and IL, leveraging the recent R2P2 work to estimate a powerful sequential model used for planning via probabilistic inference. The flexibility of the method in considering test-time cost maps and user-defined goals (e.g. to avoid potholes) is appealing, especially since it does not require on-policy data collection.
The proposed planning-as-inference method can in theory handle the multi-modality present in human demonstrations by using a probabilistic model of the observed behaviors as prior over undirected expert trajectories.
The approach seem to outperform both model-based and imitation learning baselines on a simplified version of the CARLA benchmark, including on interesting fine-grained metrics (e.g., comfort based).
# Weaknesses
The main weakness of this submission lies in its experimental evaluation, especially the absence of any dynamic objects in the tested environment ("static world CARLA", section 1). It is unclear how this approach would generalize beyond just staying on the road. How would it handle traffic lights, pedestrians, other drivers, weather variations, and more complex driving tasks than waypoint following by traversing mostly free space? How does the prior generalize to more complex behaviors (e.g, by using more contextual information \phi)? How robust is the method to noise in the demonstrations, i.e. non-expert or suboptimal behavior? It seems that estimating the generative prior on human behavior might suffer from the same issues as behavior cloning, e.g., the sample inefficiency due to the combinatorial explosion of causal factors explaining complex human behaviors. It might be in fact even harder to estimate that generative model than use a direct discriminative approach (e.g., a modular pipeline), at the cost of reduced flexibility at test time of course. The currently reported sample efficiency (7000 training samples) and near perfect success rate seem to suggest that this (non-standard) version of the CARLA benchmark is too simple (no weather variations, no dynamic obstacles). Comparison to the state of the art (beyond the baselines implemented here) on the original CARLA benchmark seems needed (especially in the "Nav. dynamic" task).
The method is only described very succinctly in section 2. I do not believe there are enough details (especially around the learning algorithm, hyper-parameters, and other important technical elements) for reproducibility at this stage. Section 2.1 is also quite dense for people not familiar with the R2P2 paper. As the main contribution of the paper is to leverage that model for planning and control, it would be great to maybe discuss a bit deeper. Finally, the input modalities are not clear, especially for the baselines: the proposed method is using LIDAR and localization whereas the IL baseline seems to use vision (while the others just use the trajectory). This makes the fairness of the comparison really unclear (LIDAR is a much stronger signal for just staying on the road).
Minor remarks:
- Why use a proportional controller as a baseline instead of the standard PID one?
- Section 2.3 seems like it's missing the extension of equation 2 to the multi-goal case?
- Typos in section 3 ("trail-and-error"), section 4 ("autonmous", "knowledge to")
# Recommendation
Although the theoretical benefits of the method are well-motivated and clear (off-policy learning, probabilistic model, flexibility at test time), the experimental evaluation (custom simple CARLA test, unclear comparison to baselines) and lack of details impeding reproducibility seems to suggest that this submission needs a bit more work. First, adding more details as suggested above and clarifying the experimental protocol seem like a must, but can be easily addressed by an update to the text. Second, it would be ideal to evaluate the approach on the standard CARLA benchmark in order to compare fairly to the prior art. This is much more involved.
I personally like the approach, so although I think it is marginally below the acceptance threshold in its current form, I reserve my judgement for the time being and look forward to the authors' reply.
# Update
The submission has been drastically rewritten (the diff is massive) and I think it is in much better shape, answering some of my concerns around reproducibility and generalization. Furthermore, it reinforces the strengths of the approach (esp. around its flexibility).
I am willing to recommend acceptance, but I have some further questions (hence I have only updated my score to a 6 for now). They are mostly related to the comparison with IL (important to validate the claim in the paper that the proposed approach is quantitatively better than both existing IL and RL methods). See discussion below for details.


Current end-to-end deep learning driving models have two problems: (1) Poor generalization ability of unobserved driving environment when diversity of training driving dataset is limited (2) Lack of accident explanation ability when driving models don't work as expected. To tackle these two problems, rooted on the believe that knowledge of associated easy task is benificial for addressing difficult task, we proposed a new driving model which is composed of perception module for see and think and driving module for behave, and trained it with multi-task perception-related basic knowledge and driving knowledge stepwisely. Specifically segmentation map and depth map (pixel level understanding of images) were considered as what & where and how far knowledge for tackling easier drivingrelated perception problems before generating final control commands for difficult driving task. The results of experiments demonstrated the effectiveness of multitask perception knowledge for better generalization and accident explanation ability. With our method the average sucess rate of finishing most difficult navigation tasks in untrained city of CoRL test surpassed current benchmark method for 15 percent in trained weather and 20 percent in untrained weathers.

# Summary
This submission proposes a multi-task convolutional neural network architecture for end-to-end driving (going from an RGB image to controls) evaluated using the CARLA open source simulator. The architecture consists of an encoder and three decoders on top: two for perception (depth prediction and semantic segmentation), and one for driving controls prediction. The network is trained in a two-step supervised fashion: first training the encoder and perception decoders (using depth and semantic segmentation ground truth), second freezing the encoder and training the driving module (imitation learning on demonstrations). The network is evaluated on the standard CARLA benchmark showing better generalization performance in new driving conditions (town and weather) compared to the CARLA baselines (modular pipeline, imitation learning, RL). Qualitative results also show that failure modes are easier to interpret by looking at predicted depth maps and semantic segmentation results.
# Strengths
Simplicity of the approach: the overall architecture described above is simple (cf. Figure 1), combining the benefits of the modular and end-to-end approaches into a feed-forward CNN. The aforementioned two-stage learning algorithm is also explained clearly. Predicted depth maps and semantic segmentation results are indeed more interpretable than attention maps (as traditionally used in end-to-end driving).
Evaluation of the driving policy: the evaluation is done with actual navigation tasks using the CARLA (CoRL'18) benchmark, instead of just off-line behavior cloning accuracy (often used in end-to-end driving papers, easier to overfit to, not guaranteed to transfer to actual driving).
Simple ablative analysis: Table 2 quantifies the generalization performance benefits of pretraining and freezing the encoder on perception tasks (esp. going from 16% to 62% of completed episodes in the new town and weather dynamic navigation scenario).
# Weaknesses
## Writing
I have to start with the most obvious one. The paper is littered with typos and grammatical errors (way too many to list). For instance, the usage of "the" and "a" is almost non-existent. Overall, the paper is really hard to read and needs a thorough pass of proof-reading and editing. Also, please remove the acknowledgments section: I think it is borderline breaking the double-blind submission policy (I don't know these persons, but if I did that would be a breach of ICLR submission policy). Furthermore, I think its contents are not very professional for a submission at a top international academic venue, but that is just my opinion.
## Novelty
This is the main weakness for me. The architecture is very close to at least the following works:
- Xu, H., Gao, Y., Yu, F. and Darrell, T., End-to-end learning of driving models from large-scale video datasets (CVPR'17): this reference is missing from the paper, whereas it is very closely related, as it also shows the benefit of a segmentation decoder on top of a shared encoder for end-to-end driving (calling it privileged training);
- Codevilla et al's Conditional Imitation Learning (ICRA'18): the only novelty in the current submission w.r.t. CIL is the addition of the depth and segmentation decoders;
- Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V., Driving Policy Transfer via Modularity and Abstraction (CoRL'18): the architecture also uses a shared perception module and segmentation (although in a mediated way instead of auxiliary task) to show better generalization performance (including from sim to real).
Additional missing related works include:
- Kim, J. and Canny, J.F., Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention (ICCV'17): uses post-hoc attention interpretation of "black box" end-to-end networks;
- Sauer, A., Savinov, N. and Geiger, A., Conditional Affordance Learning for Driving in Urban Environments (CoRL'18): also uses a perception module in the middle of the CIL network showing better generalization performance in CARLA (although a bit lower than the results in the current submission).
- Pomerleau, D.A., Alvinn: An autonomous land vehicle in a neural network (NIPS'89): the landmark paper for end-to-end driving with neural networks!
## Insights / significance
In light of the aforementioned prior art, I believe the claims are correct but already reported in other publications in the community (cf. references above). In particular, the proposed approach uses a lot more strongly labeled data (depth and semantic segmentation supervision in a dataset of 40,000 images) than the competing approaches mentioned above. For instance, the modular pipeline in the original CARLA paper uses only 2,500 labeled images, and I am sure its performance would be vastly improved with 40,000 images, but this is not evaluated, hence the comparison in Table 1 being unfair in my opinion. This matters because the encoder in the proposed method is frozen after training on the perception tasks, and the main point of the experiments is to convince that it results in a great (fixed) intermediate representation, which is in line with the aforementioned works doing mediated perception for driving.
The fine-tuning experiments are also confirming what is know in the litterature, namely that simple fine-tuning can lead to catastrophic forgetting (Table 3).
Finally, the qualitative evaluation of failure cases (5.3) leads to a trivial conclusion: a modular approach is indeed more interpretable than an end-to-end one. This is actually by design and the main advocated benefit of modular approaches: failure in the downstream perception module yields failure in the upstream driving module that builds on top of it. As the perception module is, by design, outputting a human interpretable representation (e.g., a semantic segmentation map), then this leads to better interpretation overall.
## Reproducibility
There are not enough details in section 3.1 about the deep net architecture to enable re-implementation ("structure similar to SegNet", no detailed description of the number of layers, non-linearities, number of channels, etc).
Will the authors release the perception training dataset collected in CARLA described in Section 4.2?
# Recommendation
Although the results of the proposed multi-task network on the CARLA driving benchmark are good, it is probably due to using almost two orders of magnitude more labeled data for semantic segmentation and depth prediction than prior works (which is only practical because the experiments are done in simulation). Prior work has confirmed that combining perception tasks like semantic segmentation with end-to-end driving networks yield better performance, including using a strongly related approach (Xu et al). In addition to the lack of novelty or new insights, the writing needs serious attention.
For these reasons, I believe this paper is not suitable for publication at ICLR.


Biological data often contains measurements of dynamic entities such as cells or organisms in various states of progression. However, biological systems are notoriously difficult to describe analytically due to their many interacting components, and in many cases, the technical challenge of taking longitudinal measurements. This leads to difficulties in studying the features of the dynamics, for examples the drivers of the transition. To address this problem, we present a deep neural network framework we call Dynamics Modeling Network or DyMoN. DyMoN is a neural network framework trained as a deep generative Markov model whose next state is a probability distribution based on the current state. DyMoN is well-suited to the idiosyncrasies of biological data, including noise, sparsity, and the lack of longitudinal measurements in many types of systems. Thus, DyMoN can be trained using probability distributions derived from the data in any way, such as trajectories derived via dimensionality reduction methods, and does not require longitudinal measurements. We show the advantage of learning deep models over shallow models such as Kalman filters and hidden Markov models that do not learn representations of the data, both in terms of learning embeddings of the data and also in terms training efficiency, accuracy and ability to multitask. We perform three case studies of applying DyMoN to different types of biological systems and extracting features of the dynamics in each case by examining the learned model.

This paper tackles the important challenge of making sense of temporal measurements made in biological systems. Among other, those have the peculiarity that they are not independent but with a dependency structure, which can be encoded as a graph or a network. The authors claim that their approach, DyMoN is adapted to the many challenges of biological high-throughput data sets: noise, sparsity and lack of temporal resolution. The paper presents three very different use of the method in complex biological systems in Section 3: (i) Calcium imaging of visual cortex neurons, (ii) T--cell development in the thymus, and (iii) Human embryonic stem cell differentiation. Section 4 assesses the performance of the method on simulated data sets as well as on a face-recognition data set. Moreover, the authors demonstrate how the features of the NN can be interrogated to shed new insight about the process under scrutiny. They also show the gain in running time a comapred to other approaches.
Remarks:
 - I know very little about the literautr in the subject, could you clarify how your work relates to/can be distinguished from: Testolin and Zorzi, Front Comput Neurosci. 201, Kiegeskorte's Ann. Rev. Vis. Sc. 2015 (https://doi.org/10.1146/annurev-vision-082114-035447), Kai Fan's PhD work (@ Duke University), Betzel and Bassel, Interface 2017, Wang et al. bioRxiv 2018 (https://doi.org/10.1101/247577), etc.
 - l-4p2: 'repetitions': what are those? Line below: 'sufficient for estimating $ P_{x}(y) $, means large sample size, no? No contradictory? And one line below: what is the precise meaning of 'similar' (twice)?
 - top p3: line continued from bottom of p2 -> is it to rubber out noise?
what is $ n $ in $ \mathbb{R}^{n} $?
 - Make Fig 1 (B) and (C) clearer, since the transition vectors are learnt, why are they in (B) (input states)?
 - below "...distribution approximates the distribution $ \mathcal{P}_{x} $..." -> but $ P_{x} $ also depends on $ \theta $, not an issue?
 - remark at the end of Section 2: extending DyMoN to higher-order: OK, but this might be computationally VERY expensive, don't you think?
 - link how your empirical validation data have features that, even remotely resemble those of the kind of biological data sets (on which no ground truth exist, I sympathise) you focus on.


This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions. First, we demonstrate how Graph Neural Networks (GNN), which have emerged as an effective model for various supervised prediction problems defined on structured data, can be trained to produce embedding of graphs in vector spaces that enables efficient similarity reasoning. Second, we propose a novel Graph Matching Network model that, given a pair of graphs as input, computes a similarity score between them by jointly reasoning on the pair through a new cross-graph attention-based matching mechanism. We demonstrate the effectiveness of our models on different domains including the challenging problem of control-flow-graph based function similarity search that plays an important role in the detection of vulnerabilities in software systems. The experimental analysis demonstrates that our models are not only able to exploit structure in the context of similarity learning but they can also outperform domain-specific baseline systems that have been carefully hand-engineered for these problems.

The authors present two methods for learning a similarity score between pairs of graphs. They first is to use a shared GNN for each graph to produce independent graph embeddings on which a similarity score is computed. The authors improve this model using pairs of graphs as input and utilizing a cross-graph attention-mechanism in combination with graph convolution. The proposed approach is evaluated on synthetic and real world tasks. It is clearly shown that the proposed approach of cross-graph attention is useful for the given task (at the cost of extra computation).
A main contribution of the article is that ideas from graph matching are introduced to graph neural networks and it is clearly shown that this is beneficial. However, in my opinion the intuition, effect and limitations of the cross-graph attention mechanism should be described in more detail. I like the visualizations of the cross-graph attention, which gives the impression that the process converges to a bijection between the nodes. However, this is not the case for graphs with symmetries (automorphisms); consider, e.g., two star graphs. A discussion of such examples would be helpful and would make the concept of cross-graph attention clearer.
The experimental comparison is largely convincing. However, the proposed approach is motivated by graph matching and a connection to the graph edit distance is implied. However, in the experimental comparison graph kernels are used as baseline. I would like to suggest to also use a simple heuristics for the graph edit distance as a baseline (Riesen, Bunke. Approximate graph edit distance computation by means of bipartite graph matching. Image and Vision Computing, 27(7), 2009).
There are several other questions that have not been sufficiently addressed in the article.
* In Eq. 3, self-attention is used to compute graph level representations to "only focus on important nodes in the graph". How can this be reconciled with the idea of measuring similarities across the whole graph? Can you give more insights in how the attention coefficients vary for positive as well as negative examples? How much does the self-attention affects the performance of the model in contrast to mean or sum aggregation?
* Why do you chose the cross-graph similarity to be non-trainable? Might there be any benefits in doing so?
* The note on page 5 is misleading because two isomorphic graphs will lead to identical representations even if communication is not reduced to zero vectors (this happens neither theoretically nor in practice).
* Although theoretical complexity of the proposed approach is mentioned, how much slower is the proposed approach in practice? As similarity is computed for every pair of nodes across two graphs, the proposed approach, as you said, will not scale. In practice, how would one solve this problem given two very large graphs which do not fit into GPU memory? To what extent can sampling strategies be used (e.g., from GraphSAGE)? Some discussion on this would be very fruitful.
In summary, I think that this is an interesting article, which can be accepted for ICLR provided that the cross-graph attention mechanism is discussed in more detail.
Minor remarks:
* p3: The references provided for the graph edit distance in fact consider the (more specific) maximum common subgraph problem.


LATENT CONVOLUTIONAL MODELS
We present a new latent model of natural images that can be learned on large-scale datasets. The learning process provides a latent embedding for every image in the training dataset, as well as a deep convolutional network that maps the latent space to the image space. After training, the new model provides a strong and universal image prior for a variety of image restoration tasks such as large-hole inpainting, superresolution, and colorization. To model high-resolution natural images, our approach uses latent spaces of very high dimensionality (one to two orders of magnitude higher than previous latent image models). To tackle this high dimensionality, we use latent spaces with a special manifold structure (convolutional manifolds) parameterized by a ConvNet of a certain architecture. In the experiments, we compare the learned latent models with latent models learned by autoencoders, advanced variants of generative adversarial networks, and a strong baseline system using simpler parameterization of the latent space. Our model outperforms the competing approaches over a range of restoration tasks.

# Summary
The paper proposes to embed natural images in a latent convolutional space of high dimensionality to obtain a universal image prior. Concretely, each image is embedded as a custom parameter vector of a CNN, which turns random noise into the input of a universal generator network to restore the image in pixel space.
Inference for image restoration is performed by minimizing the energy of a likelihood objective while constraining the latent representation of the restored image to be part of the learned latent space. Experiments for inpainting, super-resolution, and colorization are performed to evaluate the proposed method.
# Positive
As mentioned in the paper, I agree that the idea of learning a universal image prior is appealing, since it can be applied to (m)any image restoration tasks without adjustment.
I am not very familiar with the related work, but if I understood correctly, the paper seems to combine deep latent modeling (GLO, Bojanowski et al., 2018) and deep image priors (Ulyanov et al., 2018). The experiments show good results which qualitatively appear better than those of related methods. A user study also shows that people mostly prefer the results of the proposed method.
Did you try other standard restoration tasks, such as image denoising or deblurring? If not, do you think they would work equally well?
# Limitations
While I agree that a universal image prior is valuable, the paper should (briefly) mention what the disadvantages of the proposed approach are:
- A limitation (at least as presented) is that the corruption process has to be known analytically (as a likelihood objective) and must be differentiable for gradient-based inference.
- Furthermore, the disadvantage of the universal prior as presented in the paper is that restoring an image requires optimization (e.g. gradient descent). In contrast, corruption-specific neural nets typically just need a forward pass to restore the image and are thus easier and faster to use.
# Restoration inference
- How dependent is the restoration result with respect to the initialization? For example, when starting gradient descent with the degraded image vs. a random image.
- Roughly, how many iterations and runtime is needed for inference?
- Did you try different optimizers, such as L-BFGS?


The construction of adversarial images is a search problem in high dimensions within a small region around a target image. The goal is to find an imperceptibly modified image that is misclassified by a target model. In the black-box setting, only sporadic feedback is provided through occasional model evaluations. In this paper we provide a new algorithm whose search strategy is based on an intriguingly simple iterative principle: We randomly pick a low frequency component of the discrete cosine transform (DCT) and either add or subtract it to the target image. Model evaluations are only required to identify whether an operation decreases the adversarial loss. Despite its simplicity, the proposed method can be used for targeted and untargeted attacks -resulting in previously unprecedented query efficiency in both settings. We require a median of 600 black-box model queries  to produce an adversarial ImageNet image, and we successfully attack Google Cloud Vision with 2500 median queries, averaging to a cost of only $3 per image. We argue that our proposed algorithm should serve as a strong baseline for future adversarial black-box attacks, in particular because it is extremely fast and can be implemented in less than 20 lines of PyTorch code.

This paper proposed a simple query-efficient "score-based" black-box attack based on iteratively perturbing an input image with a direction randomly sampled (w/o replacement) from a set of orthonormal bases. In particular, the authors proposed the use of low-frequency parts of DCT (discrete cosine transformation) as in (Guo 2018) to perform this task. Experimental results on ImageNet and three different classification models demonstrate the query efficiency of the proposed method -- able to achieve high attack success rate within fewer query budgets, where the visual distortion has an L2 norm threshold set to be 10. The authors also demonstrate an untargeted score-based black-box attack on Google CloudVision API.
While the results seem promising, there are several issues that may potentially weaken the query-efficient claims made in this paper, especially due to the lack of sufficient attack comparisons (on smaller datasets) and inconsistent threat models when compared to existing works. My main concerns are summarized as follows.
1. Unfair comparison due to inconsistent threat models (knowledge known to an attacker): the proposed method (simBA) is a "score-based" black-box attack, not a "decision-based" black-box attack. The proposed method assumes knowing the prediction likelihood (or prediction score) as the model output when performing black-box attacks, whereas the compared methods in black-box settings, such as Opt Attack and Boundary Attack, are "decision-based" attack that assumes only knowing the top-1 prediction label. Therefore, the query count comparison is meaningless and unfair, since these two methods require far less information from the model.
On the other hand, ZOO/AutoZOOM is a score-based attack. But ZOO can achieve a very low L2 distortion due to its coordinate descent nature. A fair comparison is to set the same L2 distortion for all score-based methods, and compare the median/avg query counts of each image to reach the same L2 distortion. The comparison to Opt-Attack / Boundary attack makes sense only if the proposed method (simBA) can also perform decision-based attack. Nonetheless, the query count to same-distortion comparison argument still holds. The authors should specify whether simBA can apply to the decision-only attack scenario. If so, how to implement and what is the performance?
Lastly, the QL attack (Ilyas 2018) can perform both score-based and decision-based attacks. So the authors should make the query comparison (to same L2 distortion) as well. According to a recent report (Table-1, NES column) in https://arxiv.org/pdf/1807.07978.pdf, the QL-attack has a comparable performance in terms of query counts as reported in this paper.
2. More experiments on targeted black-box attacks: While untargeted attacks on Imagenet is a relatively easy task, I was a bit skeptical on the attacking performance of simBA in targeted attacks - since the selection of low-frequency bases directly limits the search space of adversarial examples, as opposed to arbitrary random directions adopted in QL-attack, Boundary-attack, and Opt-attack. It is also not clear how the target label is chosen in the targeted attack experiment.
I suggest including two more experiments to validate the function of simBA: (i) compare the performance of least-likely targeted attack (ii) show results on smaller datasets such as Cifar-10. As pointed out by the authors, Imagenet has too many image dimensions and make it more vulnerable to attack. Showing attacking results on smaller datasets can properly justify the value of the proposed attack, rather than the benefit from high dimensionality.
3. Novelty relative to LFBA (Guo) should be better differentiated: The idea of using DCT is originated from the LFBA paper. Since in that paper the authors also leveraged low-frequency DCT to perform black-box attacks, it is not clear to me what makes the proposed method perform better than the LFBA paper. The novelty and difference between this paper and the LFBA paper should be addressed.
4. The Google Cloud Vision API attack is not too appealing - the tree label is still there and the trees are obviously present in the picture, while I appreciate the effect of removing the original top-3 labels. Can the authors show another set of non-trivial (more surprising) and targeted-attack experiments? Or simply do the same experiment using the same image (men snowing -> dog)  as in the QL-attack.
----
Post-rebuttal review
I appreciate the authors' efforts in clarifying some of my concerns. However, I am still not convinced the comparison has been made fair. Many numbers from Table 1, such as ZOO, Opt-attack, QL-attack and AutoZOOM seem to be directly adapted from the papers rather than implemented and reproduced based on the same setting as the proposed attack. In particular, given that QL-attack is a published work, one of the state-of-the-art method and its codes has been released, I would really love to see a direct comparison using the same data samples and threat model. I would also like to emphasize that implementing all attacks under the same setting is crucial, since different attack methods may have a different criterion to determine attack successfulness. For example, QL-attack has some pre-defined distortion (L2 or Linfinity) for determining an adversarial example is successful, in addition to a different predicted class.


ADAPTIVE GRADIENT METHODS WITH DYNAMIC BOUND OF LEARNING RATE
Adaptive optimization methods such as ADAGRAD, RMSPROP and ADAM have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGRAD to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of ADAM and AMSGRAD, called ADABOUND and AMSBOUND respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at https://github.com/Luolc/AdaBound.

The authors introduce AdaBound, a method that starts off as Adam but eventually transitions to SGD. The motivation is to benefit from the rapid training process of Adam in the beginning and the improved convergence of SGD at the end. The authors do so by clipping the weight updates of Adam in a dynamic way. They show numerical results and theoretical guarantees. The numerical results are presented on CIFAR-10 and PTB while the theoretical results are shown on assumptions similar to AMSGrad (& using similar proof strategies). As it stands, I have some foundational concerns about the paper and believe that it needs significant improvement before it can be published. I request the authors to please let me know if I misunderstood any aspect of the algorithm, I will adjust my rating promptly. I detail my key criticisms below:
- I'm somewhat confused by the formulation of \eta_u and \eta_l. The way it is set up (end of Section 4), the final learning rate for the algorithm converges to 0.1 as t goes to infinity. In the Appendix, the authors show results also with final convergence to 1. Are the results coincidental with the fact that SGD works well with those learning rates? It is a bit odd that we indirectly encode the final learning rate of the algorithm into the \eta s.
- Am I correct in saying that with t=100 (i.e., the 100th iteration), the \eta s constrain the learning rates to be in a tight bound around 0.1? If beta=0.9, then \eta_l(1) = 0.1 - 0.1 / (0.1*100+1) = 0.091. After t=1000 iterations, \eta_l becomes 0.099. Again, are the good results coincidental with the fact that SGD with learning rate 0.1 works well for this setup? In the scheme of the 200 epochs of training (equaling almost 100-150k iterations), if \eta s are almost 0.099 / 0.10099, for over 99% of the training, we're only doing SGD with learning rate 0.1.
- Along the same lines, what learning rates on the grid were chosen for each of the problems? Does the setup still work if SGD needs a small step size and we still have \eta converge to 1? A VGG-11 without batch normalization typically needs a smaller learning rate than usual; could you try the algorithms on that?
- Can the authors plot the evolution of learning rate of the algorithm over time? You could pick the min/median/max of the learning rates and plot them against epochs in the same way as accuracy.This would be a good meta-result to show how gradual the transition from Adam to SGD is.

- The core observation of extreme learning rates and the proposal of clipping the updates is not novel; Keskar and Socher (which the authors cite for other claims) motivates their setup with the same idea (Section 2 of their paper). I feel that the authors should clarify what they are proposing as novel. Is it correct that a careful theoretical analysis of this framework is what stands as the authors' major contribution?
- Can you try experimenting with/suggesting trajectories for \eta which converge to SGD stepsize more slower?
- Similarly, can you suggest ways to automate the choice for the \eta^\star? It seems that the 0.1 in the numerator is an additional hyperparameter that still might need tuning?


REGULARIZED LEARNING FOR DOMAIN ADAPTATION UNDER LABEL SHIFTS
We propose Regularized Learning under Label shifts (RLLS), a principled and a practical domain-adaptation algorithm to correct for shifts in the label distribution between a source and a target domain. We first estimate importance weights using labeled source data and unlabeled target data, and then train a classifier on the weighted source samples. We derive a generalization bound for the classifier on the target domain which is independent of the (ambient) data dimensions, and instead only depends on the complexity of the function class. To the best of our knowledge, this is the first generalization bound for the label-shift problem where the labels in the target domain are not available. Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights. Experiments on the CIFAR-10 and MNIST datasets show that RLLS improves classification accuracy, especially in the low sample and large-shift regimes, compared to previous methods.

This paper presents a new contribution for a largely understudied problem of label shift (also called target shift), a situation occurring when the class proportions vary between the training and test sets. The proposed contribution builds upon a recent work on the subject by Lipton et al., 2018 and addresses several of its weaknesses. The paper also gives several improved generalisation bounds w.r.t. that of Lipton et al. that are further used as guidelines to tune the regularisation parameter based on the size of source and target samples. Finally, the empirical results show that the proposed algorithm outperforms that of Lipton et al. especially in cases where the shift in proportions becomes quite important.
*Pros:
   - A work in an area with very view contributions and a certain lack of theoretical results
    -Theoretical results that are actually used in the algorithmic implementation and that allow to define the regularisation parameter based on the size of the available samples
    -Improved empirical results
*Cons:
    -An incomplete state-of-the-art section that does not cite several important contributions on the subject;
    -Lack of baselines due to the incomplete state-of-the-art section;
    -Lack of clear comparison with Lipton et al. both in terms of the proposed method and the obtained theoretical guarantees.
*Detailed comments:
This paper is rather interesting and well-written.
I have several major concerns regarding this paper. They can be summarised as follows:
    There is an important part of literature review on target shift that is missing in this paper. Even though, the paper mentioned the work of Chang, 2005 and Zhang, 2013, it completely ignores several other highly relevant methods such as [1,2]. These works also propose algorithms that allow to estimate class proportions that vary between training and test data. This estimation can then be used for cost-sensitive learning to correct the target shift. The paper should mention this work and add the corresponding methods to the baselines for comparison.
    Several statements that justify the contribution of this paper are unsupported. For instance, the paper states that the estimator obtained with the inverse of the confusion matrix can be arbitrary bad when the sample size and/or the singular values are small. However, this exact dependence can be found in Lemma 1 for the proposed contribution also! This is repeated in the beginning of Section 2.2 to justify the regularised version of the estimator but once again no evidence was provided to support the claim. The obtained bound for the regularised algorithm also has these two terms and thus it is not clear why the regularised algorithm is supposed to work better.
    The paper may want to clearly state the differences between the proposed algorithm and that of Lipton et al. and also between the obtained error bounds. The paper states that it achieves a k*log(k) improvement over Lipton et al. bounds but as fair as I can see this improvement is achieved only when h_0 is an ideal estimator. Furthermore, Lipton et al.’s bounds are linear in k while the proposed bounds replace this term with log(k/delta) so that when \delta is small, ie the bound holds with high probability, the bound becomes much worse. I would suggest to add a brief discussion on the relationship between the two to better highlight the original contribution of the paper.
    The proofs are quite badly written with many lacking results used to move from one inequality to another. For instance, Lemma 2 is proved using the theorem 1.4[Matrix Bernstein] and dilation technique from Tropp but it is not clear which results the authors are using in particular; Theorem 1.4 is related to the largest eigenvalue of the sum of matrices while the authors obtain an inequality for the norm of the sum without any further comment on how this transition was made. Also, I do not see why delta is smaller than 1/2 in Lemma 2.
*Minor comments:
   - p.1: expected have -> expected to have
   - p.4: we are instead only gave access -> given access to
   - I do not understand Figure 1. Should it be n_q*n_p on the y axis ?
   - The inequality for n_q next to Figure 1 is derived from the bound (6). Why it is independent of k?
   - Why the authors choose to the black box predictor h0 to be a two-layer fully connected neural? Is there any particular reason to use this classification model?
[1] Class Proportion Estimation with Application to Multiclass Anomaly Rejection, AISTATS14
[2] Mixture Proportion Estimation via Kernel Embeddings of Distributions, ICML16


We propose a deep reinforcement learning algorithm for semi-cooperative multiagent tasks, where agents are equipped with their separate reward functions, yet with willingness to cooperate. Under these semi-cooperative scenarios, popular methods of centralized training with decentralized execution for inducing cooperation and removing the non-stationarity problem do not work well due to lack of a common shared reward as well as inscalability in centralized training. Our algorithm, called Peer Evaluation based Dual DQN (PED-DQN), proposes to give peer evaluation signals to observed agents, which quantifies how they feel about a certain transition. This exchange of peer evaluation over time turns out to render agents to gradually reshape their reward functions so that their action choices from the myopic best-response tend to result in the good joint action with high cooperation. This evaluation-based method also allows flexible and scalable training by not assuming knowledge of the number of other agents and their observation and action spaces. We provide the performance evaluation of PED-DQN for the scenarios ranging from a simple two-person prisoner's dilemma to more complex semi-cooperative multi-agent tasks. In special cases where agents share a common reward function as in the centralized training methods, we show that inter-agent evaluation leads to better performance.

This work is well-written, but the quality of some sections can be improved significantly as suggested in the comments. I have a few main concerns that I explain in detailed comments. Among those, the paper argues that the algorithms converge without discussing why. Also, the amount of overestimation of the Q-values are one of my big concerns and not intuitive for me in the game of Prisoner's Dilemma that needs to be justified. For these reasons, I am voting for a weak reject now and conditional on the authors' rebuttal, I might increase my score later.
1) I have a series of questions about Prisoner's Dilemma example. I am curious to see what are the Q-values for t=100000 in PD. Is table 1h shows the converged values? What I am expecting to see is that the Q-values should converge to some values slightly larger than 3, but the values are  ~ 30. It is important to quantify how much bias you add to the optimal solution by reward shaping, and why this difference in the Q-values are observed.
2) One thing that is totally missing is the discussion of convergence of the proposed method. In section 3.4, you say that the Q-values converge, but it is not discussed why we expect convergence. The only place in the paper which I can make a conjecture about the convergence is in figure 4c which implicitly implies the convergence of the Mission DQN, but for the other one, I don't see such an observation. Is it possible to formalize the proposed method in the tabular case and discuss whether the Q-values should converge or not? Also, I would like to see the comparison of the Q-values plots in the experiments for both networks.
3) The intuition behind (2) should be better clarified. An example will be informative. I don't understand what |Z_a| is doing in this formula.
4) One of the main contributions of the paper is proposing the reward shaping mechanism. When I read section 3.3, I was expecting to see some result for policy gradient algorithms as well, but this paper does not analyze these algorithms. That would be very nice to see its performance in PG algorithms though. In such case that you are not going to implement these algorithms, I would suggest moving this section to the end of the paper and add it to a section named discussion and conclusion.
5) Is it possible to change the order of parts where you define $\hat{r}$ with the next part where you define $z_a$? I think that the clarity of this section should be improved. This is just a suggestion to explore. I was confused at the first reading when I saw $z_a$, "evaluation of transition" and then (2) without knowing how you define evaluation and why.
6) Is there any reason that ablation testing is only done for trio case? or you choose it randomly. Does the same behavior hold for other cases too?
7) Why in figure 4a, random is always around zero?
8) What will happen if you pass the location of the agent in addition to its observation? In this way, it is possible to have one  Dual-Q-network shared for all agents. This experiment might be added to the baselines in future revisions.
Minor:
* afore-mentioned -> aforementioned
section 4.2: I-DQN is used before definition
* Is it R_a in (4)?
* I assume that the table 1f-1h are not for the case of using independent Q-learning. Introducing these tables for the first time right after saying "This is observed when we use independent Q-learning" means that these values are coming from independent Q-learning, while they are not as far as I understand. Please make sure that this is correct.
section 4.1: * who's -> whose
* This work is also trying to answer a similar question to yours and should be referenced: "Learning Policy Representations in Multiagent Systems, by Grover et al. 2018"
* Visual illustrations of the game would be helpful in understanding the details of the experiment. Preparing a video of the learned policies also would informative.
-----------------------------------------------
After rebuttal: after reading the answers, I got answers to most of my questions. Some parts of the paper are vague that I see that other reviewers had the same questions. Given the amount of change required to address these modifications, I am not sure about the quality of the final work, so I keep my score the same.