While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.

In this work, a novel inverse constraint learning method is proposed, where the goal is to find out the constraints over state-action pairs for given demonstration and MDP **including a reward function** (so different from inverse cost learning). The novelty of this work comes from introducing maximum entropy inverse reinforcement learning (MaxEntIRL) framework to previous works [1, 2], and this work mainly focused on the tabular setting. The objective of this work is to solve the optimization in (8), which tries to find out the constraint that maximizes the probability of trajectories that cannot be generated if that constraint is applied. (Such an objective minimizes the normalization constant in (5) and results in maximization of the demonstration likelihood under the constraint.) To solve this objective, the proposed algorithm first computes the feature occupancy (Algorithm 1), and then use those feature occupancy with greedy iterative constraint inference (Algorithm 2 that motivated by maximum coverage problem) to get constraints. Two experiments in the GridWorld show that the proposed method effectively works.
I think this work is quite fundamental, impactful to be accepted at the conference and is possibly extended to practical scenarios (like explainable and safety RL and imitation learning) in the future. One thing I’d like to point out is to enhance the readability by reordering contents and adding some additional explanations to clarify their arguments. There are a few comments and questions I have:
- The exact definition and usage of nominal environments and rewards are still unclear to me. For example in Figure 2 (b), how did you define and get nominal MDP?
- Since Figure 1 is related to the second experiment, I recommend moving it to the experiment section.
- At 3.2.1., “Because constraints are sets of state-action pairs, imposing a
constraint within an MDP means restricting the set of actions that can be taken from certain states.” -> Need clarification
- At 3.2.1., “For MDPs with deterministic transitions, it is clear that any agent respecting these constraints will not visit an empty state.” -> Why?
- At (8), $\{\}$ to $\emptyset$
- In Figure 3, I was a bit confused about the relationship between the threshold and the false positive rate at first glance. What I understood is that a small threshold leads to lots of iteration for constraint selection, which increases the false positive. I want authors to add some comments on that.
 Reference
[1] Chou, Bereson, Ozay, “Learning Constraints from Demonstrations,” arXiv 2019
[2] Chou, Bereson, Ozay, “Learning Parametric Constraints in High Dimensions from Demonstrations,” CoRL 2019


Deep neural networks have achieved impressive performance in handling complicated semantics in natural language, while mostly treated as black boxes. To explain how the model handles compositional semantics of words and phrases, we study the hierarchical explanation problem. We highlight the key challenge is to compute non-additive and context-independent importance for individual words and phrases. We show some prior efforts on hierarchical explanations, e.g. contextual decomposition, do not satisfy the desired properties mathematically, leading to inconsistent explanation quality in different models. In this paper, we propose a formal way to quantify the importance of each word or phrase to generate hierarchical explanations. We modify contextual decomposition algorithms according to our formulation, and propose a model-agnostic explanation algorithm with competitive performance. Human evaluation and automatic metrics evaluation on both LSTM models and fine-tuned BERT Transformer models on multiple datasets show that our algorithms robustly outperform prior works on hierarchical explanations. We show our algorithms help explain compositionality of semantics, extract classification rules, and improve human trust of models 1 .

Summary - The paper addresses the problem of hierarchical explanations in deep models that handle compositional semantics of words and phrases. The paper first highlights desirable properties for importance attribution scores in hierarchical explanations, specifically, non-additivity and context independence, and shows how prior work on additive feature attribution and context decomposition doesn’t accurately capture these notions. After highlighting the said properties in context of related work, the authors propose an approach to calculate the context-independent importance of a phrase by computing the difference in scores with and without masking out the phrase marginalized over all possible surrounding word contexts (approximated by sampling surrounding context for a fixed radius under a language model). Furthermore, based on the above, the authors propose two more score attribution approaches -- based on integrating the above sampling step with (1) the contextual decomposition pipeline and (2) the input occlusion pipeline. Experimentally, the authors find that the attribution scores assigned by the proposed approach are more correlated with human annotations compared to prior approaches and additionally, the generated explanations turn out to be more trustworthy when humans evaluate their quality.
Strengths
- The paper is well-written and generally easy to follow. The authors do a good job of motivating and highlighting the desired properties of importance attribution scores and developing the proposed scoring mechanism. The proposed scoring mechanism ties in seamlessly with the existing contextual decomposition and occlusion pipelines and leads to improved performance when the generated explanations are evaluated.
- The proposed approach involving masking out the phrase and marginalizing over possible surrounding word-concepts is novel and offers an interesting perspective on how to approach context independent scoring of phrases -- (1) phrases don’t exist independent of the surrounding context and therefore marginalizing over all possible surrounding concepts makes sense and (2) replacing the intractable enumeration over all possible surrounding concepts with samples from a language model makes the score attribution process faster and more scalable modulo the learnt language model.
- Sec. 4.4 offers interesting insights. I like that the authors performed this ablation given that the expectation over surrounding contexts is computed approximately via samples under a language model. There’s a clear increase in terms of the attribution scores as the number of samples increases and the neighborhood size is increased. It is interesting to note that there is an approaching plateau region where increasing the neighborhood size won’t affect the assigned scores. This experiment provides a holistic picture of the behavior of the interpretability toolkit (manifesting in terms of attribution scores) given the approximations involved. I would encourage the authors to flesh this out even more.
Weaknesses
Having said that, there are some minor comments that I’d like to point out / get the authors’ opinion on. Highlighting these below:
- While SOC and SCD don’t always end up outperforming other approaches (specifically Statistic) on the SST-2, Yelp and TACRED datasets (Table. 1), for the human evaluation experiments, the authors only compare with CD, Direct Feed, ACD and GradSHAP. Do the authors have any insights on how well does Statistic perform on the human-evaluation set of experiments?
- While inspiring trust in users is one aspect of evaluating explanations via humans, it’s slightly unclear what ‘trust’ in this sense inherently identifies. Although, it might implicitly capture some notion of reliability (and predictability of the explanations by humans), asking users to rank explanations across a spectrum of ‘best’ to ‘worst’ doesn’t explicitly capture that. Another possible aspect to look into could be -- ‘’Do the generated explanations help humans predict the output of the model?’’ This captures reliability in a very explicit sense. Do the authors have any thoughts on this and potential experiments that might address this? I don’t think not addressing this is necessarily detrimental to the paper but I’m curious to hear the thoughts of the authors on the same.
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed phrase attribution scoring mechanism is motivated from a novel perspective and has a reasonable approximation characterized appropriately by the ablations performed. The strengths and weaknesses highlighted above form the basis of my rating.


Recently, several studies have proven the global convergence and generalization abilities of the gradient descent method for two-layer ReLU networks. Most studies especially focused on the regression problems with the squared loss function, except for a few, and the importance of the positivity of the neural tangent kernel has been pointed out. However, the performance of gradient descent on classification problems using the logistic loss function has not been well studied, and further investigation of this problem structure is possible. In this work, we demonstrate that the separability assumption using a neural tangent model is more reasonable than the positivity condition of the neural tangent kernel and provide a refined convergence analysis of the gradient descent for two-layer networks with smooth activations. A remarkable point of our result is that our convergence and generalization bounds have much better dependence on the network width in comparison to related studies. Consequently, our theory provides a generalization guarantee for less over-parameterized two-layer networks, while most studies require much higher over-parameterization.

This paper studied the generalization performance of gradient descent for training over-parameterized two-layer neural networks on classification problems. The authors proved that under a neural tangent based separability assumption, as long as the neural network width is $\Omega(\epsilon^{-1})$, the number of training examples is $\tilde\Omega(\epsilon^{-4})$, within $O(\epsilon^{-2})$ iterations GD can achieve expected $\epsilon$-classification error.
Overall this paper is well written and easy to follow. The theoretical results on the neural network width and iteration complexity are interesting.
My major concern is that the comparison with Allen-Zhu et al and Cao & Gu seem somewhat unfair. First, Allen-Zhu et al and Cao & Gu both studied the generalization performance of GD for training multi-layer neural networks, which is fundamentally more difficult than two-layer networks. Second, they use ReLU activation functions, which brings in the nonsmoothness along the optimization trajectory. This would also make the condition on the neural network width become worse. Therefore, when claiming the advantage of the derived guarantees, the authors should clearly clarify such differences.
Another concern is that whether the derived theoretical results can be generalized to ReLU network?
When proving the generalization result, this paper takes advantage of margin-based generalization error bound. However, the generalization results in Cao & Gu are proved via applying standard empirical Rademacher complexity based generalization error bound. I would wonder which technique can give a tighter bound?
Can you provide some examples regarding which type of data can satisfy Assumption (A.4) with constant margin $\rho$?
 The authors would like to briefly discuss another data separation assumption adopted in the following papers (although this assumption is typically made for regression problem).
[1] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. arXiv preprint arXiv:1811.03962, 2018b.
[2] Allen-Zhu, Z., Li, Y. and Song, Z. (2018c). On the convergence rate of training recurrent neural networks. arXiv preprint arXiv:1810.12065 .
[3] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888, 2018.
[4] Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterization: global convergence guarantees for training shallow neural networks. arXiv preprint arXiv:1902.04674, 2019.
[5] Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. arXiv preprint arXiv:1906.04688, 2019.


Social media have emerged to be increasingly popular and have been used as tools for gathering and propagating information. However, the vigorous growth of social media contributes to the fast-spreading and far-reaching rumors. Rumor detection has become a necessary defense. Traditional rumor detection methods based on hand-crafted feature selection are replaced by automatic approaches that are based on Artificial Intelligence (AI). AI decision making systems need to have the necessary means, such as explainability to assure users their trustworthiness. Inspired by the thriving development of Generative Adversarial Networks (GANs) on text applications, we propose LEX-GAN, a GAN-based layered explainable rumor detector to improve the detection quality and provide explainability. Unlike fake news detection that needs a previously collected verified news database, LEX-GAN realizes explainable rumor detection based on only tweet-level text. LEX-GAN is trained with generated non-rumor-looking rumors. The generators produce rumors by intelligently inserting controversial information in non-rumors, and force the discriminators to detect detailed glitches and deduce exactly which parts in the sentence are problematic. The layered structures in both generative and discriminative model contributes to the high performance. We show LEX-GAN's mutation detection ability in textural sequences by performing a gene classification and mutation detection task.

In the paper, authors proposed a generative adversarial network-based rumor detection model that can label short text like Twitter posts as rumor or not. The model can further highlight the words that are responsible for the rumor accusation.
Proposed model consists of 4 sub models: a G_Where model finds the word to replace so to create an artificial rumor; a G_replace model decides what the replacement word should be; a D_classify model detects if a sequence is a rumor; a final D_explain model pinpoints the word of concern. D_ models and G_ models are trained in an adversarial competing way.
Experiments showed that the LEX-GAN model outperforms other non-GAN models by a large margin on a previously published rumor dataset (PHEME) and in a gene classification task.
My questions:
1) The task modeled is essentially a word replacement detection problem. Is this equivalent to rumor detection? Even if it performs really well on a static dataset, it could be very vulnerable to attackers. Various previous works mentioned in the paper, including the PHEME paper by Kochkina et al, used supporting evidence for detection, which sounds like a more robust approach.
2) Authors didn't explain the rationale behind the choice of model structure, e.g. GRU vs LSTM vs Conv. The different structures have been used in mix in the paper. Are those choices irrelevant or critical?
3) I would like to see more discussion on the nature of errors from those models, but it's lacking in the paper. This could be critical to understand the model’s ability and limitation, esp given that it’s not looking at supporting evidences from other sequences.
Small errors noticed: The citation for PHEME paper (Kochkina et al) points to a preprint version, while an ACL Anthology published version exists.


We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity subnetworks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.

*Summary*
This paper compares network pruning masks learned via different iterative pruning methods. Experiments on LeNet + MNIST show (a) different methods can achieve similar accuracy, (b) pruned sub-networks may differ significantly despite identical initialization, (c) weight reinitialization between pruning iterations yields more structured convolutional layer pruning than not reinitializing, and (d) pruning methods may differ in the stability of weights over pruning iterations.
*Rating*
There are interesting bits of data in this paper, but the overall story is somewhat muddled and some inferences seem to be insufficiently supported by data (1-2 below). In addition, the text would benefit from better organization and presentation (3-4 below) and replications on other datasets and architectures (5 below). As a result, my rating is currently weak reject.
(1) *Overlap in pruned sub-networks*: In the middle of Sec. 4, Fig 3-5 examine the similarity of pruning masks between methods. It seems clear from several of the plots that multiple methods produce identical layer-wise masks, e.g. Fig 3(a), while others show a wide variance. The overlap in lines makes this difficult to assess at times: perhaps a table would communicate it better? Also, are Fig 3-4 depicting the Jaccard distance between masks of unpruned or pruned weights? Is the ordering of training samples fixed in addition to network initialization? Is reinitialization used between iterations? Also, Fig 5 seems to contradict the conclusion that methods tend to learn different masks, since the structures are noticeably similar.
(2) *Weight stability during pruning*: It is difficult to discern a conclusion in Sec 5. First, a clarification on the figures: are lines for pruned weights terminated where they are pruned? If so, this would be helpful to state. The 4th paragraph claims, "we empirically find a correlation between weight stability and performance", but this is not at all obvious from Figures 6-7. I'm not sure what a more stable evolution looks like. Hybrid is shown to be accurate in Fig 1, but the conv. weights in 6(a) are a spaghetti tangle and the FC weights in 7(a) are constantly increasing in magnitude. Perhaps a mathematical formulation for stability (perhaps based on average standard deviation of each weight's values over training) with a table of values for each method/layer would help to clarify.
(3) *Organization*: Since the paper has many intertwined observations, a better organization would be helpful. Consider mirroring the structure of Sec 1.1 in a combined Sec. 4-5 with clear paragraph headers summarizing each conclusion.
(4) *Presentation*: Figure is too small throughout to read from a printed copy (or even on a screen without significant zooming). Several results could be presented with less ambiguity in tabular form, as noted above.
(5) *Replications*: The paper presents results only a single set of experiments using the MNIST dataset with the LeNet architecture. While this isn't a fatal issue, it is a significant weakness.
*Notes*
Fig 1 and 2: What spacing is used for the x- and y- axes?
Fig 8: Perhaps scale vertically by the standard deviation of the weights?


In visual planning (VP), an agent learns to plan goal-directed behavior from observations of a dynamical system obtained offline, e.g., images obtained from selfsupervised robot interaction. VP algorithms essentially combine data-driven perception and planning, and are important for robotic manipulation and navigation domains, among others. A recent and promising approach to VP is the semiparametric topological memory (SPTM) method, where image samples are treated as nodes in a graph, and the connectivity in the graph is learned using deep image classification. Thus, the learned graph represents the topological connectivity of the data, and planning can be performed using conventional graph search methods. However, training SPTM necessitates a suitable loss function for the connectivity classifier, which requires non-trivial manual tuning. More importantly, SPTM is constricted in its ability to generalize to changes in the domain, as its graph is constructed from direct observations and thus requires collecting new samples for planning. In this paper, we propose Hallucinative Topological Memory (HTM), which overcomes these shortcomings. In HTM, instead of training a discriminative classifier we train an energy function using contrastive predictive coding. In addition, we learn a conditional VAE model that generates samples given a context image of the domain, and use these hallucinated samples for building the connectivity graph, allowing for zero-shot generalization to domain changes. In simulated domains, HTM outperforms conventional SPTM and visual foresight methods in terms of both plan quality and success in long-horizon planning.

The paper presents a method for learning agents to solve visual planning, in particular to navigate to a desired goal position in a maze, with a learned topological map, i.e. a graph, where nodes correspond to positions in the maze and edges correspond accessibility (reachability in a certain number of steps). The work extends previous work (semi parametric topological memory, ref. [22]) in several ways. It claims to address a shortcoming of [22], namely the fact that the graph is calculated offline from random rollouts, by using a conditional variational auto-encoder to predict a set of observed images which could lie between the current position and the goal position, and, most importantly from a context image which describes the layout of the environment. These predicted images are then arranged in a graph through a connectivity predictor, which is trained from rollouts through a contrastive loss. Training is performed on multiple environments, and the context vector provides enough information for this connectivity network to generalize to unseen environments.  At test time, the agent navigates using a planner and a policy. The planner calculates the shortest path on a graph where edges are connectivity probabilities, and the policy is an inverse model trained on the output of the planner.
While the idea of a topological memory with dynamic graph creation is certainly interesting, the work is unfortunately not well enough executed and the paper structured written in a way which makes it up to impossible to grasp what has been really done, as much information is missing which would be required for understanding.
As a first example, we are never really told what the observations are, which the agent sees. The different figures of the paper show very small images with a 2D maze from a bird’s eye view consisting of a walls arranged in a single connected component (mostly 1 to 3 strokes) in red color and an agent shown as a position indicated as a green dot. Are these the observed images? In absence of any other information, this is what we need to assume, and then this problem is fully observable and does not seem to be very challenging. Given the figures, even a handcrafted algorithm should be able to calculate the optimal solution with Dijkstra’s algorithm on a graph calculated from the pixel grid.
This important missing information alone makes it difficult to assess the paper, but the rest of the writing is similarly confusing. The authors focus on very short and dense descriptions of mathematical properties, but seem to have forgotten to ground the different symbols and to connect them to physical entities of the problem. The technical writing is in large parts disconnected from the problem which is addressed by it.
Further examples are:
-	“the data (…) is collected in a self-supervised manner”: what does this mean? Self-supervision is way of creating loss from data without labels, but I am not sure what is meant by collecting data this way.
-	The paragraph on CPC in section 2 can only be understood if the contrastive loss is known. To make the paper self-consistent, this should be properly explained, and tied to a training procedure which details how exactly the positive and negative samples are defined … and collected.
-	The CPC objective in section 2 is only loosely connected to its usage in section 3.2. Barely writing “we optimize a CPC” objective is not sufficient for understanding how this objective is really tied to the different entities of the problem. This paper contains maths (which is always a pleasure to read), but it is not a purely and abstract mathematical problem - a real task is addressed, so it needs to be connected to it. This connection has certainly been done by the authors while they were working on the problem, but they should also communicate it to the reader.
-	The section on ML trajectory is too dense and should be rewritten. I don’t understand what the authors want to tell us here. Basically, a (generalized) Dijkstra is run on a graph, where edge weights are the density or density ratios learned by the CPC objective, and if the edge weights are probabilities, that the shortest path corresponds to a trajectory likelihood. This is known, and this information is buried in a dense set of equations which are difficult to decipher and do not add any further value to the paper.
-	The connection between the planner (generalized Dijkstra) and the policy is never explained. We don’t know how the policy is trained and how it works.
One of the downsides of the method is that it requires a context image. This image is responsible for the generalization to unseen environments, but it is a major drawback, as the image must be created beforehand. The authors claim that the context image must only contain the layout in any format which makes it possible to extract information about navigational space from it, but in the experiments the context image corresponds to the full map – and it is probably equivalent to the observed images, but we can’t be sure as we haven’t been told. In any case, it is far from sure how this could generalize to more complex environments, let alone 3D navigation as is currently addressed in standard simulators like VizDoom, GIBSON, Matterport, Deepmind Lab, Habitat AI etc.
The authors’ claim that the proposed environment requires long-term planning, but looking at the images this does not seem to be the case.
The paper claims to perform zero-shot generalization and to adapt to changes in the environment, like the slight changes in camera motion, variations in lightning, but it unclear how the solution solves this claim.
How does the agent determine that a goal has been reached, without ground truth information?
What happens, if the hallucinated images are disconnected (form several connected components) or are disconnected from the current position and/or from the goal position?
As mentioned, the method is evaluated on an environment, which is too simple. The experiments are difficult to assess, as we don’t really know what the agent observes. An information asymmetry is mentioned (visual foresight having the object’s (=agent’s) position and the others not) … but if the proposed method observes the bird’s eye view, it can infer the agent’s position (as the position of the green dot).
Subjective evaluation by humans on this kind of simple data does not seem to be meaningful, in particular with a very low number of observers (5 people).


Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attention heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We analyze how the latency controls affect the attention span and we study the relationship between the speed of a head and the layer it belongs to. Finally, we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

The paper proposes an approach for simultaneous neural machine translation. While prior works deal with recurrent models, the authors adopt previous approaches for Transformer. Specifically, for decoder-encoder attention they introduce Monotonic Multihead Attention (MMA) designed to deal with several attention heads. Each attention head operates as a separate monotonic attention head similar to the ones from previous works by Raffel et al. (2017) and Arivazhagan et al. (2019). MMA has two versions: MMA-H (hard), where each head attends only to one token, and MMA-IL (infinite lookback), where each head attends to all tokens up to a selected position.
The main novelty of the work in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions (L_var).
Experimental results on two translation tasks (IWSLT En-Vi and WMT De-En) show better quality-latency trade-offs compared to the recurrent analogs. Experiments with different number of attention heads would be helpful, but in the current state are rather confusing (see the comments below).
I can not recommend accepting this paper due to the two main reasons.
1) The proposed solution lacks novelty.
MMA-H and MMA-IL attentions, introduced in this work, are straightforward applications of previous works by Raffel et al. (2017) and Arivazhagan et al. (2019) respectively to each of the heads. Novelty is in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions. However, these alone are unlikely to make a large impact.
2) The results are experimentally weak, evaluation is questionable.
The current baseline is the recurrent model, but since the main contribution of this work is in how to deal with several heads the proper baselines would be (i) the same, but with single-head attention, (ii) the same, but without L_var (summing latency penalty over heads is straightforward). However, these baselines are absent and the improvement is likely to be due to the replacement of RNN with the Transformer - this would be a limited contribution.
Other comments on the experiments.
1) MMA-IL was tested only on one small dataset (IWSLT En-Vi), MMA-H only on 2 datasets - more experiments would help.
2) Figure 2 shows that when increasing lagging, performance first improves, then drops. While the former is expected, the latter is not: one would expect a model to reach the performance of its offline version when given maximum latency, but not to drop. The MILK model (Arivazhagan et al., 2019) behaves as expected, but the proposed MMA does not: does this mean that there is some problem with the model? An explanation of this behavior would be helpful.
3) Number of heads, Figure 4: the results are supposed to show that performance improves when increasing the number of heads, but the figure shows the opposite. If the decoder has more than one layer (which Transformer usually has), for 4 heads perform worse than one, 8 heads (which the standard Transformer setting) - not better than one, and we get the improvement only when using 16 heads with 6 layers. Could you elaborate on this? The text says “quality generally tends to improve with more layers and more heads”.
4) I can see the motivation for using the L_var loss term in this setting, but forcing heads in MMA-H attend to similar positions seems counterintuitive: there is evidence that making attention heads less correlated improves quality (see for example Li et al, EMNLP 2018 https://www.aclweb.org/anthology/D18-1317/), but L_var may end up doing the opposite. How different is the behavior of the learned heads? Figure 3 shows the average attention span, but this does not tell how many of the heads are doing different things. Some analysis would be nice.
Comments on the presentation.
1) It would be better to mention that encoder self-attention is limited only to previous positions earlier in the text and more prominently (now it’s on the bottom of the sixth page). A good place would be in Section 2.2 when talking about different attention types in the Transformer and modifications required for simultaneous translation.
2) Figure 2: it would be helpful to see offline model performance on these pictures (e.g., a horizontal line showing the BLEU scores). One of the main points of these experiments is to see how much each model drops compared to its offline version.
3) Figure 4: it is not clear from the figure which models it corresponds (for example, what is the difference between the first and the second figures: language pairs? models?)
4) Table 2 shows the BLEU scores for Transformer with unidirectional encoder and greedy decoding. It would be helpful to see also the BLEU scores of Transformer in the standard setting (full attention in the encoder, beam search inference). For now, it is not clear how much one loses by replacing the encoder to unidirectional: if much, then probably it makes sense to work also on the encoder (read and encode source sentence with a latency).
Language and typos.
The text is quite raw and could use an extra work; a lot of typos starting from the abstract (e.g., “...for multiple attentions heads”).


Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques. Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.

Final Edit:
I have reviewer the final version of the paper and have decided to increase my score to a weak accept. I maintain some concerns around the empirical evaluation in the paper (collating results from multiple sources with different experimental procedures). But my major concerns have been addressed by the authors in their response and changes to the paper.
----------------------------------
Post rebuttal edit:
Following updates to the paper manuscript, which address my concerns around the correctness of the empirical results, I have updated my review score from 1 to 3.
----------------------------------
Summary:
This paper draws on connections between virtual adversarial training (VAT) and Lipschitz regularization to utilize VAT techniques in the training of WGAN architectures. While this work may be touching on something quite interesting, I felt that the theoretical exposition of the ideas was lacking. The empirical evidence seemed promising in some direction though lacking in others.
I was requested as an emergency reviewer for this paper.
Overview:
This paper is 9 pages length in total. Unfortunately, I felt that the use of an additional page was unwarranted and that the paper contained unnecessary content.
Due to a concern over correctness of some empirical results and issues with the presented derivations I have opted to reject this paper. I hope that these issues can be addressed by the authors in which case I will reassess my score.
1) Under equation 5, "with substantially more stable training behaviour and improved sample quality". A citation should be included for this claim. In fact, recent advances in GAN methods have not required the Wasserstein distance objective [1].
2) I found some issues with Section 2.2.
a) First a comment on related work. There is older work studying the generalization properties of Lipschitz neural networks which is not mentioned in this section. For example, [2]. You also write that learning under Lipschitz constraints became prevalent with the introduction of WGAN. While this is probably true, I think it is fair to point out that many older papers also utilized similar bounds in the vein of improving generalization. For example [3], which also aimed to limit the gradient norm of deep neural networks.
b) I felt that this subsection was a little bloated and the content did not fit entirely under the heading. A large chunk of this section is dedicated to discussing potential issues arising with the gradient penalty formulation of WGAN and alternative approaches such as the Banach WGAN. While these are useful additions they did not feel critical to this work and in my opinion did not deserve an extension beyond the 8 page recommendation.
3) I found the discussion in Section~3 a little difficult to follow. I will summarize my key concerns below.
a) The authors assume that generalizing Lipschitz continuity to a premetric space is trivial. While the more important results seems believable I am not convinced by the presentation of these results and would prefer to have seen this given more careful treatment. For example, premetrics need not obey symmetry or the triangle inequality (assuming this is the definition used by the authors --- including this would be valuable). It is written (paraphrasing) that a mapping $f$ is $K$-Lipschitz iff for any $x$ the supremum over $r$ is bounded by $K$. However, $r$ only appears on the right-hand side of a potentially asymmetric distance function. Moreover, many properties of Lipschitz continuous functions depend on the triangle inequality holding in the metric space and these would fail to hold here.
b) When connecting ALP to VAT some of the differences in the formulations are hand-waved away in unconvincing ways. Under the trivial metric, the Lipschitz constant is given by the maximum distance in the output space. With this observation, it seems trivial that the VAT formulation will perform a form of Lipschitz regularization. However, the Lipschitz constant does not take into account distance in the input space in a meaningful way and so I am unsure to what extend the connection is really meaningful. Further, the $r < \epsilon$ constraint in the VAT model is treated as an inconvenient implementation detail but I am not convinced this is sufficient. Indeed, this $\epsilon$ could be used to bound the input deviations and thus could be seen as affecting the Lipschitz constant under a more reasonable metric.
4) I felt that Section~4.1 presented an important discussion coupled with an interesting toy problem to highlight benefits and shortcomings of the proposed method. However, this section was moderately long and contributed only a little towards understanding the practical settings users of ALR would care about in practice. I did not gain much intuition into how ALR might generalize to high dimensional settings and was concerned by the fact that the distribution $P_\epsilon$ used was heavily hand-engineered and did not match up with the ones used in later experiments.
a) I did not understand the comments that constraining the Lipschitz constant globally may be undesirable in WGANs. The dual optimization problem requires a Lipschitz constraint be enforced over the support of the distributions and we should not care outside of this region in any case (except in cases where the generator might move the support to a currently under-regularized region of the critic domain in which case a global constraint may be advantageous).
5) I am not particularly up to date with evaluation of GAN models but to me the presented results in the main paper looked mostly reasonable. Some major concerns did remain to me which I would appreciate being addressed by the authors.
a) I have one question on the reported "Best" inception score for the WGAN-ALP and Progressive-GAN models. You stated that each model was trained 5 times and reported the mean, standard deviation and best results. However, the difference between the best and average scores alone would constitute a higher standard deviation: $\sqrt{(8.80 - 8.56)^2 / 4} = 0.12$. Please can you clarify how exactly each of these values was computed?
b) In the main paper the authors write that ALP is able to work in high dimensional settings (though is not competitive with state of the art). In the appendix however the authors point out that they must make significant modifications to the training objective by including a squared Lipschitz constraint violation term (violating further the comparison to VAT). I do not consider this a huge issue, but it should be discussed in the main paper.
c) Finally, the authors employed a range of different hyperparameter settings through their experiments but gave little guidance on how to choose these settings in practice or how sensitive their proposed method is to changes in these hyperparameters. I believe that this would be a highly-valuable addition to the paper and would help distinguish this method from other training stabilization proposals.
Minor comments:
- In paragraph 1, you write that WGAN requires critic to consist of only 1-Lipschitz functions. This is true of the Wasserstein distance estimation problem but the WGAN only requires the correct gradient direction (scaling of the critic is fine).
- In summary points in intro, "ALR" is used before acronym is defined.
- Equation (12) and (13) are twice normalized (||r_k||^2=1 by definition). Similar issue in (22) and (23).
- First para of Section 3, you write "on the space of labels". Do you mean on the probability simplex?
References:
[1] "Large scale GAN training for high fidelity natural image synthesis", Brock, Donahue, and Simonyan
[2] "The sample complexity of pattern classification with neural networks", Bartlett
[3] "Double backpropagation increasing generalization performance", Drucker and LeCun


Bayesian Optimisation is an important decision-making tool for high-stakes applications in drug discovery and materials design. An oft-overlooked modelling consideration however is the representation of input-dependent or heteroscedastic aleatoric uncertainty. The cost of misrepresenting this uncertainty as being homoscedastic could be high in drug discovery applications where neglecting heteroscedasticity in high throughput virtual screening could lead to a failed drug discovery program. In this paper, we propose a heteroscedastic Bayesian Optimisation scheme which both represents and optimises aleatoric noise in the suggestions. We consider cases such as drug discovery where we would like to minimise or be robust to aleatoric uncertainty but also applications such as materials discovery where it may be beneficial to maximise or be antifragile to aleatoric uncertainty. Our scheme features a heteroscedastic Gaussian Process (GP) as the surrogate model in conjunction with two acquisition heuristics. First, we extend the augmented expected improvement (AEI) heuristic to the heteroscedastic setting and second, we introduce a new acquisition function, aleatoric-penalised expected improvement (ANPEI) based on a simple scalarisation of the performance and noise objective. Both methods are capable of penalising or promoting aleatoric noise in the suggestions and yield improved performance relative to a naive implementation of homoscedastic Bayesian Optimisation on toy problems as well as a real-world optimisation problem.

SUMMARY OF REVIEW
This paper discusses an interesting problem of BO in the cases of robustness and antifragility to aleatoric noise/uncertainty. To tackle this problem, the authors replace the conventional homoscedastic GP model with a heteroscedastic GP model. In the case of robustness to aleatoric noise/uncertainty, the authors have modified EI by simply either scaling down [32] or subtracting from its value more when the aleatoric noise increases. In the case of antifragility to aleatoric noise/uncertainty, they do the opposite.
The modifications of EI to handle robustness and antifragility to aleatoric noise/uncertainty are simple and straightforward, one of which is similar to the augmented EI of [32].
There are some technical ambiguities, as detailed below. In particular, the choice of objective function (equation 9) for this problem needs to be justified and motivated by the practical applications described in Section 1.
Since no convergence guarantee is given, a more extensive empirical analysis with real datasets needs to be provided to better understand the performance and behavior of the proposed BO algorithms. In particular, though the authors have motivated their problem using the compelling applications of materials and drug discovery, no experimental result for these applications has been provided to support the motivation of this work.
DETAILED COMMENTS
The authors seem to motivate the significance of their problem of interest through the key applications of materials and drug discovery which I can appreciate. Unfortunately, experimental results in such applications were not available in this paper to "close the loop" in supporting the motivation of this work, begging the question whether their proposed BO algorithm indeed works for these key applications. For example, why is the FreeSolv hydration energy dataset not used for your experiments?
For Fig. 1, how exactly do you extract the error magnitudes from the FreeSolv hydration energy dataset? How do you exactly define the notion of calculated vs. experimental uncertainties? Fig. 1 shows that the noise peaks with a relatively high frequency at a single error magnitude value. Would the assumption of homoscedastic noise at this peaked value be detrimental to BO? A sensitivity analysis would be useful here.
For the soil phosphorus fraction dataset (Fig. 2), the skewed distribution of the measurements (few extremely large measurements and many small-valued measurements) may not be due to heteroscedastic aleatoric uncertainty. In fact, in the literature of earth/environmental science, such a dataset is often modeled using a log-Gaussian process (or log-normal kriging), that is, the log-measurements follow a GP:
Webster, R., and Oliver, M. 2007. Geostatistics for Environmental Scientists. John Wiley & Sons, 2nd edition.
Can the authors provide supporting evidence (in the form of references) that such a dataset is due to heteroscedastic aleatoric uncertainty?
On page 4, step 2 of the most likely heteroscedastic GP algorithm [28] cannot be understood: What is E[x]? Isn't G_1 a GP? Why is it able to accept x_i and D as inputs? How is z_i defined?
The authors say that "A note on the form of this variance estimator is give in Appendix B." There is no Appendix B.
Can the authors give a detailed discussion why is the expression of f(x) = g(x) + s(x) in equation 9 the right one to be minimized in practice (e.g., in the context of materials and drug discovery)? For example, do material scientists use such an objective function? Provide references. Furthermore, why is the same equation 9 being minimized for both the cases of robustness and antifragility to aleatoric uncertainty?
Caption of Fig. 3: I can't understand the sentence: "The combined objective, which when optimised maximises the sin wave subject to the minimisation of aleatoric noise". This was repeated in Section 5.5: "finds the first maximum as that which minimises aleatoric noise". Isn't the minimum of the aleatoric noise at the origin (Fig. 3b)?
Can the authors provide information on how much initial data was provided prior to running BO? How much data is used for learning the GP hyperparameters?
The performance advantage of heteroscedastic ANPEI over homoscedastic does not appear to be significant for Branin-Hoo function (Figs. 5b and 6b). Can the authors explain this?
Minor issues
There are two different font types of x in bold.
Page 5: fixed aleaotric
Which phrase is correct? "is obtained by subtracting the noise function from the 1D sinusoid" in the caption of Fig. 3 or "The objective function in all cases is the principal objective g(x) minus one standard deviation of the ground truth noise function s(x)"?
Fig. 5b: Shouldn't the vertical axis be labeled as ...+ Noise?
Fig. 6a: Shouldn't the vertical axis be labeled as ...- Noise?


Many visual illusions are contextual by nature. In the orientation-tilt illusion, the perceived orientation of a central grating is repulsed from or attracted towards the orientation of a surrounding grating. An open question in vision science is whether such illusions reflect basic limitations of the visual system, or if they correspond to corner cases of neural computations that are efficient in everyday settings. Here we develop deep recurrent network architectures that approximate neural circuits linked to contextual illusions ::: We ::::::: develop : a ::::: deep ::::::: recurrent :::::: neural ::::::: network ::::::::: architecture :::: that ::::::::::: approximates :::::: known ::::: visual ::::::: cortical :::::: circuits : (Mély et al., 2018). We show that these architectures ::   this ::::::::::: architecture, which we refer to as γ-Nets, are more sample efficient for learning contour detection than the state of the art, and exhibit an :: the :::::: γ-Net , contour :::::::: detection :::::::: accuracy by driving it to prefer low-level edges over high-level object boundary contours. Overall, our study suggests that contextual illusions are :: the :::::::::::: orientation-tilt ::::::: illusion :: is a byproduct of neural circuits that help biological visual systems achieve robust and efficient perception :::::::
contours :::::::: detection, and that incorporating such circuits in artificial neural networks can improve computer vision.

The authors propose a new artificial neural network architecture that is derived from a human visual model (Mély et al., 2018). The original (human vision) model can explain some of the human visual illusions, specifically contextual ones. While the adaptation from this human visual model to artificial neural networks was previously done by (Linsley et. al., 2018a), in this paper the authors extend (Linsley et. al., 2018a) to better capture some of the constraints in the human visual model, and also to add a formulation that can also model top-down connections (across layers).
The goal of replicating the structure of the visual human model in artificial neural networks is to improve the machine vision by mimicking the human vision. The results in this paper on contour detection show an improvement regarding sample efficiency with respect to other state of the art methods. Also, the authors show that the artificial model also suffers from visual illusions, and when these are explicitly corrected, its performance drops. This shows that these illusions are a byproduct of the system improving its visual abilities.
Strengths:
1 - Well motivated by cognitive science theory and modeling of the human visual system. The artificial modeling is not just inspired by the human visual system, but derived from an actual human system, replicating it. The paper presents an extension of (Linsley et. al., 2018a), and both the base model and the extension are biologically motivated. The paper also shows that this extension is important to get good results.
2 - Clear structure and ideas. Clearly explained, well written, reasonable ablations and transparent presentation of results, assumptions, limitations, contributions and experiments.
3 - Very good results in contour detection for a very-low data regime, which proves the strength of the model inductive bias. The results show both better performance and good mimicking of the human vision behavior.
Weaknesses
1 – Limited experimental results.
- Is contour detection the only task in which surround suppression helps?
- Lack of experiments in larger contour detection datasets (like Semantic Border Dataset or Cityscapes). Does the model improve accuracy in those, or it just improves sample efficiency in (extremely) small (subsets of) datasets?
- The only results are regarding sample efficiency. Not accuracy, or number of parameters, or running time.
2 – Weak results supporting the claim that the computer vision model mimics the human visual illusions. The authors show an elegant experiment where they explicitly correct the visual illusion and obtain worse results, backing the presented hypothesis. However, the ablation experiments show that the same model without the top-down connections does not present the same results. One would expect that removing a part that is not in the initial formulation from (Mély et al., 2018) should not affect in the visual illusion experiments. Both the explanation at the end of Section 2 and at the sixth paragraph of Section 3 seem to indicate that the _non-negativity_ is the important factor to explain contextual illusions, not the top-down formulation. Can the authors explain whether or not the top-down formulation is a necessary part to model (Mély et al., 2018)? If this is not the case (and top-down is not necessary), the current results would not support the idea that implementing (Mély et al., 2018) in artificial neural networks produces visual illusions in the computer. Also, did the authors perform the visual illusion experiment without the non-negativity? Is it a necessary requisite for the visual illusions to appear? As a positive remark regarding weakness #2, the results still show that the model that performs best is the one having visual illusions.
3 – Related to weaknesses #1 and #2, did the authors perform any experiment on other contextual illusions like the ones explained in (Mély et al., 2018), namely “color induction” or “enhanced color shifts”? Consistent findings across different visual illusions would reinforce the presented hypothesis.
Additional comments:
- While it may not be a "bug", arguing that the visual illusions are a "feature" (both in machine and human visual systems) is probably too much of a claim. At some point the authors refer to it as a "byproduct" of the of the system improving its visual abilities, which I consider a more suitable word.
- For an audience outside of neuroscience, a brief explanation of the concepts “suppression” and “facilitation”, which are very important in the paper, would be convenient.
- Are 8 time-steps sufficient for reaching a steady state? Is the steady state checked in any way?
-------- Updated --------
Rating:
- Weak accept
The authors addressed the initial concerns I raised, providing detailed explanations and additional experiments. While some of these concerns still remain to some extent, I believe this paper explores an interesting direction connecting human visual models with computer vision, obtaining good experimental results. The authors softened some of their initial claims, and the current ones are more supported by the experiments.
Overall, I believe this paper can be a valuable contribution to the conference, and I recommend acceptance.
-------- Previously --------
Rating:
- Weak reject
Overall it is a clear and well-structured paper, with interesting biologically derived architectures (strength #1), but as it stands the experimental results either do not completely support the claims of the paper (weakness #2) or are limited in scope (weakness #1). If the authors can address my concerns, mostly motivating the importance of the experiments for weakness #1, and providing an explanation (possibly correcting me) for weakness #2, I will be happy to increase my rating.


Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform to study motor activity across contexts by training a model to solve four complex tasks. Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics. These representations are reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.

The authors use a virtual rodent to study flexible behavior, a key concept at the intersection of AI and neuroscience. “Flexibility” is a common buzzword in this field used to describe what it is that mammals do better than deep nets, even as deep nets can outperform them on any one task in isolation with sufficient training. As the authors note, previous work addresses “cognitive flexibility” (Yang et al. 2019) in isolation without motor commands, or studies motor dynamics of real mammals but without any task flexibility—occluding the actual advantages that mammals have over machines. However, this work goes the full mile in requiring the agent to implement flexible cognition using a complex body in a physical environment with egocentric stimuli. From a strictly AI point of view, I could imagine arguments that this is not sufficiently “novel,” in that a lot of deep RL researchers have already trained agents with complex joint structures to succeed in such virtual worlds. But the real substance of the paper is the application of data analysis techniques imported directly from neuroscience. It is important to suss out what features of neural dynamics are peculiarities of biological computation and which are natural consequences of the tasks being performed. I find the paper to be overall well motivated.
The analyses were well ordered, starting with a high-level characterization of behavior that showed a hierarchy of features, where “low-level” or “fast” features are mixed across tasks while “high-level” or “slow” features are more task specific. They then moved on to find “neural correlates” at the behavior and task levels, driving home their primary point that core layers encode high-level variables in a way not shared across tasks while policy layers implement reusable core motor functions that are used in multiple contexts. This observation coincides with the measurable frequency content of core and policy activity, which is demonstrably faster in policy layers. Finally, the authors show a visualization of the dynamical systems that underlie these computations, with videos that very clearly illustrate how core and policy dynamics dissociate to specialize in cognition vs. motor generation. This part is where they really leverage the artificial setting, where the exact dynamics are known and do not need to be inferred from noisy neural data.
While the end result of the analyses led to convincing visuals that made the intended point, the details of the analysis themselves were hard to follow and evaluate for rigor. An appendix explaining with notation how the behavioral “features” were computed, for example, would have been helpful. I acknowledge that it is a feature of the paper that so many different analysis techniques from neuroscience—dynamical systems to representation analysis to lesion studies—were applied, and the sheer breadth of the analysis was impressive, if difficult to evaluate.
Although I recognize that to do so would be difficult, I think what would add a lot to this paper would be side-by-side comparisons of real neural data with “neural” data from the artificial rodent. Perhaps the behavior would have to be modified so that some tasks match specific neuroscience paradigms, or the sample size for analysis limited to what is realistic for experimental data. It would be neat to see if, say, preparatory activity signals could emerge from training such an artificial rodent to do many tasks, including a head-fixed timed lever pull or some other classic paradigm. Maybe future work can draw a more exact analogy between policy/core layers and specific brain regions.
As to topic fit, this is niche in the context of ICLR; however, showing how state-of-the-art deep RL techniques can be applied to neuroscientific questions may be of interest to machine learning researchers even if they are not also neuroscientists.
Overall, I thoroughly enjoyed this paper and look forward to the new kinds of research that sprout out of it!


Stochastic gradient descent (SGD), which dates back to the 1950s, is one of the most popular and effective approaches for performing stochastic optimization. Research on SGD resurged recently in machine learning for optimizing convex loss functions and training nonconvex deep neural networks. The theory assumes that one can easily compute an unbiased gradient estimator, which is usually the case due to the sample average nature of empirical risk minimization. There exist, however, many scenarios (e.g., graphs) where an unbiased estimator may be as expensive to compute as the full gradient because training examples are interconnected. Recently, Chen et al. (2018)  proposed using a consistent gradient estimator as an economic alternative. Encouraged by empirical success, we show, in a general setting, that consistent estimators result in the same convergence behavior as do unbiased ones. Our analysis covers strongly convex, convex, and nonconvex objectives. We verify the results with illustrative experiments on synthetic and real-world data. This work opens several new research directions, including the development of more efficient SGD updates with consistent estimators and the design of efficient training algorithms for large-scale graphs.

The analysis of the convergence of SGD with biases gradient estimates dates back to Robins&Monroe, but the authors of this paper focused on a recent original algorithm that shows that once can estimate the approximate gradient of a large GNN network, simply by sampling nodes randomly.
When I first read of the paper, I was enthusiastic because I did not know the FastGCN approach presented at ICLR the previous year, which showed that the gradient of a GCN could be efficiently approximated by sampling a subset of the nodes. After reading FastGCN, I was less enthusiastic as most of the originality relied on the consistent estimate of the gradient, when t (number of sampled in the neighbours of the output nodes) increases.
The main contribution of the paper is the proof that the algorithm converge, but there is no theoretical analysis of the key quantity "t", which is the number of sampled nodes in the neighbours of the output nodes. I would expect to see the number of sample grow as the algorithm converge to the optimal solution since the gradient needs less bias when the algorithm converges. However, the authors do not address this point.
I did not into the details of the proofs, but it seems to me that they are quite loose and several details such as the functional spaces, and the boundedness assumptions, are not mentioned. Here are few examples:
- In the first sentence: P(x, y) of data x and associated label y. The space of x, the space of y and the probability space are not defined. In fact, no set in which variables belong is defined in the paper.
- The Theorem 1 is strange to me. I would assume that one needs some assumption of boundedness of Q and finite moments for P to avoid pathological examples where the integral (for the asymptotic expectation) is infinite, but the finite sum G_{st} is always finite, contradicting the limit in theorem 1.
Overall, while proving that the FastGCN algorithm is consistent is important, it is hard to understand how useful the results are and how they can be useful in practice. For example, what can we interpret or what can we learn from the bounds given by Theorem 2?
Finally, I might miss something, but the empirical results showed do not seem to show better gains than the Adam algorithm. The theory shows that the more bias we have, the less accurate we should be, why isn't it apparent in Table 1. Is there something such as the computational cost of Adam, that I'm missing, especially when looking at the graphs? I'm sorry if I did not get the main message of the experiments, but even after reading the paper 3 times, I did not understand what the authors wanted the reader to conclude with these experiments.


The field of Deep Reinforcement Learning (DRL) has recently seen a surge in research in batch reinforcement learning, which aims for sample-efficient learning from a given data set without additional interactions with the environment. In the batch DRL setting, commonly employed off-policy DRL algorithms can perform poorly and sometimes even fail to learn altogether. In this paper we propose a new algorithm, Best-Action Imitation Learning (BAIL), which unlike many offpolicy DRL algorithms does not involve maximizing Q functions over the action space. Striving for simplicity as well as performance, BAIL first selects from the batch the actions it believes to be high-performing actions for their corresponding states; it then uses those state-action pairs to train a policy network using imitation learning. Although BAIL is simple, we demonstrate that BAIL achieves state of the art performance on the Mujoco benchmark.

Summary:
This paper studies the problem of learning a policy from a fixed dataset. The authors propose to estimate a smooth upper envelope of the episodic returns from the dataset as a state-value function. The policy is then learned by imitating the state action pairs from the dataset whose actual episodic return is close to the estimated envelope.
Recommended decision:
The direction of imitating "good" actions from the dataset is interesting. The intuition of estimating an upper envelope of the value function seems reasonable. However, I feel like this paper is not ready to be published in terms of its overall quality, mainly due to the lack of correctness, rigorousness and justification in statements and approaches.
Major comments:
- On the top of page 4:  "Because the Mujoco environments are continuing tasks, it is desirable to approximate the return over the infinite horizon, particularly for i values that are close to the (artificial) end of an episode. To do this, we note that the data-generation policy from one episode to the next typically changes slowly. We therefore apply a simple augmentation heuristic of concatenating the subsequent episode to the current episode, and running the sum in (1) to infinity." I cannot see how this approach is validated. The reset of initial state makes cross-episode cumulative reward from a state s not an approximation to the real return from state s. Estimating the infinite horizon return from finite horizon data is indeed a challenge here and simply cut the return at the end of an episode is be problematic. But the solution proposed by the authors is wrong in principle and cannot be simply justified by "good empirical performance". I feel hard to regard this choice a valid part of an algorithm unless further justification can be provided.
- Statements of theorems (4.1 and 4.2) are non-rigorous and contain irrelevant information: "lambda-smooth" is not an appropriate terminology when lambda is the weight of the regularizer. The actual "smoothness" also depends on the other term in the loss (same lambda does not indicate same smoothness in different objectives). For the same reason, Theorem 4.2 is wrong as changing K also changes the smoothness of the learned function. Proof of Theorem 4.2 in appendix is wrong as the authors ignore the coefficients in the last equation. Theorem 4.1-(1) cannot be true unless how V_\phi is parameterized is given: e.g. if there is no bias term or the regularization is applies to the bias term V will always output 0 as lambda 0-> \infty. The "2m+d" in Theorem 4.1-(2) is irrelevant to this work and cannot be justified without more detailed statements about how the network is parameterized. I appreciate the motivation that the authors try to validate the use of their objective to learn a "smooth upper envelope" but most of these statements are somewhat trivial and/or wrong section 4.1 does not actually deliver a valid justification.
- The use of "smooth upper envelope" itself can bring both over-estimation and under-estimation. For example, if one can concatenate different parts from different episodes to get a trajectory with higher return, the episodic return for the states along this trajectory is an under-estimate. Although it is fine to use a conservative estimate it would be better to be explicit about this and explain why this may not be a concern. On the other hand, it can bring over estimation to the state-values due to the smoothness enhanced to the fitted V. It would be better to see e.g. when these concerns do not matter (theoretically) or they are not real concerns in practice (by further inspecting the experiments).
- Regarding Experiments: Why Hopper, Walker, HalfCheetah are trained with DDPG while Ant is trained by SAC? The performance of Final-DDPG/SAC after training for 1m steps looks way below what SAC and TD3 can get. Is it because they are just partially trained or noise is added to them? The baseline online-trained policy should not contain noise for a fair comparison. That said, in batch RL setting it is not necessary to compare to online-trained policy because it is a different setting. But if the authors want to compare to those, choice of baseline should be careful. An important baseline which is missing is to run vanilla DDPG/TD3/SAC as a batch-mode algorithm.
Minor comments:
- Section 3, first paragraph: It is not very meaningful to say "simulators are deterministic so deterministic environments are important". Simulators are made by humans so they can be either deterministic or stochastic. "many robotic tasks are expected to be deterministic environments" is probably not true. I do not view "assuming deterministic envs" as a major limitation but I do not find these statements convincing as well. Similarly, the argument for studying non-stationary policy seems unsupportive: if the dataset comes from training a policy online then why do we care about learning another offline policy rather than just use or continue training the online policy. One argument I can see is that the online policy is worse. But the fact that these policies are worst than running e.g. SAC for a million steps makes the motivation questionable. Again, I do not view "choice of setting" as a limitation but I just find these statements a bit unsupportive.
Potential directions for improvement:
To me the main part of the paper that looks problematic is Section 4.1 (both the approximation of infinite horizon returns and the theorems). It would be better to see a more rigorous and coherent justification of this approach (or some improved version), e.g. by either presenting analysis that is rigorous, correct and actually relevant or leave the space for more detailed empirical justification (e.g. whether potential over/under-estimating happens or not, comparing the estimated V to real episodic return of the learned policy).


Deep learning models are vulnerable to adversarial examples crafted by applying human-imperceptible perturbations on benign inputs. However, under the blackbox setting, most existing adversaries often have a poor transferability to attack other defense models. In this work, from the perspective of regarding the adversarial example generation as an optimization process, we propose two new methods to improve the transferability of adversarial examples, namely Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM). NI-FGSM aims to adapt Nesterov accelerated gradient into the iterative attacks so as to effectively look ahead and improve the transferability of adversarial examples. While SIM is based on our discovery on the scale-invariant property of deep learning models, for which we leverage to optimize the adversarial perturbations over the scale copies of the input images so as to avoid "overfitting" on the white-box model being attacked and generate more transferable adversarial examples. NI-FGSM and SIM can be naturally integrated to build a robust gradient-based attack to generate more transferable adversarial examples against the defense models. Empirical results on ImageNet dataset demonstrate that our attack methods exhibit higher transferability and achieve higher attack success rates than state-of-the-art gradient-based attacks.

This paper studies how to generate transferable adversarial examples for black-box attacks. Two methods have been proposed,  namely Nesterov Iterative Fast Gradient Sign Method (NI-FGSM) and Scale-Invariant attack Method (SIM). The first method adopts Nesterov optimizer instead of momentum optimizer to generate adversarial examples. And the second is a model-augmentation method to avoid "overfitting" of the adversarial examples. Experiments on ImageNet can prove the effectiveness of the proposed methods.
Overall, this paper is well-written. The motivation of the proposed methods are generally clear although I have some questions. The experiments can generally prove the effectiveness.
My detailed questions about this paper are:
1. The motivation in Section 3.1, which regards generating adversarial examples as training models, and transferability as generalizability, is first introduced in Dong et al. (2018). The authors should acknowledge and refer to the previous work to present the motivation.
2. It's not clear why deep neural networks have the scale-invariant property. Is it due to that a batch normalization layer is usually applied after the first conv layer to mitigate the effect of scale change?
3. It's not fair to directly compare DIM with SI-NI-DIM (also TIM vs. SI-NI-TIM; TI-DIM vs. SI-NI-TI-DIM), since SI-NI needs to calculate the gradient over 5 ensembles. It's better to compare the performance of two methods with the same number of gradient calculations.
4. Is there an efficient way of calculating the gradient for scale-invariant attacks like translation-invariant attacks in Dong et al. (2019)?


Unsupervised text encoding models have recently fueled substantial progress in Natural Language Processing (NLP). The key idea is to use neural networks to convert words in texts to vector space representations (embeddings) based on word positions in a sentence and their contexts. We see a strikingly similar situation in spatial analysis, which focuses on incorporating both absolute positions and spatial contexts of geographic objects such as Points of Interest (POIs) into models. A general space encoding method is valuable for a multitude of tasks. However, no such general model exists to date beyond simply applying discretizing or feed forward nets to coordinates, and little effort has been put into jointly modeling distributions with vastly different characteristics, which commonly emerges from GIS data. Meanwhile, Nobel Prize-winning Neuroscience research shows that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for encoding space and are critical for recognizing places and for path-integration. Therefore, we propose a representation learning model called Space2vec to encode the absolute positions and spatial relationships of places. We conduct experiments on two real world geographic data 1) predict types of POIs 2) image classification leveraging their locations. Results show that because of its multi-scale representations Space2vec outperforms well established ML approaches such as RBF kernels, multi-layer feed forward nets, and tile embedding approaches for the Location Modeling tasks and image classification tasks. Detailed analysis shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2vec 's multi-scale representation can handle distributions at different scales.  (Qi et al., 2017), POI type similarity reasoning , land cover classification (Kussul et al., 2017), and geographic question answering (Mai et al.,  2019). Developing general models for vector space representation of any point in space would pave the way to all these applications.
However, existing models often utilize specific methods to deal with geographic information and often disregards geographic coordinates. For example, Place2Vec  converts the coordinates of POIs into spatially collocated POI pairs within certain distance bins, and does not preserve information about distance and direction between POIs. Li et al. (2017) propose DCRNN for traffic forecasting in which the traffic sensor network is converted to a distance weighted graph which necessarily forfeits information about the spatial layout of sensors. There is, however, no general 1 link to repository redacted during double-blind review.

The paper introduces Space2Vec, a space representation learning model. The work is motivated by the biological grid cell’s multi-scale periodic representations and the success of representation learning of NLP. So, the key idea behind the model is two-fold. On one hand, utilize the position information and the context associated with the position. On the other hand, the authors build a multiscale point space encoder based on Theorem 1 (in the paper), which was previously proved by Gao et al. (2019).
The multi-scale point feature encoder is novel. The experimental results turn out that the whole model is good at predicting features using only location information but does not outperform the RBF kernel (on validation) in terms of using spatial context modeling.
One core selling point of this paper is dealing with location distributions with very different characteristics. This is very well motivated at the beginning. More analysis/statistics would help better understand how the model "theory" wins Table one. See comments on experiments.
I have some comments/questions about model architecture and also experimental results/analysis.
1. Regarding Contextual Embedding
-- The encoder of this paper is not doing much “contextual embedding”.  The encoder typically encodes features of each position independently thus lead to very local embedding.
-- The location decoder would reconstruct the same (or similar) type of point features given embeddings of locations of the same type. As the distributions of different types of locations are very different, an encoder capable to deal with multi-scale data is crucial here.
-- The Spatial context decoder, like a context-dependent language model, would reconstruct the current position features, given the neighboring information.
-- Overall, unlike many existing pre-training models in NLP with deep encoders, the full model of this paper is with very local encoder, while the decoder does the most work of “gathering contextual information".
As you claim your model to be "a general-purpose space representation model", can you describe/specify how you would use your model for other tasks? Will you take some intermediate output of decoders to be representations? Or will you fine-tune the whole encoder-decoder?
2. For experiments:
a.	I do not prefer saying your method outperforms RBF in the “spatial context modeling” task when you getting worse validation set performance. It is interesting that RBF is stronger in terms of validation.
b.	Is it possible for the authors to do some statistics on the different types of locations? For your first task “location modeling”, should we expect to see that your model does not have very bad performance on certain types of locations while other non-multi-scale approaches do? This is trying to provide better support to one of your core contributions.
c.    Again, to claim the model can be widely applied, try more tasks?
To clarify my "experience assessment", I mean I read many related representation learning papers rather than specific papers related to GIS data.
My actual rating for the paper is between weak reject to weak accept (but the system does not have intermediate choices). I would like to hear the author's feedback to further revise the rating.


Understanding the optimization trajectory is critical for understanding training of deep neural networks. We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along the trajectory. Based on a theoretical model, we conjecture that using a high learning rate or a small batch size leads SGD to regions of loss landscape, typically early during training, characterized by (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We refer to the point on the training trajectory after which these effects hold as the break-even point. We demonstrate these effects empirically for a range of deep neural networks applied to different tasks. Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a higher learning rate to improve loss surface conditioning compared to a network without BN layers.

This work analyzes the optimization of deep neural networks from the point of view of the different learning trajectories obtained during different learning settings as brought about by different hyperparameters in optimization. Specifically, the authors consider how the batch size (S) and the step-size (eta) hyper-parameters modify the learning trajectory. The authors conduct their analysis for networks with and without BatchNorm.
To get a quantitative understanding of the different trajectories of the optimization landscape the authors monitor and analyze the
A.  Hessian of the network with respect to the parameters
B. The covariance matrix of the per-sample gradients
The majority of the work focuses on monitoring A and B for various hyper-parameter regimes.
The authors discover a nice and clear trend, the leading eigenvalues of H (the Hessian) and K (per sample gradient covariance) are
1. Heavily correlated to each other (Figure.2 left)
2. Positively correlated with batch size (S) (Figures 3-5)
3. Negatively correlated with step-size (eta) (Figures 3-5)
The 3 trends presented empirically all originate and qualitatively agree very well with equation (1) of the paper that describes stability regions in training.  Equation (1) directly prescribes the maximum allowed eigenvalue for the Hessian matrix H for a given a learning rate, and batch-size to ensure stability. For equation (1) to hold, the authors make  the assumption that H and K share a leading eigenvector direction (this assumption is partially corroborated on the CIFAR10 dataset). The empirical analysis presented in the paper analyzes training on the CIFAR10, ImageNet, IMDB, and the MNLI datasets and uses a variety of common network architectures to evaluate the empirical claims. In general, I am quite happy with the breadth and depth of the experimental results presented in the paper!
The paper makes a few observations in addition to trends 1-3, It argues that BatchNorm networks need a large learning rate to exhibit smaller dependence on the subspace of the leading eigenvectors of K,  and that the networks reach a breaking point early in training where the leading eigenvalues of H and K are chosen by the trajectory.
Overall the paper is well written and the numerous experimental results are quite impressive. Nonetheless I have a few problems that are discouraging me from fully accepting this paper, so I am currently borderline accepting this paper.
The following problems should be addressed:
i. The theoretical results in the paper are very similar to the results given in Wu et al. (as mentioned in the main body and Appendix A). Moreover, the modified proofs presented in the appendix are not fully explained and are difficult to follow so I do not find them rigorous. It also seems like some of the empirical analysis is also presented in Wu et al.
ii. As stated, Conjecture 2 is not empirically supported in the paper. Currently, Conjecture 2 proposes that for  training with smaller batches or larger learning rates, the reciprocal of the ''conditioning number'' ( least POSITIVE eigenvalue / leading eigenvalue)  reaches a larger maximum value during training. Instead the authors "estimate" the least positive eigenvalue by the trace of the matrix which is very much a different notion than the least positive eigenvalue and also displays a different behavior.
It seems maybe measuring the trace as an un-normalized quantity for the average eigenvalue is more suitable but that is different from the least positive eigenvalue. Perhaps of the same flavor would be a statement on the stable rank of the matrices in question.
iii. Figure 1 (left) is quite confusing. I hope the authors can help clarify the heatmap shown along with the trajectories. The caption mentions that the colors signify the value of the leading eigenvalue of K.  First, since this is an explicit quantity can the authors provide a value range for the heatmap? Second, and more importantly, how is this quantity, which depends on K, can be computed for all the points in the 2D grid of UMAP to form this heat map? Recall K is a quantity that is computed for a given parameterization value and fixed number of mini-batch examples.
iv. Lack of clarity and rigor for the eigenvalue approximation: In appendix B the authors discuss approximating the eigenvalues of K and H, I think it is okay to develop numerically efficient approximations to the quantities in question but the presentation is not very intuitive nor rigorous. In the first paragraph the authors mention that since H is ill conditioned, a small subset of the training examples gives a good approximation to the Hessian, can the authors elaborate on this statement?
Other more minor comments:
Even though the default step size and batches are mentioned in appendix D it could be nice to mention them in the figures that they are presented in.
Page 7 Fig 5b | The labels of the figures should be adjusted so that they do not overlap with the adjacent figures.
Minor typos/grammar:
page 1 caption of figure 1 | vertical line marks -> the vertical line marks
page 3 in definitions paragraph | let us denote loss -> let us denote the loss
page 3 3rd assumption | training trajectory -> the training trajectory
page 3, 3rd assumption | escape region-> escape a region
page 7, other experiments paragraph | depend -> depends,  possibly?
page 7, Figure 5 (b) Middle |  lambda_k to lambda_k^1


The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-distortion tradeoff (Blau & Michaeli (2018)). Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE.

Impact of the latent space on the ability of GANs to fit the distribution
This paper purports to study the behavior of latent variable generative models by examining how the dimensionality of the latent space affects the ability of said models to reconstruct samples from the dataset. The authors perform experiments with deterministic autoencoders and WGAN-GP on CIFAR, CelebA, and random noise, and measure MSE and FID as a function of the dimensionality of z.
My take: This paper does not offer any especially intriguing insights, and many of the conclusions the authors draw are, in my opinion, not supported by their experiments. The paper is confusingly written and hard to follow—throughout my read I struggled to determine what the authors meant, and it was not clear to me what this paper is supposed to contribute. The potential impact of this paper is very low, and I argue strongly in favor of rejection.
Notes:
My most critical complaint is the central experiment set of the paper: measuring MSE and FID as a function of Dim-Z for two models. First of all, the authors assume that the reduction in MSE as a function of dim Z is indicative of increased memorization in the GAN models. I disagree that this is the case; since the GAN-based reconstruction is done via optimization it is unsurprising that increasing dim Z increases the reconstruction quality, as you are literally giving the model more degrees of freedom with which to fit the individual samples. This is glaringly evident in Figure 8, where increasing dim Z renders the model better able to reconstruct a sample of pure noise, which is almost certainly not in its normal output distribution, (or if it is, is in there with staggeringly low probability). The fact that the higher dim-z models are better able to reconstruct the noise supports the notion that it is merely the number of degrees of freedom that matter in these experiments, rather than what the model actually learns.
Second, it is important to note that FID can be easily gamed by memorization, and for an autoencoder (which has direct access to samples) with an increasingly large bottleneck it is unsurprising that increasing dim-Z tends to decrease the FID, and equally unsurprising that increasing the dim-Z for the GAN does not tend to improve results, since this does not really allow the model increased memorization capacity (not to mention the relationship between performance and dim-Z has been explored before in GAN papers).
Third, the organization of the experimental section makes it very difficult to infer what the authors are trying to conclude from these experiments. The noise experiment is presented, but no insights or conclusions are drawn, other than (a) noting that the model has a harder time reconstructing the noise than training samples and (b) that lower dim models have a harder time reconstructing the noise, both of which are just restatements of the information presented in the figure rather than an actual insight or conclusion.
-I’m not really sure what the experiment in section 3 is supposed to show. This experiment is poorly described and lacking details. First of all, what is the loss function used there? Is this the output of the discriminator or the MSE between the output and a target sample? How is z* found and what does it represent—is it just a randomly sampled latent, or is it the latent that corresponds to the z-value which minimizes some MSE loss for a target sample? If it’s the latter, why is this notation not introduced until section 4? If it’s a latent, why are you calling it a data point? Why are there no axes and no scales on these plots? How is it clear that there is an optimization path from z0 to z*; is that supposed to be inferred from z0 having a higher value than z* or appearing to be directly uphill from z*, because it’s not clear to me that that is the case in Figure 2a. In general I did not find this experiment to support the conclusions the authors draw.
-Figure 4: It is important to note that FID can be trivially gamed by memorizing the dataset, and an autoencoder is much more well-suited to memorizing the dataset as it has direct access to samples (whereas a GAN must get them through the filter of the discriminator). Authors should test interpolation or hold-out likelihood for the  autoencoder, these models are not directly comparable in this manner.
-The presentation of this paper is, in general, all over the place. The authors should focus on writing such that each point follows the next, building progressively towards their results and insights, and making it easy for a reader to follow their train of thought.
“In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task.” At what point is this lower bound (either in terms of model capacity or latent space dimensionality) specified in the paper? Is Figure 3 supposed to be this lower bound, because to me it only indicates that the autoencoder tends to have a lower MSE, not that it conclusively lower bounds the memorization capacity of the GAN. Wouldn't a method like GLO which directly optimizes for memorization be a better lower bound for this, anyhow?
“We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire distribution” What does this phrase mean? Are the authors referring to the entire distribution of natural images, of which the training set is assumed to be a subset? Or do they mean the output distribution of the generator? This was not clear to me.
Minor:
-“ style transfer by Karras et al. (2018),”, and “anomaly detection (Shocher 2018).” StyleGAN is not a style transfer paper, and InGAN  is not about anomaly detection. Please do not incorrectly summarize papers.
-“Trained GAN newtworks” While amusing, this is a typo. Please thoroughly read your paper and correct all typos and grammatical mistakes, like “combiniation.”
-“…that an accurate reconstruction of the generator manifold is possible works using first order methods”  The word “works” seems to be out of place here. Again, please thoroughly proofread your paper.
-The legend in Figure 2 has a white background, making the white x corresponding to z0 invisible. Please fix this, and add appropriate axes to this plot.
-Figure 7 and 8 may in fact have error bars, but they are not described (are they 1 std or another interval?) or referenced, and in Figure 8 (if these are error bars) they are nearly invisible.