The capability of making interpretable and self-explanatory decisions is essential for developing responsible machine learning systems. In this work, we study the learning to explain problem in the scope of inductive logic programming (ILP). We propose Neural Logic Inductive Learning (NLIL), an efficient differentiable ILP framework that learns first-order logic rules that can explain the patterns in the data. In experiments, compared with the state-of-the-art methods, we find NLIL can search for rules that are x10 times longer while remaining x3 times faster. We also show that NLIL can scale to large image datasets, i.e. Visual Genome, with 1M entities.
: A scene-graph can describe the relations of objects in an image. The NLIL can utilize this graph and explain the presence of objects Car and Person by learning the first-order logic rules that characterize the common sub-patterns in the graph. The explanation is globally consistent and can be interpreted as commonsense knowledge.
The recent years have witnessed the growing success of deep learning models in a wide range of applications. However, these models are also criticized for the lack of interpretability in its behavior and decision making process (Lipton, 2016;Mittelstadt et al., 2019), and for being data-hungry. The ability to explain its decision is essential for developing a responsible and robust decision system (Guidotti et al., 2019). On the other hand, logic programming methods, in the form of first-order logic (FOL), are capable of discovering and representing knowledge in explicit symbolic structure that can be understood and examined by human (Evans & Grefenstette, 2018).
In this paper, we investigate the learning to explain problem in the scope of inductive logic programming (ILP) which seeks to learn first-order logic rules that explain the data. Traditional ILP methods (Galárraga et al., 2015) rely on hard matching and discrete logic for rule search which is not tolerant for ambiguous and noisy data (Evans & Grefenstette, 2018). A number of works are proposed for developing differentiable ILP models that combine the strength of neural and logicbased computation (Evans & Grefenstette, 2018;Campero et al., 2018;Rocktäschel & Riedel, 2017;Payani & Fekri, 2019;Dong et al., 2019). Methods such as ∂ILP (Evans & Grefenstette, 2018) are referred to as forward-chaining methods. It constructs rules using a set of pre-defined templates and evaluates them by applying the rule on background data multiple times to deduce new facts that lie in the held-out set (related works available at Appendix A). However, general ILP problem involves several steps that are NP-hard: (i) the rule search space grows exponentially in the length of the rule; (ii) assigning the logic variables to be shared by predicates grows exponentially in the number of arguments, which we refer as variable binding problem; (iii) the number of rule instantiations needed for formula evaluation grows exponentially in the size of data. To alleviate these complexities, most works have limited the search length to within 3 and resort to template-based variable assignments, limiting the expressiveness of the learned rules (detailed discussion available at Appendix B). Still, most of the works are limited in small scale problems with less than 10 relations and 1K entities.
On the other hand, multi-hop reasoning methods (Guu et al., 2015;Lao & Cohen, 2010;Lin et al., 2015;Gardner & Mitchell, 2015;Das et al., 2016) are proposed for the knowledge base (KB) completion task. Methods such as NeuralLP  can answer the KB queries by searching for a relational path that leads from the subject to the object. These methods can be interpreted in the ILP domain where the learned relational path is equivalent to a chain-like first-order rule. Compared to the template-based counterparts, methods such as NeuralLP is highly efficient in variable binding and rule evaluation. However, they are limited in two aspects: (i) the chain-like rules represent a subset of the Horn clauses, and are limited in expressing complex rules such as those shown in Figure 1; (ii) the relational path is generated while conditioning on the specific query, meaning that the learned rule is only valid for the current query. This makes it difficult to learn rules that are globally consistent in the KB, which is an important aspect of a good explanation.
In this work, we propose Neural Logic Inductive Learning (NLIL), a differentiable ILP method that extends the multi-hop reasoning framework for general ILP problem. NLIL is highly efficient and expressive. We propose a divide-and-conquer strategy and decompose the search space into 3 subspaces in a hierarchy, where each of them can be searched efficiently using attentions. This enables us to search for x10 times longer rules while remaining x3 times faster than the state-of-theart methods. We maintain the global consistency of rules by splitting the training into rule generation and rule evaluation phase, where the former is only conditioned on the predicate type that is shared globally.
And more importantly, we show that a scalable ILP method is widely applicable for model explanations in supervised learning scenario. We apply NLIL on Visual Genome (Krishna et al., 2016) dataset for learning explanations for 150 object classes over 1M entities. We demonstrate that the learned rules, while maintaining the interpretability, have comparable predictive power as densely supervised models, and generalize well with less than 1% of the data.

This paper presents a model for effectively hierarchically ‘searching’ through the space of (continuously relaxed) FOL formulas that explain the underlying dataset. The model presented employs a three-level architecture to produce logic entailment formulas, skolemized with a set of skolem functions, i.e. ‘operators’. The three-level search, which is implemented via a stack of transformers, first searches through a space of variable and predicate embeddings to select operators, after which it searches through the space of predicates to form primitive statements of predicates and operators, and finally it generates a number of formulas from the previously ‘selected’ primitive statements. The model is applied on a toy task to showcase its speed, a knowledge base completion task, and modeling the visual genome dataset, where the model shows the ability to induce meaningful rules that operate on visual inputs. The presented benefit of the model is scalability, ability to induce explainable rules, and the ability to induce full FOL rules.
The paper is well motivated from the explainability perspective and based on the evaluation does what it claims. The model is fairly elaborate, yet manages to be faster than the competing models. In general, I think the model itself is a welcome addition to the area of neuro-symbolic models, especially logic-inducing models, and that the evaluation done is appropriate (with a few caveats). However, my major critique of the paper is in its clarity.
In the current state, the paper is quite difficult to read, partially due to its density, partially due to its confusing presentation, notational issues and missing details. It would be difficult to reimplement the model just by reading the paper, which brings me to ask: will you be releasing the code?
I would be willing to accept the paper if the authors improve the notation and presentation significantly. I’m enumerating issues that I found most confusing:
Result presentation:
- The figure captions are uninformative. In figure 1, one needs to understand what the graph is before reading the text at the end of the paper which explains what that is. It is not clear from the figure itself. Figure 2 presents the model, but it does not follow the notation from the main body of the paper.
- Table 1 is missing SOTA models. TransE is definitely not one of the better models out there: check [2, 3, 4] for SOTA results which are significantly higher than the ones presented. I would not at all say that that invalidates the contribution of the paper, but readers should have a clear idea of how your model ranks against the top performing ones.
- Please provide solving times for TransE, as it has to be by far the fastest method, given that it is super-simple.
- Are provided times training or inference times (in each of the table/figure) because one gets mixed statements from the text?
- In which units is time in Table 1?
- Can you include partially correct or incorrect learned rules in Table 4? It would be great to get some understanding of what can go wrong, and if it does, what might be the cause.
Model presentation and notation:
- You mention negative sampling earlier in the text but then don’t mention how you do it
- Notation below (6) is utterly confusing and lacking: what is s_{l,i}^(t), is there a softmax somewhere, what are numbers below ‘score’
- What is the meaning of e_+ and e_- given that you omit details of negative sampling?
- The notation does not differentiate between previous modules well so there’s V^(t-1) across modules, and it is not clear which one is used at the end --- last choice over V^(0) - V^(T) is over the LAST output, not the output from previous steps?
- The notation in the text does not follow the notation in the figure (V_ps, V_\phi)
- Notation gets quite messy at some points, e.g. R^c^(0), is e_X and e_Y in H or not? Is H_b there too?
- The differentiation of embedding symbols is not done well. H_b is an embedding for binary predicates, or a set of predicates? Does that mean that there is only a single embedding for a binary predicate and a single embedding for its operator (thought I thought that operators have an embedding, each)?
- The explanation of what a transformer does is not particularly useful, the paper would benefit more from an intuitive explanation, such as that the transformer learns the ‘compatibility’ of predicates (query) and input variables (values), etc.
- The ‘Formula generation’ subsection lacks the feel of fitting where it is right now, given that the notation in it is useful only in ‘Formula search’ paragraph. The other thing is that that subsection is wholly unclear: where do p and q come from, do they represent probabilities of predicates P and Q? How are they calculated? Does your construction imply that a search over the (p, q, a) space is sufficient to cover all possibilities of and/not combinations? In which case alpha is a vector of multiple values different for each f_s  in (5) or no? It is unclear
- What is the relationship between alpha_0 and alpha_1, sum(alpha_0, alpha_1) = 1? Are alpha_0, and alpha_1 scalars, and how are they propagated in eq 5? Because if they’re just scalars, the continuous relaxation of formulas represented in (5) cannot cover all combinations of p, q, not and and. What is the shape of the alpha vector?
p and q are probabilities of what, predicates P and Q? How are they produced?
- are \phi functions pretrained?
Related work:
- I do not condone putting it into the appendix, but I’m not taking it as a make-or-break issue.
- It is notably missing an overview of symbolic ILP, and a short spiel on link prediction models (given that you compare your model to TransE, as a representative of these models)
The paper is riddled with typos and consistent grammatical errors. I would suggest re-checking it for errors. Examples include:
- singular/plural noun-verb mismatches (rules..and does not change -> do, Methods ...achieves -> achieve)
- Q_r^(t) - choice of left operands -> right
- An one-hot -> a one-hot
Minor issues:
- The claim that the NeuralLP is the only model able to scale to large datasets is a bit too strong given [1]
- You say “We could adopt” but you “do adopt”
Clarification questions:
- Are 10 times longer rules of any use? Can you provide an example of such rules, a correct one and an incorrect one?
- How many parameters does your model use? What are the hyperparameters used in training?
- How big is T in your experiments, and why?
- Why are queries only binary predicates?
- How discrete do these rules get? You sample them during testing and evaluation, but are embeddings which encode them approximately one-hot, so that the (Gumbel-softmax) sampling produces the same rules all over again or are these rules fairly continuous and sampling is just a way to visualise them, but they are internally much more continuous?
- Just to confirm, \phi are learned, right? Given that they are parameterised with a predicate matrix, and that matrix is trainable?
- Do you have a softmax on the output? It seems  f^*(T+1) should be a softmaxed value?
- The rule generation generates a substantial number of rules, right? What might be the number of these rules? Does the evaluation propagate gradients through all of them or to a top-k of them?
- Why is the formula in Eq.4 written the way it is. I assume it can be written differently, for example “Inside(\phi_Identity(X), \phi_Car())) and Inside(\phi_Clothing(), \phi_Identity(X))”. I do not understand why Clothing was treated as an operator and Car as a predicate, while treating Inside both as an operator and a predicate. Sure, nothing in the model forces it to consistently represent a formula in the same way always, but an example such as this one would need a good explanation why you chose it or at least mention that this is but one way to present it.
- Why is Identity(X) used? Is it because you did not want to mix in variable embeddings during the primitive statement search?
[1] Towards Neural Theorem Proving at Scale
[2] Canonical Tensor Decomposition for Knowledge Base Completion
[3] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
[4] TuckER: Tensor Factorization for Knowledge Graph Completion


The recent advancements in graph neural networks (GNNs) have led to state-of-theart performances in various applications, including chemo-informatics, questionanswering systems, and recommender systems. However, scaling up these methods to huge graphs such as social network graphs and web graphs still remains a challenge. In particular, the existing methods for accelerating GNNs are either not theoretically guaranteed in terms of approximation error, or they require at least a linear time computation cost. In this study, we reveal the query complexity of the uniform node sampling scheme for GraphSAGE, the graph attention networks (GAT), and the graph convolutional networks (GCN). The key advantage of our analysis is that the complexity is completely independent of the numbers of the nodes, edges, and neighbors of the input and depends only on the error tolerance and confidence probability while providing theoretical guarantee for the approximation error. To the best of our knowledge, this is the first work to give a theoretical guarantee of approximation for GNNs within a constant time complexity. Through experiments using synthetic and real-world datasets, we demonstrate the speed and precision of the node sampling scheme and validate our theoretical results.

In this paper, the authors provide a theoretical framework for characterizing the approximation guarantees provided by node sampling to estimate embeddings in various GNN architectures. In particular, they prove several PAC learning-style bounds on the embedding and gradient estimation when using node sampling approaches. They also observe that since the number of nodes selected for sampling is not dependent on the size of the graph, this amounts to a constant time operation for determining embedding and gradient estimates.
Importantly, a variety of existing works have already proposed node sampling (“constant-time GNNs”), so the contribution of this work lies in the theoretical bounds. A moderate set of experiments suggests the empirical behavior follows the theoretical expectations.
Overall (and described in the comments below), I believe this paper provides some interesting theoretical results for an approach that is widely-used in practice. However, I believe the title and abstract are misleading; this paper does not suggest some new sampling strategy; it only gives a theoretical approximation error for existing approaches and network architectures.
Comments
While the authors do make this clear later in the paper, the contributions of this work are only the theoretical bounds; the node sampling algorithms have been proposed in existing work. Both the title and abstract are misleading.
Still, the theoretical bounds do provide an interesting understanding of an approach which is commonly used in practice.
It would be helpful to give the intuition behind the assumptions and proofs. Currently, it is difficult for a non-expert to follow the reasoning and practical implications of much of this work.
Currently, it is not clear if these results are “constructive”, in the sense that they do not obviously suggest some new sampling strategy, etc. Of course, it is still interesting to characterize commonly-used practices, but one does wish the theory suggested, say, a better sampling strategy, etc.
Typos, etc.
“r”, the number of sampled nodes, is not defined in the main body of the paper.
The text in Footnote 1 is a little confusing, in that the text refers to approximating the embedding, while the text in the footnote describes a prediction problem. A reader could easily interpret this to mean that the error on the downstream prediction problem will be bounded (which is not what this work shows).
The text mentions that O_G (v,i) refers to the i^th “neighborhood” of v. Presumably, this should be “neighbor”.
“we take” -> “We take”


Text summarization aims to extract essential information from a piece of text and transform it into a concise version. Existing unsupervised abstractive summarization models use recurrent neural networks framework and ignore abundant unlabeled corpora resources. In order to address these issues, we propose TED, a transformer-based unsupervised summarization system with pretraining on largescale data. We first leverage the lead bias in news articles to pretrain the model on large-scale corpora. Then, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are abstractive and containing even higher proportions of novel tokens than those from supervised models.

Paper's Claims
The paper introduces a new unsupervised abstractive summarization approach called TED, using a Transformer encoder and decoder. Their main contributions are as follows:
1) Pretraining the encoder and decoder on news articles using the first beginning as the target summary.
2) Fine-tune on other datasets using so-called theme modeling, and separately a denoising loss.
3) TED's performance is claimed to significantly improve over GPT-2 while not being too far from the best unsupervised extractive summarization results.
Decision
Edit: After revisions and discussions, I recommend we accept this paper.
I am leaning towards accepting this paper mostly because of the contribution #1 above. Unsupervised learning using large quantities of text that have the property of being typically written in a style that synthesizes information in the first 1-3 sentences is a powerful idea. That the performance is improved compared to other unsupervised abstractive summarization confirms the importance of this approach.
However the importance of and justification for the fine-tuning steps are comparatively much more limited in my opinion. Also, some important details about the preprocessing for pre-training appear to be missing and they could be quite important.
Detailed arguments for decision
I view this effort as aiming to reproduce the BERT approach in the context of abstractive summarization, which is a good idea. The most clever contribution is in leveraging un-labeled text using the first few sentences as the target summary for pretraining. The results of just this part are already beating previous approaches, while not requiring any in-domain data, which is quite powerful.
However, some relatively important details regarding the methodology are omitted or only glossed over and it would greatly contribute to making this work more reproducible if the details were included (see my detailed notes below, notably regarding section 2.2).
On the fine-tuning steps, I have several worries. First, why not fine-tune using supervised learning, as would be the analog to the BERT approach? Instead the authors go out of their way to do in-domain unsupervised learning, which provides a boost, yes, but still doesn't compare positively to extractive and/or supervised methods. Second, why not perform the theme modeling and denoising also -- or rather only -- on the unlabelled pretraining data? Why should it be done on the in-domain fine-tuning data instead (while not using the most valuable piece of in-domain information, namely the example summaries)? After all, it's a fully unsupervised approach and it can actually be performed on any text at all, whether a summary for it exists or not.
Again regarding the unsupervised approach, and to push the BERT analogy further, I'm wondering why not initialize the pretraining model with a BERT-style trained model? After all we could imagine building a system that adds more and more in-domain characteristics sequentially: first pretrain a BERT model, then fine-tune to summarization using what this paper calls pretraining, and then finally fine-tune again to a specific summarization domain.
So, to conclude, I find that this paper goes in the right direction and introduces important ideas for pretraining and fine tuning unsupervised abstractive summarization models, but that some decisions about how to use the various ideas (theme and denoising but no supervised learning, in-domain vs during pretraining) have not been explored enough.
Extra notes
page 2, second line: pretrainleverages (typo)
section 2.1: fix first sentence to make it an actual sentence.
section 2.2: "we obtain three years of online new articles ... via a search engine" please be more specific about your methodology.
section 2.2: You should double check more throughly that there is no data leakage in test. There could be articles about the same exact events, years apart, for example. I doubt that this would be a big effect, but there are easily ways to find highly similar articles between the pretraining data and test data to make sure.
section 2.2: "Next we conduct following data cleaning" fix (typo?). Also that sentence probably belongs to the next paragraph.
section 2.2: Why did you pick the values that you did for the preprocessing heuristics (such as between 10-150 words, 150-1200 words, 3 sentences and not 2 or 1 or 4, the ratio 0.65, etc.)? Were other values tried?
section 2.2: You mention you end up with 21.4M articles. How many were there to start with? What's the filtering ratio?
section 2.2: You mention that you pick the model with the best ROUGE-L score on the validation set. How many models were there? What was different between them?
section 2.2, OOV Problem: the information in this whole subsection would fit better in 2.1 where 'tokens' are left generic without specifying which type of token you're considering.
Figure 1: I find the upper part of this figure very confusing. Why are there arrows going from the encoder/decoder to a summary, to theme loss, to article and back to encoder/decoder? It's important that the summary is never seen by the theme loss otherwise it's not unsupervised anymore, and I also don't see why the arrow would go through article *after* theme loss. I assume there must have been a mistake, please fix.
section 2.4: "the sequence is slightly shuffled by applying a permutation /sigma such that ..." The formula given here tells me that all token indices are shuffled with another token within a window k. That seems like a lot of moving around, and also depending on the implementation a token from the beginning could possibly end up at the very tail of the sentence by being picked iteratively again and again, thus falling outside the permutation distance k. Please provide more details on how this is done and a justification for why it was decided to do it this way.
Section 3.1: I'd like to know how long (preferably number of words, or at least number of wordpiece tokens) the summaries generated are. What determines how long they are, is it a fixed size, or the model decides to stop on his own (or when hitting some limit), or something else?
section 4.2: Do you have any idea why your unsupervised approach yields more novel n-grams than a the supervised model you compare against? This can be good as much as it can be bad, in that it could be going off-track. Yes humans have high novelty, but high novelty in itself isn't necessarily good. I don't find the argument that have more novel ngrams is intrinsically, necessarily good, compelling. If I'm wrong, then it would be nice to have better explanation in the paper.


Creating a useful representation of the world takes more than just rote memorization of individual data samples. This is because fundamentally, we use our internal representation to plan and solve problems. In this paper, we introduce Plan2Vec, an unsupervised representation learning objective inspired by graph-planning algorithms and value-based reinforcement techniques. By abstracting away low-level control with a learned local metric, we show that it is possible to learn representations that inform long-range structures, entirely passively from high-dimensional sequential datasets that assume no access to action data or an expert policy. Plan2vec learns a latent space through value iteration on the graph formed by the data points, connected via a local metric function trained contrastively from context. We show that the global metric on this learned embedding can be used to plan with O(1) complexity by linear interpolation. This exponential speedup is critical for planning with a learned representation on any problem containing non-trivial global topology. We demonstrate the effectiveness of Plan2Vec on simulated as well as two real-world image datasets, showing that Plan2Vec can effectively acquire representations that carry long-range structure to accelerate planning. Additional results and videos can be found at https://sites.google.com/view/plan2vec.

## Paper Summary
While cast slightly differently in the intro, it seems to me that this paper learns a goal-conditioned value function that is used at test time to construct visual plans by selecting an appropriate sequence of data points from the training data. Similar to prior work they learn a local distance metric without supervision using a temporal proximity objective and then construct a graph of the training data points using this metric. The main novelty that this paper introduces seems to be the idea to distill the results of planning algorithms run at training time into a global, goal-conditioned value function, which allows to reduce the required planning time at test time. The authors perform experiments on constructing visual plans for a simulated toy navigation task, a robotic rope manipulation task and the StreetLearn navigation task. The paper reports favorable results under a time-constrained test setting but does not include strong baselines that were designed for this setting.
## Strengths
- bootstrapping a learned local distance metric to a global distance metric to reduce test-time planning cost is an interesting problem
- the paper has nice visualizations / analysis on the toy dataset
- the learning procedure for the local distance metric is clearly described
- the paper uses a large variety of different visualizations to make concepts and results clearer
## Weaknesses
(1) missing links to related work: the author's treatment of related work does not address the connections to some relevant papers (e.g. [1-3]) or is only done in the appendix (especially for [4]). It is not clearly delineated between techniques and ideas that are introduced in other papers (see (2) below) and the novel parts of this work. This makes it hard to understand the actual contributions of this paper.
(2) only minor contribution: the core parts of this paper build heavily on prior work: time-contrastive objectives for distance learning have been introduced in [5] and also been used in a very similar setup as here in [4], further [4, 3] also use semi-parametric, graph-like representations for planning with a learned local distance metric. The major contribution seems to be to distill the plans derived with either (a) n-step greedy rollout or (b) Dijkstra graph-search into a value-function so that planning does not need to be performed at test time. This somewhat small contribution is in contrast to the claims from the introduction that this paper "pose[s] the problem of unsupervised learning a plannable representation as learning a cognitive map of the domain".
(3) comparison to weak baselines: the main comparison in the experimental section is to a version of [4] where the authors constrain the planning horizon to a single step, which means effectively greedily using the local metric from Sec. 3.1. To be clear: this is in no way the method of [4]: they use Dijkstra-based planning at test time and it is clear that a "version" of [4] that does not use planning is not able to work. To me this seems rather like an ablation of the proposed method than a real baseline. The baseline that plans greedily with embeddings based on visual similarity has even less hope of working. The paper lacks thorough comparison to (a) baselines with the same semi-parametric structure that perform planning at test time (like the real method of [4]) and (b) methods that generate reactive policies without constructing a semi-parametric memory (e.g. off-policy RL). Only then a thorough comparison of pros and cons of planning at training/test time is possible (see detailed suggestions below).
(4) lack of qualitative samples for generated plans: for both the rope and the StreetLearn domain the authors do not provide thorough evaluation. For the rope domain only a single qualitative rollout is shown, for the StreetLearn domain no qualitative samples are provided for either the proposed method or the comparisons. (see suggestions for further evaluation below)
(5) explanation of core algorithmic part unclear: the explanation of how the local metric is used to learn the global value function is somewhat unclear and the used notation is confusing. Key problems seem to be the double-introduction of symbols for the local metric in Alg. 2 and the confusing usage of the terms "global embedding" and "value function" (see detailed questions below)
(6) terms used / writing structure makes paper hard to follow: the connection between used concepts like "global embedding", "plannable representation" and "goal-conditioned value function" are not clear in the writing of the paper. The authors talk about concepts without introducing them clearly before (e.g. problems of RL are listed in the intro without any prior reference to RL).
(7) lacks detail for reproducing results: the paper does not provide sufficient detail for reproducing the results. Neither in the main paper nor in the appendix do the authors provide details regarding architecture and used hyperparameters. It is unclear what policy was used to collect the training data, it is unclear how the baselines are working in detail (e.g. how the 1-step planning works) and how produced plans are checked for their validity.
## Questions
(A) What policy is used to collect the training data on each environment?
(B) What is the relation between the "global embedding" \Phi and the "goal-conditioned value function" V_\Phi(x, x_prime) in Algorithm 2?
(C) What is the difference between the local metric function \phi and the reward function in Algorithm 2? Are they the same?
(D) If they are the same, how can the local metric accurately estimate rewards for states x and x_g that are far apart from one another as would naturally be the case when training the value function?
(E) What does the notation N(1, \eps) in line 5 of Algorithm 2 mean?
(F) What is the expectation over the length of trajectories between start and goal on the StreetLearn environment (to estimate what percentage of that the success horizon of 50 steps is)?
## Suggestions to improve the paper
(for 1) please add a more thorough treatment of the closest related works on semi-parametric memory + learned visual planning + learned distance functions (some mentioned below [1-5]) to the main part of the paper, clearly stating differences and carving out which parts are similar and where actual novelty lies.
(for 2) please explain clearly the added value of distilling the training-plans into a value function for O(1) test-time planning and point out that this is the main difference e.g. to [4] and therefore the main contribution of the paper.
(for 3) in order to better understand the trade-offs between doing planning at test time (like [3,4]) or learning an O(1) planner contrast runtime and performance of both options (i.e. compare to the proper method of [4]). This will help readers understand how much speed they gain from the proposed method vs how much performance they loose. It might also make sense to include an off-policy RL algorithm (e.g. SAC) that uses the local metric as reward function (without constructing the graph) to investigate how much planning via graph-search can help at training time. Another interesting direction can be to investigate the generalization performance to a new environment (e.g. new street maze, new rope setup) after training on a variety of environment configurations. [3] showed that explicit test-time planning performs better than "pure" RL, it would be interesting how the proposed "hybrid" approach performs.
(for 4) please add randomly sampled qualitative results for both environments and all methods to the appendix. It can additionally be helpful to add GIFs of executions to a website. It might also be interesting to add a quantitative evaluation for the plans from the rope environment as was performed in Kurutach et al. 2018.
(for 5) please incorporate answers to questions (B-E) into the text in Sec 3.2 explaining Algorithm 2. It might also help to structure the text in such a way as to follow the flow of the algorithm.
(for 6) restructure and shorten the introduction, clarify terms like "inductive prior within image generation" or "non-local concepts of distances and direction" or "conceptual reward" or "planning network", clarify how the authors connect the proposed representation learning objective and RL. Avoid sentences that are a highly compressed summary of the paper but for which the reader lacks background, like in the intro: "training a planning agent to master an imagined “reaching game” on a graph".
(for 7) add details for architecture and hyperparameters to the appendix, add details for how baselines are constructed to the appendix. add details about data collection and evaluation for all datasets to the appendix (e.g. how is checked that a plan is coherent in StreetLearn). It might also help to add an algorithm box for the test time procedure for the proposed method.
## Minor Edit Suggestions
- Fig 2 seems to define the blue square as the target, the text next to it describes the blue square as the agent, please make coherent
- for Fig 7: the numbers contained in the figure are not explained in the caption, especially the numbers below the images are cryptic, please explain or omit
[Novelty]: minor
[technical novelty]: minor
[Experimental Design]: Okay
[potential impact]: minor
################
[overall recommendation]: weakReject - The exposition of the problem and treatment of related work are not sufficient, the actual novelty of the proposed paper is low and the lack of comparison to strong baselines push this paper below the bar for acceptance.
[Confidence]: High
[1] Cognitive Planning and Mapping, Gupta et al., 2017
[2] Universal Planning Networks, Srinivas et al., 2018
[3] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning, Eysenbach et al., 2019
[4] Semi-Parametric Topological Memory for Navigation, Savinov et al., 2018
[5] Time-Contrastive Networks, Sermanet et al., 2017
### Post-rebuttal reply ###
I appreciate the author's reply, the experiments that were added during the rebuttal are definitely a good step forward. The authors added comparison to a model-free RL baseline as well as proper comparison to a multi-step planning version of SPTM. However, these comparisons were only performed on the most simple environment: the open room environment without any obstacle. These evaluations are not sufficient to prove the merit of the proposed method, especially given that it is sold as an alternative to planning methods. The method needs to be tested against fair baselines on more complicated environments; the current submission only contains baselines that *cannot* work on the more complicated tasks. I therefore don't see grounds to improve my rating.


We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.

This paper presents a reinforcement learning agent trained to interact with a painting environment in order to reproduce target images. The novel aspect of this work seems to be a version of the agent producing partial actions (i.e., portion of the action tuple is clamped to pre-specified values). This new agent receives the clamped components of the action as an additional conditioning input.
Pros:
+ In some cases, images produced by the system look appealing.
Cons:
- The writing of the manuscript could be significantly improved. I had a hard time understanding certain parts of the paper (e.g., what “constrained” means in the context of the present work). I got an impression that there was an effort to make things look more complex than they really are.
- The proposed model lacks novelty – there seems to be only one non-trivial contribution and I’m not entirely sure how useful it is. The authors never compare their system against a simple baseline when one just overrides the actions of the agent.
- In general, the evaluation section of the paper leaves a lot to be desired. The authors report some numbers (most of which are not even for the proposed model) but I’m struggling to make anything out of that information. Why should I care about them? What interesting conclusions can I draw from them? The paper never discusses this. On top of that, there are no baselines.
Notes/questions:
* At test time, the policy receives renders from a real environment. Could that create problems since it has only been trained on images synthesized by a neural surrogate?
* Abstract: “on many benchmarks” -> “on several benchmarks”
* Section 1: “we use a constraint representation along with …” – this sentence needs to be rewritten.
* Section 2: Missing reference – Neural Painters (Nakano, 19). Considers a neural surrogate of the libmypaint environment.
* Section 2, last paragraph:  “both methods can work on either a small action space or a small observation space” – how is the present model different? The action space is very similar (albeit continuous) to the existing approaches (i.e., relies on Bezier curves much like in (Ganin et al., 18)).
* Section 2, last paragraph: The last sentence needs to be rewritten (“uncontrollable agent” looks a bit strange)
* Section 3, first paragraph: “We highlight all …” -> “We describe all …”
* Section 4.1, first paragraph: The first sentence needs to be rewritten (“the corresponding canvas by the given action”).
* Section 4.1, first paragraph: “Unlike the previous …” – not true. (Ganin et al., 18) proposes to use this environment and (Nakano, 19) trains a neural surrogate for it.
* Section 4.4: “WGAN loss (Huang et al., 19)” – this WGAN loss was introduced in (Ganin et al., 18).
* Section 5: I feel like the paper could do a better job at explaining what “constraining” really means and justifying why it’s an interesting problem to solve. In my opinion, Eq. (3), for example, obscures rather than clarifies the notion of “constraints”.
* Section 5, paragraph 2: “For each different” -> “For each”
* Section 5.1, paragraph 4: “pi” -> “\pi”
* Figure 5: The caption almost overlaps with the text below.
* Section 6.3, paragraph 2: “l2” -> “l_2”
* Section 7, paragraph 2: “differing” -> “different”
I feel like the authors should perform a major re-writing of the manuscript before it’s ready for publication. Moreover, I failed to see any significantly novel aspects of the proposed system (maybe due to the poor presentation) and therefore I wouldn’t recommend the paper for acceptance.


Likelihood from a generative model is a natural statistic for detecting out-ofdistribution (OoD) samples. However, generative models have been shown to assign higher likelihood to OoD samples compared to ones from the training distribution, preventing simple threshold-based detection rules. We demonstrate that OoD detection fails even when using more sophisticated statistics based on the likelihoods of individual samples. To address these issues, we propose a new method that leverages batch normalization. We argue that batch normalization for generative models challenges the traditional i.i.d. data assumption and changes the corresponding maximum likelihood objective. Based on this insight, we propose to exploit in-batch dependencies for OoD detection. Empirical results suggest that this leads to more robust detection for high-dimensional images.

The paper makes the observation that likelihood models trained with batch norm assign much lower likelihoods to "training batches" of OoD data (batch norm statistics computed over over minibatch) than evaluation batches of OoD data (batch norm statistics over entire training set).
One issue with comparing this method to most other OoD detection works is that it considers OoD detection on *batches* of (all OoD data) or (all in-distribution data). As soon as the problem is changed to "classify between OoD batches" and not single samples, there are a large number of possible statistical tests one can perform to perform OoD (T-test between likelihoods of each batch) and the problem becomes *much* easier. In some ways, this makes things more well-defined (hard to compare distributions when one of them is just a single sample from an arbitrary distribution).
However, that brings me to a big concern I have with the evaluation protocol. The batch size used for train/evaluation is rather large (64, this detail is hidden in the Appendix and I would have appreciated the number put in the main experiments section). If you take a likelihood model and evaluate on 64 samples from SVHN, you are all but guaranteed to sample a sample with *exceedingly* low likelihood, which dominates the mean statistic, making it possible to separate SVHN batch from CIFAR10 batches. I suspect that OoD datasets have plenty of these "extremely low likelihood" examples that will drag the mean likelihood down a lot.
This is consistent with your batch normalization experiments: in training mode, the likelihood is computed from mean activations over a batch of OoD samples, several of which probably contribute to the low likelihoods. In evaluation mode, likelihoods for each OoD sample are evaluated independently, which results in a similar observation to prior work showing that CIFAR10 likelihoods are inaccurate for SVHN. In other words, I think there is a mistake made here: it is the phenomenon that *batch likelihoods*, not *batch norm*, that is responsible for this method working well. One experiment that is missing from your paper (and would prove my hypothesis wrong) would be if you adapted the OoD criteria to compare the *mean* likelihoods in the evaluation mode, and show that for OoD datasets, the difference between batches still remains small.
Nits:
- "...such as learning a mixture of Gaussians", I believe this toy example was on univariate gaussians, not mixtures.
- Choi et al. 2018 and should also be included in the citation that "CIFAR10 gives higher likelihood estimates to SVHN than CIFAR10 ones" (this was a concurrent discovery between the two papers)
- Choi et al. 2018 is not the right citation for "we evaluate the area under the ROC curve (AUC) and average precision (AP)" for each binary classification task, a more appropriate one would be Hendryks and Gimpel 2017.
- No doubt the authors realized already but Page 7 has some "related work still needs some work, but there...." which should be deleted.
- It took me awhile to find the batch size used in training and evaluation mode (64), which was on page 16.


Neural models achieved considerable improvement for many natural language processing tasks, but they offer little transparency, and interpretability comes at a cost. In some domains, automated predictions without justifications have limited applicability. Recently, progress has been made regarding single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal. In this context, a justification, or mask, consists of (long) word sequences from the input text, which suffice to make the prediction. Existing models cannot handle more than one aspect in one training and induce binary masks that might be ambiguous. In our work, we propose a neural model for predicting multi-aspect sentiments for reviews and generates a probabilistic multi-dimensional mask (one per aspect) simultaneously, in an unsupervised and multi-task learning manner. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable.

This paper proposes a neural model for multi-aspect sentiment of reviews. Apart from sentiment classification, the architecture is able to return a mask for each aspect highlighting the sequences of words used to make the sentiment prediction for that aspect. This improves the interpretability of the model. The aspects are fixed and the training set requires a multi-labeled dataset with one binary positive/negative label per aspect.
The network architecture is mostly standard: a text review is turned into a list of word embeddings which are fed through a trainable masker outputting one mask per aspect, essentially "selecting" a different set of words (embeddings) per aspect. From here on, a standard encoder can be used (the authors end up using a CNN with max-pooling but LSTM or other could be used), followed by a classification step (standard MLP) outputting a sentiment score per aspect. A couple tricks are used to make the masks converge to something meaningful by adding regularization terms to control the number of selected words and encouraging consecutive words to be selected.
I'm leaning toward rejection partly due to the limited novelty in the approach but mainly due to the experimental evaluation section which requires improvements.
- High aspect correlation baseline: the average aspect correlation across reviews in the training set is high (that is, if one aspect is positive, most aspects will be and viceversa). A natural baseline to include is the optimal "no-aspect" decision boundary: in other words, if for each review we were given the majority rating across aspects (is this review mostly positive or negative) and were able to predict this always correctly, what would the macro-aspect F1 score be if we were to simply predict that label to all aspect of that review?
- Please quantify how much does your model rely on this aspect correlation. For example, if you were to sort the reviews in the validation set by their aspect correlation, and chart the F1 score as a function of the prefix of this sorted dataset what would we see? This would be especially telling when comparing the different baselines and classifiers again each other in this chart.
- SVM: you mentioned not running it on the larger datasets due to lengthy training time. I would suggest including NB-SVM implemented with logistic regression rather than actual SVM since it's always a great baseline and fast to compute (in "Baselines and Bigrams: Simple, Good Sentiment and Topic Classification").
- The final model that ends up giving the best results is really a 2-step pipeline (rather than end-to-end trainable) and requires training twice: once to learn the masks and once to actually train the rest of the network after the word embeddings are extended with the masks.


Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks. The recent success of large pre-trained language models such as BERT and GPT-2 (Devlin et al., 2019;Radford et al., 2019) have suggested the effectiveness of incorporating language priors in down-stream NLP tasks. However, how much pre-trained language models can help dialog response generation is still under exploration. In this paper, we propose a simple, general, and effective framework: Alternating Roles Dialog Model (ARDM). ARDM models each speaker separately and takes advantage of the large pretrained language model. It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM outperforms or is on par with state-of-the-art methods on two popular task-oriented dialog datasets: CamRest676 and MultiWOZ. Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion. In persuasion tasks, ARDM is capable of generating human-like responses to persuade people to donate to a charity.

The paper is concerned with reducing the amount of manual dialog state or dialog act labeling in task-oriented dialog, following the lead of Eric and Manning (2017) who did not require any such explicit annotation. The authors note that Eric and Manning’s approach didn’t do well on more recent dialog datasets in comparison to e.g. Sequicity (Lei et al., 2018), which trains an explicit belief state. The first main contribution of the submission is to exploit GPT-2 to outperform Sequicitiy and other techniques, doing so without explicit dialog state/act supervision. The second contribution is an alternating parameterization of the model that distinguishes between agent and user utterances. Experimental results show that the method of the paper (ARDM) is either superior or on par with methods that exploit hand-labeled dialog states/acts.
Originality/impact:
I think the approach of the paper is well motivated, though quite simple. The idea of improving on a generation task with GPT-2 is a tried one, and its effectiveness isn’t really surprising. The alternating parameterization is interesting but has been done before in (e.g.) Zhao and Kawahara (2019) and (Zhang et al., 2019) [1]. Overall, I think the paper doesn’t bring much novelty, but results are promising, and this work represents a step towards making task-oriented dialog system less reliant and hand coded linguistic information. It could have an impact for the dialog community.
Empirical contribution:
My main concern with the paper is a lack of ablation, which makes it hard to understand where the improvement comes from. More specifically:
* It is not known how much the alternating nature of ARDM contributes the overall gains compared to (1) other details of the model; (2) pre-training with GPT-2. One way to ablate ARDM’s alternating mechanism from other characteristics of the model would be for example to tie the parameters of the user and the agent.
* Table 1 compares GPT-2 with ARMD, so we don’t know if the improvement comes from either fine tuning on CamRest676, from ARDM’s alternation between speakers, or from other characteristics of ARDM. Note: The text characterizes GPT-2 as “a pre-trained large-scale language model GPT-2”, so I presume “GPT-2” here means without fine-tuning.
* Results of Table 3 are perplexing and more analyzes would be needed to understand what is really happening. Considering that both TransferTransfo and ARDM are based on transformer and GPT-2, and are apparently fine-tuned on the same data, I find it strange the gap between the two is so big on perplexity (10.1 vs TransferTransfo’s 19.9) while TransferTransfo is actually superior on BLEU. I understand BLEU is problematic for dialog evaluation as it treats references (gold responses) as the only reasonable responses (Liu et al., 2016), but that makes perplexity as an evaluation metric problematic for exactly the same reason. In fact, (Schwenk et al., 2006)[2], (Luong et al., 2016)[3], and others have shown that BLEU and perplexity are highly correlated, which isn’t surprising as both penalize generated responses to the extent that they lexically differ from the gold standard. So, these results look a bit strange/suspicious to me, and I think deserve further investigation. Sample outputs of both TransferTransfo and ARDM might help better understand the problem (Why showing only ARDM’s? When showing generation examples, I think it is standard practice to show outputs of multiple systems, and not just for the system of the paper).
Minor comments:
- [Manning and Eric, 2017] should be [Eric and Manning, 2017]
- ARDM with “recurrent” maybe isn’t a good name as it may imply it is RNN instead of Transformer based.
- Section 3.1: I suggest you explain the “memory mechanism” more formally and in more details, e.g., M_{t-1} doesn’t seem to be used anywhere.
- “assume there is only one layer in Transformer”: Is it for the sake of the presentation only, I assume?
- Tables 1-3: perhaps add confidence intervals as some of the differences are quite small, and so are some of the test sets.
- Table 3 doesn’t seem to be referenced anywhere.
- Table 4: I would recommend showing ARDM and TransferTransfo side by side.
[1]: https://arxiv.org/abs/1903.05759
[2]: https://www.aclweb.org/anthology/P06-2093/ (e.g., Fig. 5 shows the high correlation between BLEU and perplexity)
[3]: https://arxiv.org/abs/1410.8206
Questions:
(1) In 4.3.1, are capacities of the two models the same? If ARDM has (roughly) double the number of parameters due to its modeling of two speakers, what about comparing TransferTransfo at same model capacity to make the comparison more meaningful?
(2) I am confused how you (the authors) compute belief state success F1 for GPT-2, as the text says there is only pre-training with this model, which I think implies there is no fine-tuning on the belief state identification task. Could you provide more details?


To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data distribution on workers is non-identical, Local SGD requires O(T 3 4 N 3 4 ) communications to maintain its linear iteration speedup property, where T is the total number of iterations and N is the number of workers. In this paper, we propose Variance Reduced Local SGD (VRL-SGD) to further reduce the communication complexity. Benefiting from eliminating the dependency on the gradient variance among workers, we theoretically prove that VRL-SGD achieves a linear iteration speedup with a lower communication complexity O(T 1 2 N 3 2 ) even if workers access non-identical datasets. We conduct experiments on three machine learning tasks, and the experimental results demonstrate that VRL-SGD performs impressively better than Local SGD when the data among workers are quite diverse.

This paper tackles the problem of data-parallel (synchronous) distributed SGD, to optimize the (finite) sum of N non-convex, possibly different (in the so-called non-identical case), loss functions. This paper focuses on improving the communication efficiency compared to several existing methods tackling this problem.
To that end, the authors contribute:
· A novel algorithm and its asymptotic communication complexity.
· The proof that the common metric of the sum over the training steps of the expected squared norm of the gradient at the average of the N parameters is bounded above.
· Experimental results (training loss function of epoch number) comparing this algorithm with 2 existing ones, solving 3 problems under reasonable settings.
  The training time per epoch of VRL-SGD is claimed to be identical to the one of Local SGD, as the algorithm only have minor differences.
- strengths of the paper:
· The main paper is very easy to follow.
· Good effort to give intuitions on why VLR-SGD can improve the convergence rate of existing algorithms.
  Such an effort is to be highlighted.
· No obvious mistake in the main.   I have not thoroughly checked the full proof though.
-  weaknesses of the paper:
· The algorithm, while having differences, is quite reminiscent of Elastic Averaging SGD (EASGD) [1].
  Indeed in both algorithms the model update at the workers consists in both descending the local gradient plus descending toward some "moving-average"obtained through averaging all the local models.
  In EASGD, this "moving-average" is common to every worker and the master, which updates it every k steps.
  In this paper, each worker has its own "moving-average", which update computations are different than in EASGD as the use the instant average of the workers' models instead of the previous "moving-average".
[1]Sixin Zhang, Anna Choromanska, Yann LeCun, Deep learning with Elastic Averaging SGD,  NeurIPS, 2015
- Questions I would like the authors to respond to during the rebuttal:
· Could Elastic Averaging SGD (in particular their fastest variant EAMSGD) be applied as-is to solve the non-identical, non-convex optimization problem at hand?
  Despite the authors of EASGD not studying their algorithm in the non-identical case, following what is done in the intuition part of VRL-SGD (in particular Equation (8)), it seems that the update rule of the "moving-average" in EASGD is then equivalent to having a momentumSGD with dampening (instead of the "generalized SGD form" obtained with the approach of VRL-SGD).  Hence my question.
I suggest acceptance. However I'm willing to change my opinion after reading other more qualified reviewers in the sub-area of variance-reduction techniques.
note: If EASGD was to be sound in the non-identical case as well, my decision would not change much.


Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties. First, we study the role of normalization. Batch Normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This twodomain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness. Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.

This paper reveals some interesting properties of neural networks when trained adversarially at ImageNet scale. The total cost of the experiments is quite impressive, therefore the results are valuable references. With extensive experiments, the authors reveals two intriguing properties of neural networks when trained adversarially, and devotes most of the paper to studying the first one, i.e., why networks with Batch Normalization cannot achieve high robustness when both clean and adversarial images are used for training. I am not fully convinced by this point, but I do think the second point is very interesting. Different from previous work arguing that more data is needed for improving adversarially robust generalization, this paper shows that adversarial robustness can be improved consistently just by making ResNet deeper.
I tend to accept this paper for the valuable results and the discovery of the positive correlation between network capacity and adversarial robustness, but I am not fully convinced by the explanations for the problems with BN. I hope the authors could address my concerns before I can be more confident about my decision.
To me, it seems the correct way to do adversarial training is to only use adversarial samples, since we are trying to minimize the maximum risk inside a norm ball around the clean sample (Madry et al. 2018). All the experiments with clean images in the objective are consistently worse than training only on adversarial samples under the same setting in this paper. It therefore becomes not that important to study the effect of Batch Normalization on training with both clean and adversarial images.
Also, I am a little bit unconvinced that the running mean and variance of BN is the cause of bad performance for the mixed training. First, I want to know the standard and adversarial accuracies of the network with GN in the "100% adv + 0% clean" setting. It seems missing in the paper. Despite being much better than BN in the "100% adv + 100% clean" setting, it is not sure whether it is caused by the improvement in some other property (e.g., capacity) from GN. If the robust accuracy with "100% adv + 0% clean" is also much higher with GN than BN (seems unlikely though), then replacing GN with BN does not solve the problem and it is still the objective to blame.
I am also not fully convinced by the experiments with separated BN parameters for clean and adversarial samples. By doing so, the network is actually trained to approximate its behavior in the "100% adv + 0% clean" setting. Since the adversary is maximizing the loss, the gradient (of the conv layers) from adversarial samples will dominate the total gradient in the "MBN 100% adv + 100% clean" setting, and the "MBN_{adv}" network will be trained similarly as the network trained in the "100% adv + 0% clean" setting. This explains why the adversarial accuracy can be very close to "100% adv + 0% clean" but still lower.  Comparing results (under different ratios of clean samples) for networks without BN is very important and much more effective at explaining the phenomenon than trying to make BN work.
It would also be great if the authors could provide some curve showing the tendency of the running variance of BN. Sec 4.3 does make lots of sense from a practical aspect, i.e., fixing mean and var for training in the last 10 epochs could improve the results using same number of epochs, but what if we just train more epochs? Will the variance converge in just tens of epochs?
Finally, though it seems that deeper networks are more robust,  the robustness might be a misconception caused by gradient vanishing. Could the authors provide the average gradient norms on the correctly-classified images (remaining correct after attack) in the first step of PGD attacks for the models in Figure 7? If deeper networks indeed have much smaller gradient norm, could the authors try scaling the loss by some factor to make the attacks stronger?
===========================================================
I am not fully convinced by your explanations. Could you give the results for the gradients, or use the loss function from CW attack + PGD to report the robustness of the deeper models? I am not sure which type of residual connection you are using, but if you are using the "original" version as mentioned in this paper [1], then it does not necessarily avoid gradient vanishing, since if ReLU is deactivated, the gradient will be brought to zero. Another possible cause for the possible misconception of robustness might be caused by the saturated cross entropy loss, which could also give 0 gradients but could be verified by switching to the margin loss as in the CW attack. I have encountered such cases when I achieved non-zero robust accuracy (evaluated with PGD on cross entropy loss, and the robust accuracy is similar to your improvement from ResNet152 to ResNet638) on a naturally trained network as I increased the number of layers, but when I switched to CW loss (or margin loss), the robust accuracy goes to zero. Even for resnet, when you multiply the number of layers by ~4 as done in the paper, such phenomenon will actually happen.
[1] He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.
=====================================================================
Raising my score. Found a recent good paper based on the findings of this paper.


Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, Liu et al. (2018) proposed an approach that avoids the curse of horizon suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data be drawn from the stationary distribution of a known behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a certain operator, and develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected. We analyze its asymptotic consistency and finite-sample generalization. Experiments on benchmarks verify the effectiveness of our approach.

Summary :
This paper proposes an approach for long horizon off-policy evaluation, for the case where the behaviour policy may not be known. The main idea of the paper is to propose an estimator for off-policy evaluation, that computes the ratio of the stationary state distributions (similar to recent previous work Liu et al.,) but when the behaviour policy may be unknown. This is an important contribution since in most practical off-policy settings, the behaviour policy used to collect the data may not necessarily be known.
Comments and Questions :
	- This is an important and novel contribution where the authors propose an approach for OPE that comes down to solving a fixed point operator. This is simlar to solving for fixed point Bellman operators, but where their problem can be formulated as a backward recursive problem.
	- The operator defined as in equation 3 is in terms of the backward view of time - this allows the operator to capture the visitation from the previous s,a to the current s. This is the backward flow operator with which a fixed point equation can be described. Although the authors note that similar operators have appeared in the literature before - their main contribution is in terms of using such operators for the OPE problem, which seems novel to me and is an interesting approach with potential benefits as demonstrated in this paper.
	- The core idea comes from equation 9 which tries to minimize the discrepancy between the empirical distribution and the stationary distribution. This can be formulated as an optimization problem, and the paper uses blackbox estimators, as described in section 4.2 for solving this problem.
	- The next interesting part of the paper comes from solving the optimization problem in equation 9 with Maximum Mean Discrepancy (MMD) - this is a popular approach that has recently become well-known, and the authors make use of it minimize the differences between the empirical and the stationary state distribution.
	- Section 4.2 appears a bit confusiing to me with some details missing - it would perhaps be useful if the authors could include more details for 4.2, especially with some explanations of how they arrived at the expression in equation 10. This would also make the paper more self-contained, for readers in the RL community perhaps not so well-read with literature on MMD.  Appendix C contains the detailed derivation, but more intuitive explanations might be useful for the context of the paper.
	- The proposed black box estimator seems quite useful as demonstrated in figures 2 and 3. Although the authors evaluate their approach of few simple domains - it would have been useful if there were more extensive experiments performed for OPE problems. This would be useful since from fig 2, it appears that the proposed method only outperforms in 3 out of 4 evaluated problems.
	- For experiments, it would also be useful to demonstrate the significance of not knowing the behaviour policy and what are the usefulness of it. The paper is motived in terms of unknown behaviour policies that generated the data - so few experiments that explicitly shows the benefit of it would perhaps strengthen the paper more.
	- I am curious to know more about the bias-variance trade-off of the proposed OPE estimator as well. Ie, does the proposed method introduce any bias, or has significance in terms of lower variance for the long horizon problem? Experimentally, would it be possible to demosntrate whether the approach has lower variance compared to existing baselines?
Score :
- Overall, I think the paper has useful contributions. It is a well written paper, but some additonal details in section 4.2 might be useful, especially on the appriach with MMD. Experimentally, I think there are some experiments missing and doing those can significantly strengthen the paper as well. The proposed method seems simple and elegant, and I would recommend for a weak acceptance of the paper.


Automatic neural architecture search techniques are becoming increasingly important in machine learning area. Especially, weight sharing methods have shown remarkable potentials on searching good network architectures with few computational resources. However, existing weight sharing methods mainly suffer limitations on searching strategies: these methods either uniformly train all network paths to convergence which introduces conflicts between branches and wastes a large amount of computation on unpromising candidates, or selectively train branches with different frequency which leads to unfair evaluation and comparison among paths. To address these issues, we propose a novel neural architecture search method with balanced training strategy to ensure fair comparisons and a selective drop mechanism to reduce conflicts among candidate paths. The experimental results show that our proposed method can achieve a leading performance of 79.0% on ImageNet under mobile settings, which outperforms other state-of-the-art methods in both accuracy and efficiency.

This paper introduces a better searching strategy in the context of automatic neural architecture search (NAS). Especially, they focus on improving the search strategy for previously proposed computationally effective weight sharing methods for NAS. Current search strategies for the weight sharing NAS methods either focus on uniformly training all the network paths or selectively train different network paths with different frequency, where both have their own issues like wasting resources for unpromising candidates and unfair comparison among network paths. To this end, this paper proposes a balanced training strategy with “selective drop mechanism”. Further, they validate their approach by showing leading performance on ImageNet under mobile settings.
Overall, I appreciate the effort on exploring new strategies for better search algorithm for NAS in the context of weight sharing methods. However, the analysis of the current approach is limited and some relevant papers are missing. More details below.
Arguments:
1) This paper’s main focus is on developing a better search strategy for NAS based weight sharing methods. The validation of their approach is supported via showing improvement in the accuracy on ImageNet task. However, my main argument is that you have to validate the proposed approach with better analysis on the search space along with improvement on the tasks. For example, “Evaluating the Search Phase of Neural Architecture Search” paper which studies the effectiveness of current search strategies is not referenced in this paper. I would suggest to do analysis of you search strategy by following the analysis experiments conducted in the above paper.
2) The related work doesn’t discuss the latest papers which suggest why and in what ways the current search strategies are based for weight-sharing based NAS approaches. As this paper is trying to address this problem, I would naturally assume that this is well discussed, but it’s missing!


We address the task of unsupervised retargeting of human actions from one video to another. We consider the challenging setting where only a few frames of the target is available. The core of our approach is a conditional generative model that can transcode input skeletal poses (automatically extracted with an off-the-shelf pose estimator) to output target frames. However, it is challenging to build a universal transcoder because humans can appear wildly different due to clothing and background scene geometry. Instead, we learn to adapt -or personalize -a universal generator to the particular human and background in the target. To do so, we make use of meta-learning to discover effective strategies for on-the-fly personalization. One significant benefit of meta-learning is that the personalized transcoder naturally enforces temporal coherence across its generated frames; all frames contain consistent clothing and background geometry of the target. We experiment on in-the-wild internet videos and images and show our approach improves over widely-used baselines for the task.

This paper proposes a novel and interesting task that learn to retarget human actions with few-shot samples. The overall pipeline is built by applying  meta-learning strategy on pre-trained retargeting module. It follows a conditional generator and discriminator structure that leverages few-shot frames to retarget the action of source video. The approach is technically sound.
The evaluations are compared with two baseline methods,  Pix2PixHD and Posewarp. Their evaluations are satisfactory and convincing, the results demonstrate some improvements over baseline models regarding both the selected metrics and the visualizing performance.
Though the proposed problem is novel and somewhat interesting, there are also several weaknesses of this work:
- The novelty of methodology is somewhat limited. It is more about merging several state-of-the-art modules in different tasks to tackle the few-shot retargeting problem. Though efforts may be needed to make the pipeline work, the overall contribution is not significant.
- The improvement obtained with proposed MetaPix module is not significant in few-shot setting according to Table 1. Additionally, could the authors provide some visualizing results for different K number, which would be interesting for analysis.


Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous fashion, where delay occurs, can make training even more scalable. However, asynchronous training has its pitfalls, mainly a degradation in generalization, even after convergence of the algorithm. This gap remains not well understood, as theoretical analysis so far mainly focused on the convergence rate of asynchronous methods. Contributions: We examine asynchronous training from the perspective of dynamical stability. We find that the degree of delay interacts with the learning rate, to change the set of minima accessible by an asynchronous stochastic gradient descent algorithm. We derive closed-form rules on how the learning rate could be changed, while keeping the accessible set the same. Specifically, for high delay values, we find that the learning rate should be kept inversely proportional to the delay. We then extend this analysis to include momentum. We find momentum should be either turned off, or modified to improve training stability. We provide empirical experiments to validate our theoretical findings.

The authors model A-SGD as a dynamical system, where parameters are updated with delayed gradients. The authors analyze the stability of this system, and they first derive that the learning rate must scale linearly with the inverse of the delay around a minimum to remain stable. Using a similar analysis they show that the standard way of incorporating momentum into A-SGD requires small learning rates for high momentum values, and they propose "shifted momentum," which allows for stability under higher momentum values. Experimentally, the authors show that around minima the learning rate needed to retain stability scales linearly with the inverse of the delay, that there appears to be an analogous threshold when training models from scratch, that shifted momentum allows for higher momentum values, and finally that on several datasets A-SGD with an appropriate learning rate is able to generalize at least as well as large batch synchronous training.
This is a nice paper with a large number of interesting theoretical and experimental results, and I believe it should be accepted. I think there are some largely presentational issues that should be addressed, however:
- I think the authors should attempt to make a stronger case for the practical implications of their analysis: in particular, in the most practical setting (where we don't have a minimum obtained from synchronous training), what does the provided analysis allow us to do? Part of this might involve being more explicit about the results in Table 1: what exactly was the procedure for selecting the learning rates? Is it meaningfully different than just lowering the learning rate?
- Equation (3) is rather obscure without the Appendix, especially since unbolded x hasn't been introduced anywhere. I think the authors should try to convey more of what's going on in this equation in the main text.
Minor: 'looses' should be 'loses' throughout, and it might be good to include a conclusion section.


We propose a solution for evaluation of mathematical expression. However, instead of designing a single end-to-end model we propose a Lego bricks style architecture. In this architecture instead of training a complex end-to-end neural network, many small networks can be trained independently each accomplishing one specific operation and acting a single lego brick. More difficult or complex task can then be solved using a combination of these smaller network. In this work we first identify 8 fundamental operations that are commonly used to solve arithmetic operations (such as 1 digit multiplication, addition, subtraction, sign calculator etc). These fundamental operations are then learned using simple feed forward neural networks. We then shows that different operations can be designed simply by reusing these smaller networks. As an example we reuse these smaller networks to develop larger and a more complex network to solve n-digit multiplication and n-digit division. This bottom-up strategy not only introduces reusability, we also show that it allows to generalize for computations involving n-digits and we show results for up to 7 digit numbers. Unlike existing methods, our solution also generalizes for both positive as well as negative numbers.

The paper proposes a method to design a NN based mathematical expression evaluation engine. I find that the paper could benefit a lot from some rewriting as it is not very clear and over claiming at points.
The introduction states that almost all ANNs lack generalization, this is in my opinion an overstatement. Domain shift and adaptation are techniques to cope with situations where the test data distribution is not coherent with the training data distribution. If this would be true in general we would have not seen such a resurgence and widespread use of ANN in the past years.
The paper lacks also proper citations to previous work and I find the background section and motivation rather weak.
The fundamental operations presented in section 3 do not involve any learning at all, contrary to the referenced work of Trask et al where parameters are actually learned (see relaxation of sign function with tanh etc.). I therefore find the use of ANN as basic constituent of the block to be wrong, each network has fixed hand-crafted weights. If I were to replace ANN with the ordinary corresponding function nothing would in the presented framework.
Multiplication and division as explained in the algorithms do not require learning at all. I am afraid the ML contribution of this work is in my opinion almost non existent. I see every component as being scripted rather than learned from the data, which would be of course much more interesting.
Experiments are not clear at all, setup and explanation of results are not sufficient and I find them not thoroughly executed. Table 1 mentioned classification but the task is never clearly explained. Experiment 2 compares to NALU but in the proposed work nothing is learned, unless I misunderstood the work entirely.


Fine-tuning with pre-trained models has achieved exceptional results for many language tasks. In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse language-understanding benchmarks. However, in many downstream tasks, information between layers is ignored by BERT for fine-tuning. In addition, although self-attention networks are well-known for their ability to capture global dependencies, room for improvement remains in terms of emphasizing the importance of local contexts. In light of these advantages and disadvantages, this paper proposes SesameBERT, a generalized fine-tuning method that (1) enables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring. Furthermore, we demonstrated the effectiveness of our approach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations. The experiments revealed that SesameBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.

Summary:
The paper proposes adding two mechanisms to the BERT architecture for NLU. The first is based on integrating information from all layers of the encoder via a method called Squeeze and Excitation. The second uses Gaussian blurring to encourage information sharing among neighboring words. The proposed method improves modestly on BERT on the GLUE suite of problems. It also substantially improves on BERT with respect to a class of examples that are designed to confound models that learn superficial heuristics based on word occurrence.
I learn toward rejecting this paper. The method shows some performance gains over BERT on some GLUE tasks, but these are fairly small for the most part, and BERT outperforms the proposed method by a similar amount on a similar number of tasks. The strongest result is the HANS "lexical_overlap" case, where the proposed method has a clear advantage. I have no experience with these kinds of NLU models, so I can't say with confidence whether the architectural additions proposed are well-motivated, but to me it feels like there is not a strong justification for adding these particular features to the BERT architecture, and the results do not clearly demonstrate their utility except in the "lexical_overlap" case.
Details / Questions:
* It seems to me that the GLUE results might be within the margin of error. Is it feasible to replicate training with different random seeds to see what the variance in the performance numbers might be? I suspect that a statistical analysis [1] might conclude that BERT and the proposed method are indistinguishable on the GLUE suite.
* Were the proposed architectural additions conceived with the HANS "counterexamples" in mind (i.e. is there a specific reason to think that these types of methods would avoid the "superficial" reasoning that these examples are supposed to reveal)? Were other methods of adding context considered?
* I suggest using the same x-axis scale on the two charts in Figure 3 to avoid confusion about the magnitudes of the differences.
References:
[1] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1-30.


Accompanying with the flourish development in various fields, deep neural networks, however, are still facing with the plight of high computational costs and storage. One way to compress these heavy models is knowledge transfer (KT), in which a light student network is trained through absorbing the knowledge from a powerful teacher network. In this paper, we propose a novel knowledge transfer method which employs a Student-Teacher Collaboration (STC) network during the knowledge transfer process. This is done by connecting the front part of the student network to the back part of the teacher network as the STC network. The back part of the teacher network takes the intermediate representation from the front part of the student network as input to make the prediction. The difference between the prediction from the collaboration network and the output tensor from the teacher network is taken into account of the loss during the train process. Through back propagation, the teacher network provides guidance to the student network in a gradient signal manner. In this way, our method takes advantage of the knowledge from the entire teacher network, who instructs the student network in learning process. Through plentiful experiments, it is proved that our STC method outperforms other KT methods with conventional strategy.

The paper suggests a new method for knowledge transfer from teacher neural network to student: student-teacher collaboration (STC). The idea is that teacher not only provides student with the correct answer or correct intermediate representation, but also instructs student network on how to get the right answer. The paper suggests to merge the front part of the student network and back part of the teacher network into the collaboration network. In this way, the weights in the student subnetwork are learned from the loss backpropagated from the teacher subnetwork. The target labels for collaboration network are outputs of teacher network. The method is adapted for different tasks, classification and bounding box regression are presented in the experiments. It outperforms previous methods on various datasets. Furthermore, the method shows good results when integrated with traditional knowledge distillation.
Overall, the paper is a significant algorithmic contribution. The related work section provides thorough review of different methods to decrease computational costs, including not only knowledge distillation, but also pruning, compressing and decomposition approaches. The idea is elegant and, to the best of my knowledge, has never been suggested in other works. Considering the theoretical part, it is clearly shown how the gradient signal from the teacher sub-network guides the student network on which part of the weights should be paid attention on. All derivations are easy to follow. The paper also considers how the suggested idea is aimed to solve the problems of previous knowledge transfer methods. The experimental section is consistent and clearly shows the advantage of the suggested method. Teacher and student networks used are different sizes of ResNet, Wide ResNet and VGG. The paper presents classification experiments on CIFAR-10, CIFAR-100, ImageNet and object detection experiment on PASCAL VOC 2007 dataset. STC outperforms previous methods, both with KD integration and without. The performance is always better than pure student training (which was not always the case for previous methods) and sometimes the results are even better than teacher performance. Finally, the choice of teacher output as target over soft target and ground truth, which was previously motivated in the theoretical section, is shown to be superior in the experiment.
Possible improvement of the paper is the instruction on how to choose the intermediate layer from where to teach the representation, i.e. where the student sub-network ends and teacher sub-network begins. For object detection experiment the choice of the border is naturally defined by the architecture of the network in Faster-RCNN approach. Could the choice be different? May be somewhere inside the BackBone part of the networks? For classification, it could be interesting to study how this choice influences the results. However, this question didn’t affect my score of the paper, and, as far as I know, it is also not considered in the previous works on knowledge distillation.
Minor comments
1.	In the context of accelerating the models using decomposition, Lebedev et al., ICLR 2015 could be cited.
2.	Page 2: difference tasks -> different tasks
3.	Page 2 the first bullet point: additionally utilizes -> additionally utilizing/which additionally utilizes
4.	Page 2 the third bullet point: brings good generalizability -> which brings good generalizability
5.	Page 5: “training strategy is more accelerate than” – odd phrase
6.	Page 6: while KT has conflicts with KD in some cases -> while FT has conflicts with KD in some cases.


Cloud computing is becoming increasingly popular as a platform for distributed training of deep neural networks. Synchronous stochastic gradient descent (SSGD) suffers from substantial slowdowns due to stragglers if the environment is nondedicated, as is common in cloud computing. Asynchronous SGD (ASGD) methods are immune to these slowdowns but are scarcely used due to gradient staleness, which encumbers the convergence process. Recent techniques have had limited success mitigating the gradient staleness when scaling up to many workers (computing nodes). In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers. Our evaluation on the CIFAR, ImageNet, and WikiText-103 datasets shows that GA outperforms the currently acceptable gradient penalization method, in final test accuracy. We also provide convergence rate proof for GA. Despite prior beliefs, we show that if GA is applied, momentum becomes beneficial in asynchronous environments, even when the number of workers scales up.

The paper introduces a new variant of asynchronous SGD, GA-ASGD, for distributed training. The goal is to mitigate the gradient staleness issue caused by asynchronously applying gradients to an old version of parameters. Prior work addresses this issue by penalizing the learning step of a worker linearly to its missed updates since getting a replica of parameters. However, that approach does not consider the differences between the old and new versions of parameters, which can cause over-penalization and under-penalization. The main contribution of this paper is to introduce a new way of measuring weight staleness and to explore the idea of penalizing the gradient itself to mitigate the staleness issue. The paper does a good job of discussing how GA can be applied to existing optimizers such as Adam. It performs empirical studies on ImageNet and Transformers-XL to conclude that GA-ASGD outperforms ASGD and the prior staleness aware approach. It also demonstrates the scalability of GA by scaling up to training with 128 asynchronous workers.
Strengths:
+ Introduced a novel approach to measure the parameter staleness, which helps penalize the gradients instead of the learning step.
+ The approach seems to be easily applicable without much additional hyperparameter tuning.
+ Evaluation on both image and NLP tasks demonstrate that GA leads to improvement over prior staleness aware approaches.
Weaknesses:
- Despite great improvements over prior ASGD based approaches, there is still a quite noticeable accuracy degradation compared to the SGD baseline.
- The evaluation is done with simulation, and no report of the number of training steps to converge and end-to-end training time, making it difficult to compare the efficiency with SGD based approaches.
Overall, I think this is good work. The idea of defining weights stableness as the minimal number of updates required to traverse the current distance between the master and worker weights seems to be reasonable. The comparison to prior ASGD based approaches are extensive, and the improvements seem decent. The fact that it does not require much or additional hyperparameter tuning also seems neat.
My major concern comes from results compared to SGD and motivation. The fact that almost all ASGD based workloads, even with the help of GA, cannot get on-par accuracy as the SGD baseline bothers me. In particular, the accuracy drops considerably as the number of asynchronous workers grows (Fig 1. and Fig. 2, and Table 3). For example,  the gap between the accuracy of GA vs. SGD can be as large as 3.46% when there are 128 workers. That gap might be closed with additional hyperparameter tuning, but it is unclear from the current draft and results. In the Transformer-XL example, GA has to limit the number of asynchronous workers to 4 in order to get close to baseline accuracy. It would have been better to show the trade-off between accuracy and performance in comparison with SGD.
On the other hand, the recent advance of SGD using large-batch training achieves great results on large model training such as BERT [1]. It helps to improve the compute/communication ratio, which seems to mitigate the straggler issue. Given that today's cloud service largely uses homogenous accelerators (TPU/GPU of the same SKU), it is less clear whether it is really beneficial to train with ASGD despite the improvements from GA.
Question:
One question is about how the batch size should be changed as the number of workers increases. The convergence analysis indicates that by increasing the batch size, the convergence speed of GA-ASGD will decrease. Does that mean the batch size should remain relatively unchanged to avoid negatively impacting the end-to-end training?
Is there any benefit to applying Gap to model parallelism paradigm such as pipeline parallelism?
[1] "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes", by You et. al.