Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data. Applications of neural networks often consider learning in the context of a single task. However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks. Such multi-task learning settings have the potential to improve data efficiency and generalization by sharing data and representations across tasks. However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is difficult to learn a single model that can solve all the tasks while realizing data efficiency and performance benefits. Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide. In this work, we develop an approach that endows a single model with the ability to represent both extremes: joint training and independent training. To this end, we introduce matrix-interleaving (Mint), a modification to standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix. By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks. On three challenging multi-task reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both.

The paper propose a single-network approach to multi-task learning by adding a task-specific linear transformation layer after each fully connected layer. The authors prove that the addition of such a layer keeps the expressive power of the network for each task. They also discuss how the linear transformation (parameterized by a transformation matrix and a bias vector) can be represented in a discrete manner in usual multi-task supervised learning and in a continuous manner (by two other neural networks) in goal-conditioned reinforcement learning. Experiments demonstrate the superiority of the proposed single-network approach.
The proposed approach is single and elegant. It is recommended to weak-reject the paper because of the following key reasons.
(1) Problem formulation is far from clear, perhaps because of the lack of clarity in writing. In particular, the super short Section 2 did not clearly illustrate what's being trained and what's being tested, and whether we care about the generalization performance of each task, or the generalization performance to new tasks (generated from P(T)). Some notations are confusing there---for instance, i seems to be indicating tasks, but then there is a z_k as task indicator. Even for the main proposed approach in Section 3.1, the notations are loosely used in nature. For instance, it is hard to understand what the authors mean by "train separate neural networks to output the Mint matrices and biases"---there is no information about the "training data" for learning those neural networks.
(2) Theoretical justification is at best shallow, or at least in the context that the authors have put it. While having an universal expressive power is good, it is easily achieved by adding an indicator variable (z_k) per layer (similar to task-specific-all-fc in the experiments). So the guarantee does not seem to be closely related to explaining the proposed approach (though the guarantee is nice to have). The authors contrast the guarantee with what FiLM (a competitor approach) can do, but in the experiments FiLM is not taken as a competitor in multi-task supervised learning, leaving a big gap between theory and practice. In the flow presented by the authors, it is strongly suggested to introduce FiLM in more detail and compare it with the proposed approach more clearly in design, theory and experiments.
(3) It is hard to understand whether the experiments are reasonably designed. In particular, the two settings take different sets of competitors, and there is little information on why those competitors are selected, whether they represent state-of-the-art, etc.. The authors highlight that the proposed approach uses much fewer parameters but other than that it is hard to infer why the proposed approach is better. Is it better because there is more overfitting for the competitor's approaches given more parameters? Is it better because it is easier to tune? The task-specific-all-fc (which is of similar # parameters to the proposed approach) result particularly looks suspicious to me but there is no other information to double-check on why the proposed approach is better. In particular, I believe the authors have *not* answered their first proposed question "does our method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there is little overlap" properly---their best evidence may have been MT10 and MT50 experiments, but even in those experiments, I am not sure whether the authors want to take the results as suggesting there are "substantial overlap" or "little overlap."
Some other suggestions:
(4) It is suggested to analyze the matrices learned by the proposed approach. Do the matrices contain reasonable task correlations (i.e. for two similar tasks, are the matrices somewhat similar) to understand more about the proposed approach.
(5) It looks a bit strange to me that there is no discussion on regularizing the linear transformation matrices, as it seems possible to embed the task relations through the regularization. Have the authors considered the possibility?
(6) The authors are overly-emphasizing what they want to do (interpolating between independent networks and shared network). This occupies multiple redundant paragraphs in the early sections. It is better to remove some of those and use the space for more solid results, such as clarifying the notations.
(7) One baseline that could have been considered is to just train a fully-shared network (without z_k), and a fully-independent one. Then, use validation to select the better network and compare with the proposed approach.


We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.

After reading authors' response, I am sticking to my original decision. Authors addressed most of the issues I raised and I am happy with their response; however, I still believe the paper should not be accepted since it is not adding enough value. The problem is important and impactful. However, the algorithmic idea comes from LML (Amos 2019), and the impact on the real problems has not been demonstrated. Hence, it is adding no value algorithmically, and adding a very small value from application perspective. It is basically saying LML can be trivially applied to differentiate through CEM, and it works on some simple toy problems. To me this is mostly a sanity check. Hence, I am sticking to my weak-reject decision.
-------
The manuscript is proposing a method to make cross-entropy method (CEM) differentiable. CEM is a widely used zeroth-order optimization method. The main idea in the paper is applying the recently proposed limited multi-label projection (LML) layer in a straight-forward manner to the CEM since the major computational tool in CEM iteration is top-k selection. The authors apply the proposed method to synthetic energy-based learning and continuous control problems.
The proposed method is definitely impactful. Considering the fact that CEM is a powerful and widely used tool, I believe the work will lead to many interesting follow-ups. In addition to these, the work is addressing computational scalability of model-based RL which is both under-explored and important problem.
The proposed model is novel from a modelling perspective since it makes CEM part of end-to-end learnable models. Whereas, it has no algorithmic novelty since it is a straightforward application of the LML layer to the CEM problem. Lack of algorithmic novelty is not an issue but the authors should at least discuss similarities to LML (Amos 2019) in a clear manner in related work. Not including it in the related work is somewhat surprising to me.
The exposition can clearly be improved. First of all, Proposition 1 is an existing result, hence authors should give a proper citation in its definition. Second of all, Proposition 3 does not include anything about asymptotic (tau -> 0) whereas the stated one-line proof is using asymptotic arguments. Finally, there are other minor issues like Lemma1 not having a proof, proposition 1 has no statement about its proof etc.  The manuscript would significantly benefit from a thorough proof reading for mathematical completeness and correctness.
One major issue with the manuscript is the experimental study. 1) The only additional algorithmic element introduced by the manuscript is the tau and it is not experimented. Is it crucial to use the temperature parameter? If yes, what is the effect of it? Manuscript needs a collection of ablation studies discussing the tau. 2) The main claim of the paper is "...make solving the control optimization process significantly less computationally and memory expensive." This might be true but not really experimented. Authors do not report any quantitative computation time and/or memory requirement study. I believe the latent DCEM is more memory and computation efficient but quantifying this is important.
I am curious on the choice of CEM. There are other methods which can be utilized since this is basically a bi-level optimization problem. One can use implicit gradients or similar methods (like: https://arxiv.org/abs/1602.02355, https://arxiv.org/abs/1809.01465, https://arxiv.org/abs/1909.04630, http://proceedings.mlr.press/v22/domke12/domke12.pdf).  Can these methods also be utilized instead of back-propagation through optimization procedure? If yes, you should compare with them or explain why you did not. If no, you should explain why.
In summary, the paper is very impactful. On the other hand, the proposed empirical study significantly lacks in many aspects. I would be happy to increase my score if authors can address these issues.


Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method. However, MAML is notorious for being hard to train because of the existence of two learning rates. Therefore, in this paper, we derived a sufficinent condition of inner learning rate α and meta-learning rate β for a simplified MAML to locally converge to local minima from any point in the vicinity of the local minima. We find that the upper bound of β depends on α. Moreover, we show that the threshold of β increases as α approaches its own upper bound. This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network. Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible α; next, tune β based on the chosen value of α.

This paper studies the two learning rates \alpha and \beta used in Model-Agnostic Meta-Learning (MAML) algorithm by [Finn et al., 2017]. MAML is known to be difficult to train, and part of the reason why is the need to tune the two learning rates. Under simplifications, the paper derives some necessary conditions on \alpha and \beta for the MAML iterates to converge to local minima, and then verifies the theory by experiments on synthetic and real-world data.
Overall, I think this paper should be rejected. In my opinion, this paper analyzes a simplified setting which is far from the original MAML setting and yet derives conditions that are not very meaningful or useful. Although it is interesting to see some match between theory and experiments (especially Fig 4), the theory part seems to have room for improvement, and I will detail the reasons in the following.
I believe that the analysis is done on an overly simplified setting and easily breaks without such simplifications.
- The paper assumes that training set is equal to test set and these sets do not change over iterations, which is essentially equivalent to assuming full gradient access; this is different from the stochastic setting of MAML.
- The "necessary condition" is derived in the case where there is only one task, and there is no discussion on generalizing to some other tasks. This means that the setting is too simplified so that it is not even a meta-learning setup.
- In the extension to multiple tasks, the paper derives "sufficient conditions" to a set of "necessary conditions" for convergence of MAML to local minima. This means that, if we consider the sets of events A = {convergence of MAML to local min}, B = {necessary conditions for A}, and C = {sufficient conditions for B}, A is a subset of B and C is a subset of B and nothing can be specified between A and C. In the extreme case, A and C may be even disjoint subsets of B. Thus, at least in theory, the conditions for multiple tasks do not tell us anything about convergence.
- The analysis relies on a number of "A is approximately equal to B" arguments without careful handling of errors, e.g., (3) and (4), and Tg = 0.
I also have concerns about the correctness of the analysis in the single task case.
- In Section 3.1.2 and eq (12), the paper reparametrizes \theta to v and analyzes updates on v. How is (12) obtained from the original update rule of \theta? In fact, the paper analyzes steepest GD but doesn't provide the explicit update rule; for example, is it steepest GD with respect to which norm? It'd be helpful to have the details in the main text.
- Even if (12) is true, there are pathological cases lying in a set of measure zero, where (12) converges even when (13) is not satisfied. For simplicity, consider v(t+1) = G v(t), where G is a real symmetric matrix. Assume all but one eigenvalue \lambda_1 are greater than 1 and |\lambda_1| < 1. Let v_1 be the corresponding eigenvalue for \lambda_1. Then, if v(0) = v_1, the sequence v(t) converges to zero. This means that the "necessary condition" derived in (16) may not actually be a necessary condition for convergence.
There are also some points in the main text that doesn't describe the setting clearly; if the reader has no prior knowledge of MAML algorithm, they may get confused.
- In the update rule (1) and (2), it'd be better to mention that \nabla_\theta L_\tau (\theta) is an *estimate* of the true gradient using the training data and test data, respectively. In their current status, the gradients in (1) and (2) will read as the full gradients of the population where the training/test data points are sampled from.
- Section 2.1 only introduces the case where the update (1) is done only once in the inner loop. However, later Section 2.2 and Section 3, the paper says "only one step is taken for update...". Without the prior knowledge that there are versions in which multiple updates (1) are done, the readers can easily get confused.
Minor comments
- The word "minima" used throughout should better be corrected to "local minima," as mere minima may be understood as "global minima."
- In the abstract, there is a phrase "in contrast to the case of using the normal gradient descent method." Which setting do you mean by the "normal gradient descent"?
- After equations (3) and (4), the paper says "The above is known as the first-order approximation...", but as far as I'm concerned, the first-order approximation version of MAML is (3) without the hessian term, not (4). This statement can potentially be misleading.
- Column vectors / row vectors are mixed up. In (3), the partial derivatives are row vectors, but in (4) g_\tau(\theta)'s are column vectors. However, right above (3), g_\tau(\theta) is a row vector this time.
- Did (5) come from the approximation (4)? If so, as \tilde L(\theta) is defined to be L(\theta'), the two sides of (5) are "approximately equal" not "equal."
- In Eq (19): P_\tau (\theta - \theta^*) -> (\theta-\theta^*).
- At the end of page 6, what do you mean by "not a sum but a mean"? Isn't sum of n things equal to the mean, except for a factor of 1/n?


We present GQSAT, a branching heuristic in a Boolean SAT solver trained with value-based reinforcement learning (RL) using Graph Neural Networks for function approximation. Solvers using GQSAT are complete SAT solvers that either provide a satisfying assignment or a proof of unsatisfiability, which is required for many SAT applications. The branching heuristic commonly used in SAT solvers today suffers from bad decisions during their warm-up period, whereas GQSAT has been trained to examine the structure of the particular problem instance to make better decisions at the beginning of the search. Training GQSAT is data efficient and does not require elaborate dataset preparation or feature engineering to train. We train GQSAT on small SAT problems using RL interfacing with an existing SAT solver. We show that GQSAT is able to reduce the number of iterations required to solve SAT problems by 2-3X, and it generalizes to unsatisfiable SAT instances, as well as to problems with 5X more variables than it was trained on. We also show that, to a lesser extent, it generalizes to SAT problems from different domains by evaluating it on graph coloring. Our experiments show that augmenting SAT solvers with agents trained with RL and graph neural networks can improve performance on the SAT search problem.

The paper proposes learning a branching heuristic to be used inside the SAT solver MiniSat using reinforcement learning. The state is represented as a graph representation of the Boolean formula as in previous works, and the policy is parameterized as a graph neural network. At each step of an episode the policy selects a variable to branch on and assigns a value to it. The episode terminates once the solver finds a satisfying assignment or proves unsatisfiability. The reward function encourages the policy to reach terminal state in as few steps as possible. The policy is trained using DQN. Results on randomly generated SAT instances show that the learned policy is able to solve problems with fewer steps than VSIDS, the branching heuristic commonly used by state-of-the-art solvers.
Pros:
- The paper is nicely written and easy to read.
- The general idea of using RL to learn distribution-specific branching heuristics is a very interesting research problem, and SAT is a difficult test case for it.
- The experiments provide interesting insights, especially Figure 2 and the graph coloring results in Table 3.
Cons:
- Showing improvements in the number of steps compared to VSIDS is not interesting because VSIDS as implemented in MiniSat and state-of-the-art solvers like Glucose has been tuned to minimize running time rather than number of steps.  A better comparison would be to compare against a branching heuristic that is designed to be step-efficient -- e.g., the branching heuristic GGB proposed in Chapter 3 of Liang’s PhD thesis (available here: https://drive.google.com/file/d/1RzJtmdbjFeT2N84WDWQkBfQoGPF-qRoT/view?usp=sharing). GGB is more expensive than VSIDS, but if the time needed for branching is removed, a solver using GGB is faster than one using VSIDS (see figure 3.1 in the thesis).
- A better discussion of how to scale up the proposed approach to instances with millions of variables is needed. Although most ML papers on SAT deal with at most hundreds of variables, such small instances are trivial for the state-of-the-art solvers. The real challenge is to scale up to the instance sizes that are considered in SAT competitions. For example, Selsam and Bjorner 2019 https://arxiv.org/pdf/1903.04671.pdf tackle such instances in their NeuroCore work. (While the paper references that work, it doesn’t compare to it.) There is no attempt to address the scalability issue in this work. Apart from considering the challenges of successfully learning on large instances (briefly discussed in the conclusion), there should be an analysis on the inference cost of the graph neural network (which scales linearly with the number of variables and clauses) to make a single branching decision vs. that of VSIDS, and what that implies on how much reduction in the number of steps a learned branching policy would need to achieve before it can provide time savings over VSIDS. It may turn out that the reductions required of a learned branching policy are implausibly large. Without a better understanding of these challenges, it is not clear that learning can help much to improve the state-of-the-art SAT solvers.


In spite of the slow convergence, stochastic gradient descent (SGD) is still the most widely used optimization method due to its outstanding generalization ability and simplicity. On the other hand, adaptive methods have attracted much more attention of optimization and machine learning communities, both for the leverage of life-long information and for the deep and fundamental mathematical theory. Taking the best of both worlds is the most exciting and challenging question in the field of optimization for machine learning. In this paper, we revisit existing adaptive gradient methods from a novel point of view, which reveals a fresh understanding of momentum. Our new intuition empowers us to remove the second moment in Adam without the loss of performance. Based on our view, we propose a novel optimization method, the acute adaptive momentum (Acutum). To the best of our knowledge, Acutum is the first adaptive gradient method without second moments. Experimentally, we demonstrate that our method has a faster convergence rate than Adam/Amsgrad, and generalizes as well as SGD with momentum. We also provide a convergence analysis of our proposed method to complement our intuition.

This paper attempts to remove the use of the second moment in Adam in order to improve the generalization ability of Adam.
- Apriori, it is not clear why removing the second moment is important. Does it improve the generalization or decrease the runtime substantially?
- Please clarify how our method compares against Yogi (Adaptive Methods for Nonconvex Optimization, Zaheer' 2018) and the more recent RADAM (ON THE VARIANCE OF THE ADAPTIVE LEARNING
RATE AND BEYOND, Liu 2019) , both theoretically and empirically.
- The paper is meandering and confusing. State your update/algorithm and then explain how it compares against the other methods. Besides, there might be technical problems with it.
See the detailed review below:
- Section 1: "the generalization results (of adaptive methods) cannot be as good as SGD". Please cite the relevant papers, (for example, Wilson. 2017)  that show this empirically.
-  Section 1: "the proposed algorithm outperforms Adam in convergence speed" - Please clarify what "convergence speed" refers to. Is it the number of gradient evaluations, the rate of convergence or the wall-clock time. Please state how did you conclude this.
- Section 2: The update rule of Adagrad is incorrect. The step-size is constant \alpha and it is decreased over time because of the v_t term.
- Section 3: There is no guarantee on the approximation in Equation 5. Young's inequality and the resulting upper bound can be quite loose.
- Section 3: In general, it is not possible to have an update that decreases the loss on the current batch, but does not increase the previous batches loss. It is always possible to construct a counter-example to this.
- "In practice, the computational cost of computing" the gradient for all i is expensive. Indeed, this is batch gradient descent. I am not sure how this is relevant to the discussion in the paper.
- Cite and compare against the variance reduced methods (Stochastic Average Gradient, Schmidt, 2013; SVRG, Johnson, 2013) as these try to "approximate" the full gradient in order to decrease the variance.
- The derivation/formulation of Equation 8 is not clear to me. Why is the \hat{m} normalized?
- In algorithm 1, it seems you need to choose the sequence of step-sizes \alpha_t and \beta_t. How is this an adaptive method then? How are these sequences chosen theoretically and practically? Please clarify this.
- Section 4: Please compare the resulting regret bound to that of Adam, Adagrad and AMSgrad. Why does \alpha_t = O(1/t)? If we have to decrease the step-size according to a sequence, why should I not use standard SGD?
- Section 5: "we decay the learning rate by 0.1 every 50 epochs" This is not aligned with either the theory or the algorithm you proposed.


This paper makes two contributions towards understanding how the hyperparameters of stochastic gradient descent affect the final training loss and test accuracy of neural networks. First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typically arises when the batch size is large. In the noise dominated regime, the optimal learning rate increases as the batch size rises, and the training loss and test accuracy are independent of batch size under a constant epoch budget. In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade as the batch size rises. We support these claims with experiments on a range of architectures including ResNets, LSTMs and autoencoders. We always perform a grid search over learning rates at all batch sizes. Second, we demonstrate that small or moderately large batch sizes continue to outperform very large batches on the test set, even when both models are trained for the same number of steps and reach similar training losses. Furthermore, when training Wide-ResNets on CIFAR-10 with a constant batch size of 64, the optimal learning rate to maximize the test accuracy only decays by a factor of 2 when the epoch budget is increased by a factor of 128, while the optimal learning rate to minimize the training loss decays by a factor of 16. These results confirm that the noise in stochastic gradients can introduce beneficial implicit regularization.

The paper attempts to clarify the debate on large-batch neural network training, particularly on the relationship between learning rate, batch sizes and test performance. The authors claim two contributions towards understanding how the hyper-parameters of SGD affect final training and test performance: (1) SGD exhibits two regimes with different behaviours and (2) large-batch training leads to degradation of test performance even with same step budgets.
Overall, the authors did a comprehensive study on large-batch training with the support of extensive experiments. But I'm concerned with the novelty and contributions of this paper. I tend to reject this paper because (1) the first contribution of the paper is not new as it has already been recognized by a few paper that SGD exhibits two different regimes; (2) this paper makes the debate of large-batch training even muddier.
Main argument:
The paper does not do a great job in clarify the debate. Particularly, the authors mixed their observations up with the results of published works, making it hard to identify the contributions of this paper. For example, the two regimes mentioned in the paper has been identified by a few other works and the contribution of this paper is just to verify them again. Also, I find the experiments done in section 3 and 4 are similar to previous works and even the conclusions are similar. The only new observation I'm aware of in these two sections is that the training loss and test accuracy are independent of batch size in the noise dominated regime.
Back to introduction section, the goal of this paper (as claimed in the beginning of second paragraph) is to clarify the debate. But does this paper really achieves this goal? In terms of learning rate scaling, this paper gets similar conclusions as Shallue et al. (2018). In terms of the difference between vanilla SGD and SGD with momentum, Zhang et al. (2019) already argued that the difference depends on specific batch sizes and SGD with momentum only outperforms SGD in the curvature dominated regime.
I think the authors should instead focus on the discussion of generalization performance and the observation that training loss and test accuracy are independent of batch size in noise dominated regime. To my knowledge, this part is novel and interesting.
In summary, I'm inclined to reject this paper given the current version. However, I think the paper is still worth reading if the authors can reorganize the paper and I might increase my score if my concerns get resolved.


Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, clipping controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum. This intuition has been made precise in a line of recent works, which show that suitable clipping can yield significantly faster convergence than vanilla gradient descent. In this paper, we propose a new lens for studying gradient clipping, namely, robustness: informally, one expects clipping to mitigate the effects of noise, since one does not overly trust any single sample. Surprisingly, we prove that for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness. On the other hand, we show that a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function. As a special case, this yields a simple, noise-robust modification of the standard cross-entropy loss which performs well empirically.
In this paper, we propose a new lens with which to study gradient clipping, namely, robustness: intuitively, clipping the gradient prevents over-confident descent steps, which is plausibly beneficial in the presence of noise. Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classification, which has received significant recent interest (Scott et al., 2013;Natarajan et al., 2013;Liu & Tao, 2016;Patrini et al., 2017;Ghosh et al., 2017;Han et al., 2018;Zhang & Sabuncu, 2018;Song et al., 2019;Thulasidasan et al., 2019;Charoenphakdee et al., 2019). We study this question, and provide three main contributions:
(a) we show that gradient clipping alone does not endow label noise robustness to even simple models. Specifically, we show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a "Huberised" loss (Lemma 1, 2). While such Huberised losses preserve classification calibration (Lemma 3), they are not robust to label noise (Proposition 4). (b) we propose composite loss-based gradient clipping, a variant that does have label noise robustness.
Specifically, for losses comprising a base loss composed with a link function, (e.g., softmax crossentropy), we only clip the contribution of the base loss. The resulting partially Huberised loss preserves classification calibration (Lemma 6), while being robust to label noise (Proposition 7). (c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise ( §5).
To illustrate the difference between standard and composite loss-based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz. the logistic loss for binary classification. Recall that the logistic loss comprises the log-loss (or cross-entropy) composed with a sigmoid link. The Huberised loss arises from loss-based gradient clipping, and linearises the entire logistic loss beyond a threshold. The partially Huberised loss arises from composite loss-based gradient clipping, and linearises only the base loss (i.e., log-loss) beyond a threshold (right panel). Once combined with the sigmoid link, the overall partially Huberised loss asymptotically saturates. Our analysis in §3, §4 establishes that Huberised losses are not robust to label noise, but partially Huberised losses are (Proposition 4, 7).
In (a), we relate gradient clipping to loss "Huberisation", which linearises the logistic loss when it exceeds a fixed threshold. In (b), we introduce composite loss-based gradient clipping, which is equivalent to a "partial loss Huberisation" that linearises only the cross-entropy loss but leaves the sigmoid link untouched. Figure 1 illustrates the Huberised and partially Huberised logistic loss.

This paper studied a fundamental problem in robust machine learning, that is, can gradient clipping mitigate label noise? The paper is well written, clearly motivated, highly novel and significant not only in a theoretical sense but also in a practical sense.
As argued in the abstract, gradient clipping is a widely-used technique which is generally motivated from the OPTIMIZATION point of view. In this paper, the authors proposed an entirely new motivation of gradient clipping from the ROBUSTNESS point of view, since intuitively it should be able to mitigate label noise. Surprisingly, the authors proved that for some simple binary classification with label noise, standard gradient clipping does not provide robustness; on the other hand, a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function.
The major contributions were well summarized in the bottom of page 1, the related work was discussed in sec 4.3, and the caveats of this new methodology were also given in sec 4.4. Note that the proposed composite loss-based gradient clipping is applicable even on top of existing noise-robust losses, for example, the generalized cross-entropy loss, and this serves as a convincing demonstration of the great significance of the paper. Actually, the paper is full of insights, and I really enjoyed reviewing it.
I have a few questions on Tables 1 and 2. Table 1 said the standard gradient clipping is not robust according to Proposition 4. However, Proposition 4 is for the loss-based gradient clipping in (7) rather than the standard gradient clipping in (6). Perhaps I have missed something, but why the non-robustness of (7) can imply the non-robustness of (6)?
In Table 2, Linear loss on MNIST, the test accuracy was 9.6 when rho=0.6. Is this a typo? The accuracy was still 78.8 when rho=0.4. Moreover, the authors explained why the theoretically grounded linear loss performed so badly, that is, the optimization was so difficult. How about CE+clipping loss on CIFAR-100? The standard gradient clipping was also harmful here, even when there was no label noise at all. Was this due to bad optimization or robustness (or both)?
Since distributionally robust supervised learning is a future direction to go, I think the authors may be interested in a thought-provoking paper:
W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? ICML 2018.


We present a 3D capsule architecture for processing of point clouds that is equivariant with respect to the SO(3) rotation group, translation and permutation of the unordered input sets. The network operates on a sparse set of local reference frames, computed from an input point cloud and establishes end-to-end equivariance through a novel 3D quaternion group capsule layer, including an equivariant dynamic routing procedure. The capsule layer enables us to disentangle geometry from pose, paving the way for more informative descriptions and a structured latent space. In the process, we theoretically connect the process of dynamic routing between capsules to the well-known Weiszfeld algorithm, a scheme for solving iterative re-weighted least squares (IRLS) problems with provable convergence properties, enabling robust pose estimation between capsule layers. Due to the sparse equivariant quaternion capsules, our architecture allows joint object classification and orientation estimation, which we validate empirically on common benchmark datasets.

This work builds a capsule network, for use with point cloud data, that has units that are equivariant to SO(3) 3D rotations.  I found that the authors made the case for rotation equivariance well and I liked the analysis of the dynamic routing approach and its mapping to the Generalized Weiszfeld Iterations.
As a caveat to the review I should point out that while I am familiar with capsule networks, they are not my main area of expertise (I do work on geometry however) so this should be taken into consideration - apologies if there is literature I have missed.
My main concern is that group equivariance for Capsule networks has been studied before - I acknowledge that SO(3) can present additional challenges but would have expected a work that specifically targeted SO(3) to investigate the relative benefits of the different approaches to encoding the lie algebra and not just quaternions. How do we know that quaternions are the best fit for the Capsule framework? Could the authors show empirical results that indicate that other representations for the equivariance do not work as well? My experience from optimization is that the best representation can often be application specific and it would be very helpful to understand their relative merits for this sort of architecture.
The preparation of the local patches and the estimation of the rotations seems very important to these applications to me and would be worthy of more discussion and an empirical presentation of their different merits. I know the authors discussed the issue of the second axis but it would seem worth including an experiment to detect the sensitivity in a similar manner to the point cloud density.
My biggest concern about the paper is the presentation of the results. Please could the authors provide appropriate error bars for the tables - this is the least action necessary to at least start to estimate significance. Ideally something like histograms or violin plots would be more useful. The use of a single dataset makes it hard to identify the efficacy of the approach - especially when it is not clear how the training and test data are set-up. Are the same objects used in training and test but with different rotations? Or different objects from the same categories. What is the protocol for the AR test set?
I agree that table 1 confirms the equivariance so that is great but it is less clear that the network is suitable for the pose task since the baselines seem very simple for the pose estimation experiment (and the related work covers a lot of prior work in this area) and again, without error bars it is hard to judge significance.
It is nice that the authors discuss limitations and I agree with the comment on symmetries - would it be possible to include an illustrative example to confirm these suspicions?
Other notes:
I have no issue with the proofs - they all seemed to make sense to me.
I found Figure 1 (and to a lesser extent Figure 3) to be unclear and uninformative - the notation doesn't match the main part of the paper (also where does C come from in the caption?) and its not clear what the illustrations are meant to represent. Would it not be much more clear to include something like Algorithm 1 instead of Figure 1 which is much more clear and easy to replicate the precise process.
I think there is a lot of material in the appendix that should really be in the main paper - I'm happy for the proofs to be in the appendix but I think it is a bit wrong to essentially violate the page restrictions by moving important related work into the appendix. The whole of section 2 is standard text-book information about quaternions - would it not be more appropriate for that to be in the appendix and the related work to be in the main paper?
Could the authors indicate the use of the Frobenius norm in eqn (5)? There also seems to be a mixing of notations for inner products through-out the paper and it might be helpful to standardize this?


The intensive computation and memory requirements of generative adversarial neural networks (GANs) hinder its real-world deployment on edge devices such as smartphones. Despite the success in model reduction of convolutional neural networks (CNNs), neural network quantization methods have not yet been studied on GANs, which are mainly faced with the issues of both the effectiveness of quantization algorithms and the instability of training GAN models. In this paper, we start with an extensive study on applying existing successful CNN quantization methods to quantize GAN models to extreme low bits. Our observation reveals that none of them generates samples with reasonable quality because of the underrepresentation of quantized weights in models, and the generator and discriminator networks show different sensitivities upon the quantization precision. Motivated by these observations, we develop a novel quantization method for GANs based on EM algorithms, named as QGAN. We also propose a multi-precision algorithm to help find an appropriate quantization precision of GANs given image qualities requirements. Experiments on CIFAR-10 and CelebA show that QGAN can quantize weights in GANs to even 1-bit or 2-bit representations with results of quality comparable to original models.

Summary:
The authors address the quantization of Generative Adversarial Networks (GANs). The paper first performs a sensitivity study for both the generator and the discriminator to quantization methods. Building upon the conclusion of this study, the authors propose a scalar quantization method (QGAN) and compress models to 1 bit of 2 bits weights and show generated images and metrics by the compressed models.
Strengths of the paper:
- As well stated in the introduction, the compression of GANs (in particular the generator, which is used at inference time) is of practical interest and, to the best of my knowledge, novel. This novelty can be explained by (1) the fact that it takes tome for quantization methods to percolate the entire deep learning field and/or (2) the fact that quantizing GANs has its specificities and own challenges that have not been yet addressed (this is the claim of the authors).
- The sensitivity study is of interest for the community that can build upon this work. The conclusions (discriminator more sensitive than generator to quantization, quantizing both generator and discriminator helps) are sensible and interesting.
Weaknesses of the paper:
- The related work section could be greatly improved, thereby showing the limited novelty of the proposed method (QGAN). Indeed, the authors propose to learn the optimal scaling parameters alpha and beta. Many works perform this already and are currently missing in this section, see for instance the two recent surveys: "A Survey on Methods and Theories of Quantized Neural Networks", Guo, "A Survey of Model Compression and Acceleration for Deep Neural Networks", Cheng et al.
- Results. The results are not sufficient to justify the performance of the method for two reasons. (1) First, the scale is crucial in assessing the performance of a quantization method. As an example, it is easier to quantize small ResNets on CIFAR-10 than large ResNets on ImageNet. Thus, scales enables to better discriminate between various approaches. I acknowledge that this requires large computational resources but this would greatly strengthen the paper (2) Second, GAN metrics have known shortcomings (see for instance "How good is my GAN?", Shmelkov et al.), so the strength of Table 2 is limited. This is in part alleviated by the authors by showing generated images (which is a good practice), but again echoing point (1), larger images would have helped assess better the quality of the quantization.
Justification of rating:
The authors propose a sensitivity study that is interesting for the community. However, the proposed method lacks novelty and the results are not convincing enough. I encourage the authors to pursue in this interesting direction which has important practical implications.


Graph neural networks have been proved very effective for a variety of prediction tasks on graphs such as node classification. Generally, a large number of labeled data are required to train these networks. However, in reality it could be very expensive to obtain a large number of labeled data on large-scale graphs. In this paper, we studied active learning for graph neural networks, i.e., how to effectively label the nodes on a graph for training graph neural networks. We formulated the problem as a sequential decision process, which sequentially label informative nodes, and trained a policy network to maximize the performance of graph neural networks for a specific task. Moreover, we also studied how to learn a universal policy for labeling nodes on graphs with multiple training graphs and then transfer the learned policy to unseen graphs. Experimental results on both settings of a single graph and multiple training graphs (transfer learning setting) prove the effectiveness of our proposed approaches over many competitive baselines.

Thank you for the author response.
Original review:
This paper presents a method for active learning on graphs, including a novel setting of transferring an active learning policy to unseen graphs.  The problems tackled here are important and the method is shown to improve over previous work in some cases.  On the down side, the evaluation may be missing one important method of comparison, the reasons for the proposed approach winning over previous work are not made explicit, and the empirical advantage of the approach is inconsistent (by my count, in the majority of cases the F1 advantage over previous work is within the standard error).  This paper has strengths but I feel it needs further refinement before publication.
The introduction of the paper claims that existing approaches for active learning on graphs are domain-specific and may not apply well to new domains.  But later, in the experiments, different reasons for the proposed approach's wins are given (in particular, on large graphs the proposed approach does relatively better vs. AGE which the paper suggests is due to the more complex nonlinear models used in the proposed approach).  In general, this paper’s approach tends to win over the primary baseline (the AGE method from Cai et al. 2017), but the wins are relatively small and inconsistent (esp. taking into account the standard error) whether the methods are evaluated in the single-graph setting or the transfer learning setting (of the homologous or heterogeneous variety).  If the limitation of previous work was domain-specificity, I would expect to see much larger wins on the transfer learning setting.  In general, an analysis that explains more what is driving the gains of this approach over the AGE approach would help us know how to build on this paper’s method in future work.
I was curious why the paper does not compare against the following work, which also presents an approach that wins over AGE:
“Active Discriminative Network Representation Learning,” Gao et al., IJCAI 2018
Lastly, the distillation-based approach, which learns graph-specific policies that are trained to fit their target graphs and to minimize their KL divergence from a single global shared policy, was interesting.  The fact that it doesn’t work much better than the joint policy is somewhat disappointing, but it’s still interesting.
Minor:
Sec 3.2: unclosed parenthesis in first paragraph
“Moreover, we also average the struc2vec features of all previously annotated nodes to capture the historical information” -- since the model has only node-level features, I didn’t understand how this average across multiple nodes was fed in as a node feature.  Is it used in all nodes?  Only annotated nodes?


The allocation of computation resources in the backbone is a crucial issue in object detection. However, classification allocation pattern is usually adopted directly to object detector, which is proved to be sub-optimal. In order to reallocate the engaged computation resources in a more efficient way, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different feature resolution and spatial position diectly on the target detection dataset. A two-level reallocation space is proposed for both stage and spatial reallocation. A novel hierarchical search procedure is adopted to cope with the complex search space. We apply CR-NAS to multiple backbones and achieve consistent improvements. Our CR-ResNet50 and CRMobileNetV2 outperforms the baseline by 1.9% and 1.7% COCO AP respectively without any additional computation budget. The models discovered by CR-NAS can be equiped to other powerful detection neck/head and be easily transferred to other dataset, e.g. PASCAL VOC, and other vision tasks, e.g. instance segmentation. Our CR-NAS can be used as a plugin to improve the performance of various networks, which is demanding.

<Strengths>
+ This paper performs architecture search for object detection, especially for the computation allocation across different resolutions. It is a new application for NAS research.
+ The proposed approach shows some marginal improvement of object detection accuracy across multiple backbones and datasets.
+ This work proposes a new formulation to apply NAS approach to object detectors, including linking between the ERF and the computation allocation of backbone and two-level hierarchical search for stages and convolution operations.
<Weakness>
1. This paper can be regarded as an engineering work for a new domain of problem with little technical novelty.
- From the perspective of NAS research, the proposed approach has little technical novelty; instead, it seems like an application of existing techniques (or even simpler ones) (e.g.  to a new domain of problem - object detection.
- Given that the NAS is only applied to CNN backbones in this paper, the novelty (of proposing a new task) may be further weakened.
- The novelty of this work over existing works of “NAS on detection” (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019) is not justified.
2. Experimental results are rather weak.
- This paper only focuses on showing that the proposed method marginally improves the performance of basic backbones (MobileNet V2 and ResNet 18/50/101). However, the improvement gaps are rather marginal (about 1.0% in average as shown in Fig.5).
- This paper does not compare its performance with other SOTA detection methods but only compare with some baselines instead. However, the performances of baselines are too low; for example, in Table 1, the mAP of the ResNet101 baseline is 38.6, which is lower by about 10 than the SOTA detector with the same ResNet101+FRCNN+FPN. It may not be convincing to use the baselines that have more than 20% lower accuracy compared to SOTA and show only ~1.x improvement over it.
- Given that the proposed approach is applicable to any backbones and detectors, more experiments should be done using recent stronger baselines (including many recent tricks like RoIAlign and DCN).
- Another experimental weakness is lack of comparison with existing NAS methods on object detections such as (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019). Even though experiment settings here may be different from those of these papers,  they should be compared in any reasonable ways. Moreover, the detection accuracies reported in this paper are not as good as the numbers of these papers. This paper argues that the proposed approach is complementary to these methods, so it would be good to report the results of the combined model with them.
3. The paper is written poorly.
- There are many grammatically wrong and awkward expressions. The draft should be thoroughly proofread.
<Conclusion>
Although this work shows a new application of NAS for object detection, my initial decision is ‘weak reject’ mainly due to lack of technical novelty, limited experiments and poor writing.
<Post-rebuttal comments>
Authors' responses partly resolved my concerns on the experiments. I have no object to accept this paper.


We argue that flow-based density models based on continuous bijections are limited in their ability to learn target distributions with complicated topologies, and propose localised generative flows (LGFs) to address this problem.
LGFs are composed of stacked continuous mixtures of bijections, which enables each bijection to learn a local region of the target rather than its entirety. Our method is a generalisation of existing flow-based methods, which can be used without modification as the basis for an LGF model. Unlike normalising flows, LGFs do not permit exact computation of log likelihoods, but we propose a simple variational scheme that performs well in practice. We show empirically that LGFs yield improved performance across a variety of density estimation tasks.

Summary:
This paper conjectures that normalizing flows are fundamentally limited due to the architecture assumption that the generative function g is continuous in x. It is argued that this constraint makes maximum likelihood estimation difficult in general. Localised generative flows are proposed as a solution and consist in modeling the generative model as a continuous mixture of bijections. Experiments suggest an improvement over MAF.
Decision:
The observation that continuity imposes a hard constraint on the network is sound, and the proposed solution appears to show some improvement. However, in its current state, this work appears to be quite fragile both from a theoretical and experimental point of view. First, it is only conjectured that this constraint poses actual problems. Second, the experimental evaluation is weak and insufficient. It omits comparisons with more recent generative flows that have shown to be able to model discontinuous densities. For this reason, I do not recommend the paper for acceptance.
Further arguments:
- The whole paper rests on intuition without strong theoretical backup.
- The experiments are quite poor and results frankly oversold. It is said the method "improves performance across a variety of common density benchmarks". While we see improvements in Table 1 over MAF, the comparison omits all recent architectures based on Normalizing Flows, such as TAN (Olivia et al, 2018), NAF (Huang et al, 2018), B-NAF (De Cao et al, 2019) or SOS (Jaini et al, 2019). All of those methods have reported better results than those provided in Table 1. They have also been shown empirically to work for discontinuous densities. While I understand that LGF can be combined with any flow architecture, the question remains whether using a continuous mixture translates into significant improvements for those baselines as well. The experimental benchmarks also omit datasets such as BSDS300, for which the higher dimensionality is usually challenging. The same goes for Table 2 which omits recent and better results, such as Glow or FFJORD.
- Closer to LGF, a proper experimental comparison to RAD (Dinh et al, 2019) would be appreciated.
- The proposed architecture supposedly enables better generative models. However, this comes at the price that the density can no longer be evaluated exactly and analytically. Since normalizing flows are also typically slow for sampling, this makes the benefits of the proposed architecture quite limited. In particular, it is not clear why generative models that are good at sampling only (e.g., GANs) should not then be preferred?
- As a result of the point above, the experimental results are reported only in terms of approximated negative log-likelihood. I do not think this is fair, since models like MAF do provide exact values. It also makes the comparison with previous methods more difficult.
Further feedback:
- As per ICLR policy, higher standards should be applied to papers with 9 or more pages. I am confident the paper could be written within 8 pages only.


Counterfactual regret minimization (CFR) is a fundamental and effective technique for solving Imperfect Information Games (IIG). However, the original CFR algorithm only works for discrete states and action spaces, and the resulting strategy is maintained as a tabular representation. Such tabular representation limits the method from being directly applied to large games. In this paper, we propose a double neural representation for the IIGs, where one neural network represents the cumulative regret, and the other represents the average strategy. Such neural representations allow us to avoid manual game abstraction and carry out end-to-end optimization. To make the learning efficient, we also developed several novel techniques including a robust sampling method and a mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) method, which may be of independent interests. Empirically, on games tractable to tabular approaches, neural strategies trained with our algorithm converge comparably to their tabular counterparts, and significantly outperform those based on deep reinforcement learning. On extremely large games with billions of decision nodes, our approach achieved strong performance while using hundreds of times less memory than the tabular CFR. On head-to-head matches of hands-up no-limit texas hold'em, our neural agent beat the strong agent ABS-CFR 1 by 9.8±4.1 chips per game. It's a successful application of neural CFR in large games.

The authors proposed a double neural counterfactual regret minimization algorithm (DNCFR) that uses a RegretSumNetwork to approximate cumulative regret and an AvgStrategyNetwork to approximate the average strategy. To help the training, the authors use robust sampling and a mini-batch training methods. The main contributions of the paper are: First, the DNCFR algorithm and the recurrent neural network architecture; Second, an efficient sampling and training method; Third, plenty of experiments that corroborate the effectiveness of DNCFR.
It is interesting and meaningful to develop neural-based CFR, in order to eliminate the manual abstraction and apply CFR to large-scale imperfect information games. The authors tested their algorithm on a medium scale game HUNL(1) (with 2 * 10 ^ 8 information sets) and trained a blueprint strategy on large scale game HUNL(2), which is combined with value networks from DeepStack and beats ABS-CFR a lot. It is great to see that DNCFR works on large scale games. However, both HUNL(1) and HUNL(2) are one-round games and it not clear how to combine the blueprint strategy trained by DNCFR with DeepStack. What’s more, as DNCFR is only effective on first round as the blueprint strategy trainer when played against ABS-CFR, it is more likely that DeepStack beats ABS-CFR, instead of DNCFR beats it. So the result in Figure 7(c) is not so convincing.
Unlike tabular CFR that save regrets and strategies for all the information sets or other neural-based algorithms that need large reservoir buffers. It only needs to save data sampled from the most recent iterations, which saves much memory. In fact, this is a bootstrap method borrowed from Reinforcement learning. Though the method save memory and has lower variance than methods that use reservoir buffers, it is bias as it trains the new RSN and ASN based on the output of the old networks. It seems good when the game size is small and the CFR iterations is small. It may needs very large CFR batches and very many gradient descent updates when training on large scale games, in order to control the bias. The results in Figure 7(a) and 7(b) are limited in CFR iterations. Experiments using different gradient descent updates and different CFR batch while given more CFR iterations should be tested, in order to show the effect of the bias training.
In “Algorithm 4”. The calculation of average strategy seems wrong. Because you are using MCCFR, According to “Monte Carlo sampling and regret minimization for equilibrium computation and decision-making in large extensive form games”, you may need a method call “stochastically-weighted averaging”. It should be noted that the sampling probability of
each information set is not equal. You may need to discuss this.
The authors train the network for 2000 updates when the batch size is 256 for Leduc and 100000 for HUNL(1) and HUNL(2) in every CFR iteration (I am not sure how much gradient updates are used in HUNL(2), it is not given). There's quite a lot of updates in every CFR iteration. But it is acceptable when compared to Deep CFR proposed by Brown, which uses 4000 updates and the batch size is 10000.
Experiments:
1. In the ablation studies, the algorithms are tested on small scale game Leduc(5). It is quite a small game that event the size neural parameters is larger than the size of information sets. It is OK but larger games make more sense. Especially in the
experiment of “Individual network”, as this experiment is important to show that
DNCFR is comparable to tabular CFR and the bias is acceptable.
2. The paper didn’t show what the learned regret and average strategy looks. If they are
showed, it would be helpful to understand the bias in the bootstrap learning.
3. In the part “Is robust sampling helpful”, the authors want to show that the robust sampling with k=1 is better than outcome sampling. But I didn’t find how they set the exploration parameter in outcome sampling and I am afraid that it doesn’t make sense. Because outcome sampling has a parameter to adjust the exploration. According to "Monte Carlo sampling and regret minimization for equilibrium computation and decision-making in large extensive form games", the best exploration parameter is different in different game, but it is almost sure that totally exploration is not the best setting (it is equivalent to the robust sampling with k = 1).
4. In the part “Do the neural networks generalize to unseen infosets”. The authors claims that it is true. But the experiment only shows that the neural network don’t forget
information sets that trained before.
5. In the part “How well does DNCFR on larger games”, the DNCFR is limited to 100
iterations while is allow to run for 1000 iterations in other experiments. 100 iterations
are too few to show the effectiveness of DNCFR on these games.
6. The algorithm is tested on HUNL(1) and HUNL(2), which are one round and action- abstracted version of HUNL. But the authors should give more detail description of
these games.
7. It is not clear how to combine the blueprint strategy trained by DNCFR with
DeepStack, as DeepStack uses continual resolving and don’t need any blueprint strategy. And it would be interesting if the head-to-head performance of DNCFR agent on large scale games (for example, the FHP with two rounds and more than 1e^9 information sets) is reported, instead of the performance of the agent that combined with DeepStack.
8. In section 5.4, “When variance reduction techniques are applied, Figure 7(c)...”. The authors didn’t explain why the variance reduction techniques are needed here, but in order to compare the algorithm directly, some other advanced techniques should not be used here.


Learning disentangled representations of data is one of the central themes in unsupervised learning in general and generative modeling in particular. In this work, we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate. To this end, we present a hierarchical model and a training method (CZ-GEM 1 ) that leverages some of the recent developments in likelihoodbased and likelihood-free generative models. We show that CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variable disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in the supervised and unsupervised settings.

This paper proposes a method for learning disentangled representations.  The approach is used on both supervised (where the factors to be disentangled are known) and unsupervised settings. The authors demonstrate the efficacy of their approach in both settings on several datasets with both quantitative and qualitative results.
This task is an important one. However, I found that the contribution of this paper is fairly small. The proposed approach seems reasonable but it is mostly a work of engineering and provides little insights into the problem nor the proposed model.
The setup where labeled data (c) also seems a bit unnatural (this also seems to be confirmed by the fact that the authors had to build datasets for the problem). Perhaps the authors could give examples of situations where this would naturally arise. In practice, it seems difficult to obtain these data for all required variables to be disentangled.
The unsupervised results are more interesting but not very much explored (a single set of sampled faces). I was also curious as to why the learned Y's are blurry. This sort of two-stage generation is also potentially interesting, I was wondering if the authors had ideas to generalize this idea.
I also was not convinced by the experiments which are mostly qualitative. I did not find that this set of experiments provide enough support to the proposed method.
Detailed comments:
- It is a bit unclear to me how the authors propose to obtain independent posteriors over z and c. Is it purely empirical or is there a formal reason that guarantees it?
- Some of the figures your report are compelling but it is a bit unclear to the reader if the results are general (e.g., the examples could have been hand-picked). Are there any quantitative measures you could provide (in addition to Tables 1 and 2 which don't measure the quality of the approach)?
- Comparing to CGAN seems reasonable but given the task at hand, it seems like other methods could have been tried (although I do realize that no one may have done this before for deep generative models).
Other comments:
- In Figure 3, it would be good to label the upper trapezoid.
- Some paragraphs are very long and the manuscript may benefit from segmenting them into multiple paragraphs.


We propose a new model for making generalizable and diverse retrosynthetic reaction predictions. Given a target compound, the task is to predict the likely chemical reactants to produce the target. This generative task can be framed as a sequence-to-sequence problem by using the SMILES representations of the molecules. Building on top of the popular Transformer architecture, we propose two novel pre-training methods that construct relevant auxiliary tasks (plausible reactions) for our problem. Furthermore, we incorporate a discrete latent variable model into the architecture to encourage the model to produce a diverse set of alternative predictions. On the 50k subset of reaction examples from the United States patent literature (USPTO-50k) benchmark dataset, our model greatly improves performance over the baseline, while also generating predictions that are more diverse.

The authors present an approach to improve performance for retro-synthesis of chemical targets in a seq2seq setting using transformers. The authors have encouraging results and the paper was fairly easy to read and follow. However there are a variety of concerns that the authors need to address:
The technical contributions in this paper are somewhat thin. The main contributions are data augmentation techniques, pre-training and a mixture model that seems to improve performance on the USPTO-50K dataset. The novelty is quite low and it’s not clear if this will transfer to another domain. The impact is also low as the pre-training techniques using bond breaking and template-based are specific to this problem task. Additionally, mixture models for encouraging diversity is a simple instance of ensembling.
Using deep learning to this application area is also not novel. This paper largely builds upon previous work with Transformers from Schwaller et. al and Karpov et. al.
Other clarifications/issues:
 - The experimental results are based only on the USPTO dataset. It’s unclear how significant the results are. The authors can consider using diverse datasets or applying their techniques to another application domain to bolster their claims.
 - Table 3, lists the average number of unique reactions classes. The authors say “ … we predict the reaction class for every output of our models… “ . It’s not clear how it makes sense to calculate diversity when there’s no ground truth available for determining if the predicted output is a valid synthesis for the target. To say this another way, what good is diversity if the prediction is incorrect?
 - Table 3, lists human eval results. The details here seem quite vague. How does a human determine something to be more diverse? What is the rubric they use? How qualified is the human in being able to judge this task?
 - Figure 6 does not have a color scale.


Initial value problems, i.e. differential equations with specific, initial conditions, represent a classic problem within the field of ordinary differential equations (ODEs). While the simplest types of ODEs may have closed-form solutions, most interesting cases typically rely on iterative schemes for numerical integration such as the family of Runge-Kutta methods. They are, however, sensitive to the strategy the step size is adapted during integration, which has to be chosen by the experimenter. Here, we show how the design of a step size controller can be cast as a learning problem, allowing deep networks to learn to exploit structure in the initial value problem at hand in an automatic way. The key ingredients for the resulting Meta-Learning Runge-Kutta (MLRK) are the development of a good performance measure and the identification of suitable input features. Traditional approaches suggest the local error estimates as input to the controller. However, by studying the characteristics of the local error function we show that including the partial derivatives of the initial value problem is favorable. Our experiments demonstrate considerable benefits over traditional approaches. In particular, MLRK is able to mitigate sudden spikes in the local error function by a faster adaptation of the step size. More importantly, the additional information in the form of partial derivatives and function values leads to a substantial improvement in performance. The source code can be found at https://www.dropbox.com/sh/ rkctdfhkosywnnx/AABKadysCR8-aHW_0kb6vCtSa?dl=0

## Summary ##
The authors present a method for learning a step-size adaptation strategy for Runge-Kutta methods. The principal contributions of the paper are:
1. They define a loss function that better captures global performance of the controller, rather than just local behavior.
2. They propose a set of input features to a step size controller. It includes the intermediate Runge-Kutta evaluation values, which allow the controller to approximate the derivatives of the Jacobian function $g$.
3. They describe a recurrent architecture for adapting step size over time.
Runge-Kutta methods are a workhorse of ordinary differential equations and choosing step size is one of the central challenges involved in their application. Better methods for step size selection would definitely be of broad interest. As the authors point out, existing methods often consist of hand-tuned heuristics---a feature that often suggests machine learning could provide significant improvements.
While the premise of the paper is very promising, I don't think it is ready to be accepted to ICLR at this time. Most significantly, the experimental results are not particularly compelling. I believe the authors should refine their method, aim for better experimental results and resubmit. I have included more specific comments below.
## Specific Comments ##
1. I think the paper would benefit from a clearer description of the RK step-size selection problem. For instance, for a p^th order RK solver, at each time step,
   * Inputs: t, y(t), g  # Also possibly intermediate values from previous time steps.
   * Select a step size h(t)
   * Evaluate g at p different points based on h(t).
   * Use these evaluations to compute a value of y_(t + h_t)
   For those that aren't familiar with these methods (at ICLR there will be many!) I think this would help explain where the authors' method (and the other methods your describe) fits into the larger algorithm.
2. I think a short explanation of error estimation would be helpful in addition to the reference to Butcher. This estimation is critical to step size adaptation. In particular I think this would be clearer if the authors expanded the paragraph at the bottom of page 4, where they describe error in a polynomial in (t_n, h) whose coefficients are derivatives of g.
3.  The authors' proposed loss function (Eq. 5) includes the true value of y at t_n, y(t_n). They acknowledge that this may make it computationally prohibitive, but I think this point warrants further discussion. Does this mean that their loss function can only be used on problems for which we have a closed form solution (such as harmonic oscillators)? I noticed that in their van der Pol experiments, the authors switch to a more standard loss (Eq. 6). Is this because Eq. 5 is intractable in this example? Is there a reasonable approximation to Eq. 5 that could be used when a closed form solution is not known?
4. There are a number of issues with the experiments that I think could use clarification or improvement:
  (a) The authors compare to a 'baseline' but I don't believe this baseline is defined anywhere. Is it one of the adaptation methods described in Section 2?
  (b) In Table 1, the baseline method achieves lower error, while MLRK uses fewer steps. It is difficult to assess if this is an improvement since this this cost-accuracy tradeoff is at the heart of the Lagrangian formulation. Ideally, shouldn't we be able to adjust 'tol' to trace out some Pareto frontier for the cost-accuracy tradeoff? In this case, wouldn't we hope for MLRK to be able to achieve better accuracy given the same computational budget?
  (c) In the van der Pol experiments, the authors switch to local L1 loss (Eq. 6). Was the intention to experiment with local L1 loss and van der Pol provided an interesting class of examples? Or was the reasoning that Eq. 5 is intractable for van der Pol so they had to use local L1 loss? If Eq. 5 can still be evaluated on these experiments, this might be a more convincing comparison.
  (d) The number of steps required was provided for the harmonic oscillator experiments, but not the van der Pol ones. This would be helpful for comparing the methods.
  (e) What is the computational overhead of running an RNN alongside your solver? Although it doesn't tell the whole story, it would be informative to report wall clock time along with the number of steps required for each method.


In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e., the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for 2-5 players on the self-play part of the Hanabi challenge. Our ablations show the contributions of SAD compared with the best practice components.

This paper introduces a novel exploitation of the centralized training for decentralized execution regime for Dec-POMDPs with publicly/commonly known actions.  In particular, the paper augments independent value-based reinforcement learning by allowing each agent to announce the action it would have taken, had it acted greedily. This relaxation is consistent with decentralized execution, at which time agents always act greedily and no counterfactual announcement is required. The paper demonstrates the utility of this trick in Hanabi, where it achieves strong performance when combined with distributed Q-learning and an auxiliary task. The paper is largely well-written.
I think that the trick introduced in this paper is a valuable contribution to the community. As the paper discusses, SAD is simpler and easier to implement than BAD, the algorithm with the most comparable ambitions with respect to scalability. Also, unlike BAD, it is model-free. I think that investigating lightweight alternatives to common/public knowledge based approaches, such as SAD, is an important research direction.
That being said, I have some issues with the presentation of the content in the paper.
1. Section 4 is problematic. The entirety of the section is based on the assumption that agent a’ observes the Markov state. The paper claims that this assumption is made for simplicity: “where for simplicity we have assumed that agent a’ uses a feed-forward policy and observes the Markov state of the environment.”
    a. First, this assumption is unmet, with few exceptions. Other agents do
        NOT in general observe the Markov state. This is a very significant
        part of why Dec-POMDPs are so difficult. Justifying SAD in a setting
        that so radically departs from its use case is not informative.
    b. Second, the claim made by the paper (that the assumption is made
        for simplicity) is misleading. The assumption is made out of
        necessity. The cost of neglecting public/common knowledge, as SAD
        does, is that principled Bayesian reasoning is not possible. It is
        important that the paper acknowledges this to counterbalance its
        (valid) criticisms that public/common knowledge based approaches
        are difficult to scale.
2. That the experiments are expensive is understandable and a fully complete ablation study is not expected. But the results of these experiments should be presented clearly.
    a. In both Table 1 and Table 2, presenting the “best seed” is
        not appropriate. The paper should report the mean or the median.
        Given that only three seeds could be afforded, it is understandable
        that the results would have high variance.
    b. In Table 2, (if I am understanding the paper correctly) it is claimed
        that the s.e.m. is less than or equal to 0.01 over a set of three seeds
        for all of the experiments. But looking at Figure 4, this is clearly not
        the case, especially in the four and five player settings. The intended
        meaning should be clarified.
    c. In Table 1, the paper should report the results for the baseline, VDN,
        VDN with greedy input, and SAD separately, as is done in Table 2.
        These are the main experimental results and merit space in the main
        body of the paper. Moreover, there are a number of results that
        deserve written attention.
        i. It looks the baseline is very competitive with BAD in two player
           Hanabi. This in itself is a very interesting finding and is worth
           discussing.
        ii. Looking at Table 2, it appears that VDN with GreedyInput
            outperforms SAD in three of the four settings and quite
            significantly in the five player setting. If this is also the case when
            results are aggregated over the median or mean, it should be
            discussed.
3. The greedy input approach is specific to Dec-POMDPs in which actions are publicly/commonly available. This should at least be mentioned somewhere.
Overall, I think the ideas and experimental findings in the paper are very interesting. However, as outlined above, there are a number of issues. At a high level, I think that the paper is too concerned with 1) justifying greedy input with Bayesian reasoning and 2) promoting state of the art results. The ideas and experimental findings are more than sufficient for strong contribution without these things.
In its current state, I feel that this work is a rejection. However, its issues are relatively easily amendable:
1. For each algorithm and each setting, separately report median or mean results over the seeds, with uncertainty.
2. Reallocate space to section 6.2 for a scientific discussion of the experimental results.
3. Remove or qualify the arguments made in section 4.
4. Mention that the greedy input approach is specific to Dec-POMDPs with publicly/commonly available actions.
Were these issues addressed, my opinion would change.


Designing of search space is a critical problem for neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms. This search space facilitates direct selection of channel numbers and kernel sizes in convolutions. In addition, we propose a resource-aware architecture search algorithm which dynamically selects atomic blocks during training. The algorithm is further accelerated by a dynamic network shrinkage technique. Instead of a search-and-retrain two-stage paradigm, our method can simultaneously search and train the target architecture in an end-to-end manner. Our method achieves state-of-the-art performance under several FLOPs configurations on ImageNet with a negligible searching cost.

[Summary] This paper proposes a channel-wise neural architecture search (NAS) approach. The NAS search algorithm is similar to previous one-shot NAS, but the search space is channel-wise: each channel has it’s own kernel size, which is quite novel and interesting. Results are strong in terms of FLOPS and parameters.
[High-level comments]:
1. I like the novel idea of channel-wise search space, which provides great flexibility for NAS.  Although some recent works (e.g., MixNet) have tried to partition channels into groups, this paper goes further and searches for different kernel size for each single channel.  With this channel-wise search space, it naturally enables a combination of per-channel kernel size selection and per-channel pruning, leading to strong results in terms of FLOPS and parameters, as shown in Figure 4 and Table 1.
2. In general, channel-wise NAS is difficult as different channels are often coupled in various ways. However, the authors observe that recent NAS (such as MnasNet/MixNet/SCARLET-A/EfficientNet) are mostly based on a common MB pattern. By targeting to this specific pattern and applying some additional constraints (e.g. fixed input/output channel size and fixed number of layers per stage as shown in Figure 3), the authors successfully make the channel-wise NAS work well.  I appreciate the authors efforts, but I am also a little concerned that the proposed approach might be limited to this specific small search space.
[Questions and suggestions to authors]:
3.  How do you justify the generality of the channel-wise search space?  For example, is it possible to also search for input/output channel size (column f in Figure 3) and #layers per stage (column n in Figure 3)? Adding some discussions for this would be very helpful.
4. The title seems too broad. I recommend the authors including “channel-wise” in the title.
5. Please provide some justifications on how to set λ  and c_i in Equation (5).
6. When you say “expands the input channel number from C to 3 × 6C”, what does “3x6C” mean? Is it 18C? What’s the reason for choosing this specific value?
7. Could you show the accuracy and complexity (either FLOPS or params) of the supernet during the training? This information would be helpful to interpret and justify your algorithm 1.
8. The network architecture in Figure 5 is vague. Could you provide the network source code or frozen graph for this specific model?


We develop a normative theory of hierarchical model-based policy optimization for Markov decision processes resulting in a full-depth, full-width policy iteration algorithm. This method performs policy updates which integrate reward information over all states at all horizons simultaneously thus sequentially maximizing the expected reward obtained per algorithmic iteration. Effectively, policy path programming ascends the expected cumulative reward gradient in the space of policies defined over all state-space paths. An exact formula is derived which finitely parametrizes these path gradients in terms of action preferences. Policy path gradients can be directly computed using an internal model thus obviating the need to sample paths in order to optimize in depth. They are quadratic in successor representation entries and afford natural generalizations to higher-order gradient techniques. In simulations, it is shown that intuitive hierarchical reasoning is emergent within the associated policy optimization dynamics.

This paper presents a reinforcement learning method that exploits the full-depth backup. The policy update based on the full-depth backup is derived for entropy-regularized MDP. The state-action correlation function is introduced and the Fisher information matrix is computed with it. The proposed method is evaluated on tasks with discrete states and actions.
I understand the concept of using the full path for updating the policy, but I do not see significant novelty of the proposed method from the current manuscript. The proposed method looks equivalent to the natural policy gradient with full-depth backup for entropy-regularized MDP, which is a special case of existing methods.
My concern is that the scalability of the proposed method. The use of the full-depth backup should suffer from the large variance, and I think the proposed method will not work on tasks with the high-dimensional state space.
The evaluation is limited to tasks of which the state space is small, and the proposed method is not compared with existing methods.
Due to the unclear novelty and limited empirical results, I give weak reject to the paper in the current form.
I request authors to answer the following questions to improve the clarity.
- Is the proposed method equivalent to use the natural policy gradient with the full-depth backup for a softmax energy-based policy? If they are different, what it the crucial difference?
- I think that the variance of the estimation of the gradient is large when using the full-depth back up. I'm curious about the performance of the proposed method in high-dimensional tasks. However, the evaluation is very limited to simple tasks in which the state space is relatively small compared with tasks commonly used in deep RL papers.
Does the proposed method scale to more complex tasks, such as Atari games?
- When using the n-step TD learning, increasing n does not always improve the performance, and n should be set to an intermediate value
What is the motivation of using the full paths for updating the policy? Does the proposed method outperform existing methods? Especially, the comparison with natural policy gradient methods is necessary to show the benefit of the proposed algorithm.
Minor comments:
- In page 2, "A state-space \mathcal{X} is composed of states x \in \mathcal{X}" <- Authors may want to replace x with s in this sentence.
 - In page 6, "0\lambda < 1" I think that "<" is missing between "0" and "\lambda"
- In page 6, some variables are explained after Equation (10). However, it took me a while to find "lambda" in Equation (10), since Equation (10) has 9 lines and many terms. I think the description in page 6 can be improved. For example, I recommend to use "\exp" instead of "e^" for readability.


Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of the current imitation learning methods fail because they focus on imitating actions. We propose a novel state alignment based imitation learning method to train the imitator to follow the state sequences in expert demonstrations as much as possible. The state alignment comes from both local and global perspectives and we combine them into a reinforcement learning framework by a regularized policy update objective. We show the superiority of our method on standard imitation learning settings and imitation learning settings where the expert and imitator have different dynamics models.

Summary of claims:
The paper proposes an imitation learning method that aims to align state distributions rather than state-action distributions to account for cases where the imitator dynamics differ from expert dynamics. They achieve this by two objectives: one local, the other global. The local objective aligns the next state to be close to the expert's next state in each transition by first training a VAE on the expert demonstrations, and using the trained VAE in conjunction with a pretrained inverse dynamics model to compute the action that the imitator needs to imitate. The global objective tries to do a global alignment of states encountered in the imitator and expert trajectories, by minimizing the Wasserstein distance between the two trajectory distributions. The paper claims that using these two objectives results in a method that outperforms existing inverse reinforcement learning and behavior cloning approaches in settings where the imitator and expert dynamics differ.
Decision:
I recommend the paper to be rejected. I have three main reasons for my decision (with more details in the next section):
1. The paper is very poorly written : A lot of details are missing in the paper, notation is not standardized,  related work is just a list of previous papers without any context on how the proposed method is related, previous methods are referred to without any citations, and quite a few blanket statements which are not substantiated.
2. Incomplete approach description: Quite a few components of the approach are not explained (or even discussed), no intuition provided for the choices made in the approach, the concept of different dynamics is not formalized, some technical inconsistencies in the algorithm, no formal problem statement (which would really help in standardizing notation), and most claims made about the approach are not justified or substantiated
3. Poor experiments: Experiments are not well chosen to reflect the premise and claims of the paper, little to no details given for how the baseline approaches were trained, no details on policy parameterization, and missing comparison with baseline approaches in some experiments
Comments:
(1) Problem setup:
(a)Problem setup is very vague and not formalized.
(b) Differing dynamics could mean several things: different agent dynamics (like different action spaces; different actuators; etc.), different environment dynamics (different moving obstacles in the world;) etc.
(c) Basically, different dynamics can mean a lot more than what was accounted for in the paper
(2) Blanket Statements:
(a) The authors keep saying that their framework is more flexible without any justification as to why,
(b) "simply train an inverse dynamics model"- training an inverse dynamics model can be very hard especially when environment dynamics are stochastic/when the inverse model is multimodal,
(c) "constraint becomes loosened" - this statement doesn't make any sense without more explanation,
(d) Several other blanket statements about existing approaches
(3) Notation :
(a) Notation was never standardized in the paper,
(b) what is \phi? what is the input-output of \phi?
(c) What is \theta_old? What is \sigma?
(d) There are a lot of things that needed explanation, especially in the algorithm
(4) Related Work :
(a) The related work section is just a dump of citations without giving any context for where the proposed work lies in the spectrum of these works. How does it compare? Why is it better/worse?
(b) Missing related work that was publshed in ICML 2019 that has a very similar approach in matching state distributions ("Provably efficient Imitation Learning from Observations Alone" or FAIL) and works very well.
(5) Motivation :
(a) The approach, in general, needs better motivation. The toy example in the introduction was good but the experiments did not reflect the complexity of that example.
(b) Try to have a running example in the paper that will help you motivate the approach better.
(6) Background :
(a) The background section is very minimal and lacks any details necessary.
(b) I had to read the beta-VAE paper to understand what it does.
(c) The section also lacks any minimal background in IL/RL and the notation could also have been standardized in this section
(7) Figures:
(a) All the figures in this paper could use a lot of improvement in terms of descriptive captions, more informative legends, bigger fonts, descriptive text, and more figures as well
(8) Algorithm :
(a) The algorithm was not referenced anywhere in the text,
(b) No definition for \tau, details on pretraining inverse dynamics model lacking in both text and algorithm,
(c) *Policy prior is used to pretrain policy before the VAE was trained!* which doesn't make any sense since the policy prior is obtained using the VAE,
(d) the equation at the end of Sec 4.3 has r(s, a) whereas reward is defined as r(s_t, s_{t+1}) but they are not equivalent when dynamics are stochastic
(9) Experiments:
(a) What is AIRL? No reference was given.
(b) Why is keeping the variance of the policy constant reasonable? How do you come up with the value?
(c) How do you pretrain the VAE, invserse dynamics model?
(d) The setup of making the ant's legs smaller or body heavier seems very artificial. I am sure its easy to come up with more realistic setups in navigation domains, for example. Try to use more realistic experiments in the future.
(e) For results in Fig 3, is AIRL, GAIL also pretrained with VAE or by BC? Seems like SAIL was pretrained but the others weren't since SAIL starts off with a high score at the start.
(f) For Sec 5.1.2, comparison with baseline approaches are missing. Legends for the plots are terribly small.
Conceptual questions:
1. How do you account for cases where due to differing dynamics, states reached by expert in demonstrations are unreachable by the imitator?
2. How do you account for cases where the environment dynamics changes between expert and imitator?
3. Why does using Wasserstein distance make sense? Why not other f-divergences? Also, matching global distributions can be very misleading if you have states that are visited multiple times in the same trajectory. FAIL recommends matching state distribution at each time-step instead and is much more stable
4. How is this different from GAIL where we match state visitation distribution (instead of state-action visitation distribution?)
Things to improve:
1. Writing needs to be improved a lot
2. Better experiments - more realistic domains
3. Approach needs to be explained more formally