Continually learning new skills without forgetting old ones is an important quality for an intelligent system, yet most deep learning methods suffer from catastrophic forgetting of the past. Recent works have addressed this by regularising the network weights, but it is challenging to identify weights crucial to avoid forgetting. A better approach is to directly regularise the network outputs at past inputs, e.g., by using Gaussian processes (GPs), but this is usually computationally challenging. In this paper, we propose a scalable functional-regularisation approach where we regularise only over a few memorable past examples that are crucial to avoid forgetting. Our key idea is to use a GP formulation of deep networks, enabling us to both identify the memorable past and regularise over them. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation methods are naturally combined with memory-based methods.

Summary: The paper uses a Gaussian Processes framework previously introduced in [1] to identify the most important samples from the past for functional regularization. For evaluation authors report their average accuracy on Permuted MNIST, Split-MNIST, and CIFAR10-100 and achieve superior performance over EWC, DLP, SI, VCL-Coreset, and FRCL.
Pros:
(+): The paper is well-written, addressed the prior work quite well despite missing a few important work from the past (more on this later)
(+): The paper is well motivated
Cons that significantly affected my score and resulted in rejecting the paper are as follows:
1- lack of support for “scalability”:
Authors claim their method is scalable in several parts of the paper (abstract in line 7, Section 3 in the 1st paragraph, and Section 5 in Discussion). However, this claim is not supported in the experimental setting as the benchmark used are only toy datasets (Permuted MNIST, Split MNIST, and CIFAR10 followed by CIFAR100) where the maximum # of task considered is 10 and the maximum size of the datasets is 60K which is not convincing for ability to scale. There is also no time complexity provided.
2- Incremental novelty over the prior work (FRCL by Titsias et al 2019):
This baseline is the closest prior work to this work which according to the experiments shown in Table 2 are slightly outperformed by the proposed method. (for example for P-MNIST the gain is 0.6%+-0.1) where there is a lack of complete discussion on how the two methods are different. Particularly I suggest that the authors elaborate more on their claimed differences stated on page 4, paragraph 5 such as “tractability of the objective function only when we assume independence across tasks”. Do authors mean assuming clear task boundary between tasks? If so, have they considered a “no-task” or an "overlapping” task boundary in their experiment? Isn't it necessary to back up this if it is stated as a shortcoming of FRCL? Also, how are these methods differ in their computational expenses?
3- Lack of measuring forgetting:
This is the most important drawback in the experimental setting. Authors indicate on page 3 “Our goal in this paper is to design methods that can avoid such catastrophic forgetting.” and reiterate on this on other parts of the paper yet there is no forgetting evaluation to support this claim. Authors can simply report the initial performance of the model on each task so that readers can compare it with the reported accuracy after being done with all tasks. Having a method with high average accuracy does not necessarily mean it has minimum forgetting. You can use forgetting measurements such as Backward Transfer (BWT) introduced in [1] or forgetting ratio defined in [4] for this assessment.
4- Ambiguous claims about prior work:
(a) On page 1, paragraph 3, when authors mention that methods such as GEM or iCaRL use random selection to pick previous samples, I think the line of follow-up work on these methods should be mentioned as well that have explored different techniques for sample selection and have provided benchmark comparisons (ex. [2,3]). In fact it would be beneficial if authors could compare the samples selected by their method versus other sampling techniques.
(b) On page 1, paragraph 3, they mention some prior work such as GEM and iCaRL “do not take uncertainty of the output into account”. While it is true, there have been methods proposed that use uncertainty of the output for parameter regularization [5]. It appears to be a parallel work to this but it’s worth mentioning to prevent false claims.
5- Claim on the state of the art should be double-checked:
	Although the results shown for the experiments are superior to the provided baselines, there is an important baseline missing which has achieved higher performance than the reported ones. Also missed to be cited in the prior work list. Serra et al [4] proposed a method at ICML 2018 called HAT, which is a regularization technique with no memory usage that learns an attention mask over parameters and was shown to be very effective on small and long sequence of significantly different tasks. They do not use samples from previous task but yet achieved good average ACC as well as minimum forgetting ratio. Note that 5-Split MNIST is not reported in [4], but a recent work has reported HAT’s performance on this dataset (https://openreview.net/forum?id=HklUCCVKDB) that achieves 99.59%. I recommend authors provide comparison of their own on the given benchmarks with the original HAT’s implementation (https://github.com/joansj/hat) before claiming to be SoTA. In my opinion, it is not an issue if a novel method achieves a slightly lower performance to the sota because I think it still adds value and proposes a new direction. However, a false claim should not be stated.
Less major (only to help, and not necessarily part of my decision assessment):
1- Providing upper bound?
It is common to show an upper bound for any continual learning algorithm by showing joint training performance which is considered to be the maximum achievable performance. I also recommend showing the naive baseline of fine-tuning for the proposed method  which often can give insight to maximum forgetting ratio.
2- Forward transfer?
Regularization techniques combined with memory might have an ability to perform zero-shot transfer or so called FWT. I recommend authors provide such metric to further support their method.
3- Hyper parameter tuning?
It is also worth mentioning how the tuning process was performed. In continual learning we cannot assume that we have access to all tasks' data, hence authors might want to shed some light on this.
References:
[1] Khan, Mohammad Emtiyaz, et al. "Approximate Inference Turns Deep Networks into Gaussian Processes." arXiv preprint arXiv:1906.01930 (2019).
[2] Chaudhry, Arslan, et al. "Continual Learning with Tiny Episodic Memories." arXiv preprint arXiv:1902.10486 (2019). (https://arxiv.org/abs/1902.10486)
[3] Aljundi, Rahaf, et al. "Gradient based sample selection for online continual learning." arXiv preprint arXiv:1903.08671 (2019). (https://arxiv.org/abs/1903.08671)
[4] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557
[5] Ebrahimi, Sayna, et al. "Uncertainty-guided Continual Learning with Bayesian Neural Networks." arXiv preprint arXiv:1906.02425 (2019).
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
POST-REBUTTAL Response from R1:
Thank you for taking the time and replying to comments. Here are my responses to authors' replies:
[Authors' response:] 1. Scalability: Our algorithm only adds a small computational overhead on top of Adam on a standard neural network. This is what we mean by scalable. The additional complexity scales cubically in M, the coreset size. This is due to the inversion of the kernel in fr_grad. Another overhead is the computation of Jacobian which is order PKM, where K is the dimensionality of the output and P is the number of network parameters. Both of these additional costs are small for small coreset sizes M. We will add these details to make these points clear in the paper.
[Reviewer's response:] I still insist on the fact that simply explaining the overhead of a method is not a support for scalability claim versus showing the performance on a large scale dataset and comparing it with other CL methods that also have high scalability given the fact that authors only use MNIST and CIFAR datasets.
[Authors' response:] 4. Prior work: We discuss other works in Section 1 (“two separate methods are usually used for regularisation and memory-building”), and we will expand upon this sentence, going into more detail, and also referencing iCaRL and other works (including [3]). Note that our method of choosing a memorable past follows directly from the theory in Section 3.1, and is achieved with a single forward-pass through the trained network (as mentioned in the paper). Other techniques for sample selection do not integrate so naturally with the framework, and are not as straightforward to understand or implement either.
[Reviewer's response:] I disagree with authors on this because GEM, its faster version (A-GEM (Chaudhry et al. 2018)), and all other methods explored in the recent study which I mentioned in my review (Ref#2) use the single epoch protocol and are perfect match to be compared with this method but there is no memory-based baseline except for VCL with coreset and FRCL (only for MNIST variations) which makes it difficult to measure this method's capabilities (performance, memory size, and computational time) against methods which only require one epoch to be trained.
Authors have provided FWT for their method as 6% which is unbelievably large for this metric (see GEM paper) and hence does not make sense to me. Please double check whether you computed this value right.
While I accept the response for the remaining questions from authors but I am still concerned about the weak experiments and an issue brought up by R3 regarding lack of enough comparisons with FRCL on any other datasets besides split MNIST and P-MNIST. Also in  CIFAR experiment, what is the architecture used across the baselines? More importantly in results reported for VCL on CIFAR, it is not clear to me how authors obtained this results. Did they use a conv net? VCL was originally shown on MLPs only and it is one of the downside of this method that was never shown to be working in convolutional networks. Therefor, it is important to mention how they are obtained. This might explain the reason for the huge forgetting reported for VCL with coreset (−9.2 ± 1.8) as opposed to −2.3 ± 1.4 for EWC which is really strange as VCL even without coreset (on permuted mnist for example) is reported superior to EWC by a large margin (6%) in the original VCL paper. Overall I am concerned about the experimental setup and some of the reported results and hence intend to keep my score.


We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new openvocabulary language modelling benchmark derived from books, PG-19.

## Updated review
I have read the rebuttal. First I'd like to thank the authors for the detailled rebuttal.
The latest version of the paper adressed all my concerns, hence I change my rating to Accept.
## Original review
This paper presents a new variation of the Transformer model, named Compressive Transformer. The key novelty of this model is to preserve long range memory in a compressed form, instead of discarding them as previous models have done. This improves the long-range dependencies modelling capabilities of the approach. The model is evaluated on two common language modelling benchmarks and yields state of the art results in both of them. The paper also introduces a new benchmark for long-range dependencies modelling composed of thousands of books. The paper finally presents an analysis of the compressed memory and provide some insights, including the fact that the attention model uses the compressed memory. The model is also evaluated on two other tasks: speech generation and reinforcement learning on videos.
I think this paper should be accepted, mainly because:
- The proposed model is novel as far as I can tell.
- The presented approach is significant, as modelling long-range dependencies is an important milestone in sequence modelling.
- The new benchmark is a good addition.
- The comparison with the relevant literature is thorough and well done.
- The experiments are convincing and demonstrate the viability of the approach, although some aspects can be improved (see below).
Detailed comments:
- About the character-level language modelling on Enwik8, the improvement is very small, it seems that the task doesn't benefit from have long-range memory, could it be because character-level modelling is less dependent on the long-range past? can the authors comment on that? It would also been interesting to evaluate the gain of the memory, for instance by varying the size of the compressed memory from 0 to 1152.
- The WikiText-103 evaluation is interesting, specially Table 6, which shows the advantages of the model. However when comparing with the literature, it's not clear if the performance gain is due to the compressed memory or to the network capacity. A study with different lengths of the compressed memory (starting at 0) would bring some insights about that.
- In Section 5.6.2: can the authors justify why the attention weights were split in only 6 bins? creating a trended curve on only 6 points could be problematic, and I don't see why more bins couldn't be used.
- The speech analysis section (5.7) is not very insightful. It shows that the proposed model is on par with WaveNet on unconstrained speech generation, which is not very useful and feels a bit half-finished. I think that the authors should either commit to this study by constraining the model with linguistic features like in (Oord et al. 2018) and evaluate it in a TTS framework with subjective evaluation or discard this section entirely.


We present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from neural networks. The HTA starts with several decoupled systems with low dimensional structure and tracks the solution to the high dimensional coupled system. The decoupled systems are easy to solve due to the low dimensionality but can be connected to the original system via a continuous homotopy path guided by the HTA. We have proved the convergence of HTA for the non-convex case and existence of the homotopy solution path for the convex case. The HTA has provided a better accuracy on several examples including VGG models on CIFAR-10. Moreover, the HTA would be combined with the dropout technique to provide an alternative way to train the neural networks.

Summary
This paper proposes an algorithm to address the issue of nonlinear optimization
in high dimensions and applies it to convolutional neural networks (VGG models) on CIFAR 10.
They show 11% relative reduction in error for this particular task with this
particular network. In addition, they prove additional theoretical results on
the convergence of SGD using their method in the convex case as well as
convergence of SGD to a stationary point in the nonconvex case when the homotopy
parameter is fixed which is not done in practice.
Given an optimization problem, their method first solves multiple independent
lower-dimensional optimization problems each with a subset of the parameters and
then optimizes a new objective function controlled by a monotonically decreasing
parameter L that interpolates the original objective function and the
previously-solved lower dimensional problems. L can be seen as a regularization
parameter that is gradually decreased as we optimize the new optimization
function. When L = 0, we recover the original optimization problem.
The authors prove that (1) SGD with their procedure will find a stationary point
under the Robbins-Monro conditions for a fixed L and (2) SGD with their
procedure will converge for convex problems as L is decreased to 0.
Decision and reasoning
This paper should be rejected because (1) the proposed algorithm attempts to address
the original issue of high dimensional nonlinear optimization of neural networks but
violates the algorithm's assumption in practice, (2) the
empirical evaluations are lacking - having only evaluated their method on a toy
problem with up to only 6 dimensions and a relatively simple image classification task,
and (3) the assumption of fixing the homotopy parameter in the theorem on the
non-convex case directly violates the intention of the algorithm.
Regarding (1): The proposed procedure requires initializing L at a large value
and reducing L towards 0 in order to recover the original optimization problem.
However, in practice for CIFAR 10, the authors initialize L to be 0.01 and
gradually reduces it to 0.005 which is hardly the original intent of the
algorithm. There is also no demonstration whether or not this gradual reduction in
L actually has an effect on the optimization of the new objective function. For
example, since the start and end values of L are similar, will we get similar
results if we simply fix L to be 0.005 or 0.01? The authors also show that their
method outperforms a quasi-newton method by combining the optimization with
their procedure on a non-convex example by Chow et al. 2013. However, this example
only goes up to n=6 dimensions, which is hardly comparable to the original
problem of high dimensional non-convex optimization that this paper sought to
address.
Regarding (2): The authors evaluated their procedure on CIFAR10, a relatively
simple image classification task that modern neural networks can solve easily
and is not representative of the types of nonlinear optimization problems
prevalent in deep learning. There's also an issue of using only VGG networks for
their evaluations while VGGs are typically eschewed in favor of ResNets today.
Given that the optimization is easier with residual connections, it may be the case that
their procedure does not significantly improve the accuracy of ResNets.
Regarding (3): By fixing L in Theorem 3.1, the authors essentially show that SGD
converges to a stationary point for their new objective function which can be
seen as a regularized version of the original objective function, which is not a
strong result. Furthermore, fixing L goes against the original procedure's
motivation of recovering the original optimization function as L decreases to 0.
Additional comments and questions
There are passages that are difficult to understand because not enough context
is given. For example in the "remark" passage, it is not clear where the
"necessary condition" comes from. In addition it seems like it doesn't even
type-check since the first term is 2n dimensional while the second term is 4n
dimensional.
There are also many errors in the writing that hinder the presentation. A subset of
them includes:
- "nerual netowrks on roboticsKonda et al." -> "neural networks on robotics Konda et al."
- "based on homotopy continuation method" -> "based on the homotopy continuation methods"
- "random chosen point" -> "randomly chosen point"
- "we choose \tilde{\theta} = 0 in the dropout" -> reword
- Fourth term in Equation 3 should be \theta_2 - \tilde{\theta_2}
- "By gradually increasing parameter L" -> "By gradually decreasing parameter L"
- "where \xi is a random variable due to random algorithms" -> reword and possibly say the randomness is from SGD
- After equation 6, should have b_i instead of \beta_i
- In equation 20, should be g(\theta_*^0) instead of g(\theta_*^1)
- In theorem 3.2 you never explained what \theta_*^{L_k} is
- "We compared the traditional optimization method (the quasi-Newton method)" -> which quasi-Newton method?
- Figures 2 and 3 label the x-axis with "epochs". However only 4 epochs were run, so I believe the x-axis should be "iterations""
Besides improving the quality of writing in the paper, I would strongly suggest that the
authors improve their empirical evaluation.  Possibilities include evaluating on
CIFAR 100 or ImageNet, using a wider variety of networks including ResNets,
evaluating on tasks other than image classification.


Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning. The advent of distributed agents that interact with parallel instances of the environment has enabled larger scale and greater flexibility, but has not removed the need to tune or tailor exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning. We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress. The data distribution is controlled via modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without significant overhead. The adaptation speed of the bandit can be increased by exploiting the factored modulation structure. We demonstrate on a suite of Atari 2600 games how this unified approach produces results comparable to per-task tuning at a fraction of the cost.

This papers studies how to explore, in order to generate experience for faster learning of policies in context of RL. RL methods typically employ simple hand-tuned exploration schedules (such as epsilon greedy exploration, and changing the epsilon as training proceeds). This paper proposes a scheme for learning this schedule. The paper does this by modeling this as a non-stationary multi-arm bandit problem. Different exploration settings (tuple of choice of exploration, and the exact hyper-parameter), are considered as different non-stationary multi-arm bandits (while also employing some factorization) and expected returns are maintained over training. Arm (exploration strategy and hyper-parameter) is picked according to the return. The paper demonstrates results on the Atari suite of RL benchmarks, and shows results that demonstrate that their proposed search leads to faster learning.
Strength:
1. The paper tackles an interesting and important problem. The proposed solution is simple, yet effective.
Shortcomings:
1. The presentation is somewhat convoluted. The paper motivates the problem that we need to pick out an exploration sequence that optimizes learning progress, but then approximates it as simply measuring the return. Given there is no theoretical justification for the approximation, I believe the paper claims more than what it delivers and should change the presentation, so as not to claim that it is measuring and capturing learning progress to learn faster.
2. I am confused by Figure 4, and in general with the relative rank metrics. Specifically, in Figure 4, is it that the proposed bandit approach not as good as picking a single hyper-parameter for the different settings (T=0.01, eps=0.01, omega=2.0)? Similarly, for Figure 2, a singe fixed z, seems to do better than the bandit versions. Why doesn't the proposed bandit algorithm not pick out the best hyper-parameter? How well would a simpler hyper-parameter search procedure (picking the best hyper-parameter after the first 2000 episodes)?
3. This apart, I think that the experiment section is pretty hard to read, given all the metrics and methodology is in the Appendix. An alternate organization that presents all the main results in the main body in a self-contained manner will help.
4. Comparison with past works. I believe there are other existing works that should be cited and compared to. Using bandits to decide between different hyper-parameters is common (for example, see [A] for a service to do this with ML models), [B] uses improvements in accuracy as a way to pick between which question type to train on. Such past works should be cited and compared against.
[A] https://ai.google/research/pubs/pub46180
[B] Learning by Asking Questions
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta and Laurens van der Maaten


Current Deep Reinforcement Learning (DRL) methods can exhibit both data inefficiency and brittleness, which seem to indicate that they generalize poorly. In this work, we experimentally analyze this issue through the lens of memorization, and show that it can be observed directly during training. More precisely, we find that Deep Neural Networks (DNNs) trained with supervised tasks on trajectories capture temporal structure well, but DNNs trained with TD(0) methods struggle to do so, while using TD(λ) targets leads to better generalization.

Summary:
This paper performs an empirical evaluation of generalization by TD methods with neural nets as function approximators. To quantify generalization, the paper considers the change in the loss function at similar states to the one where the update rule is being applied (where “similar” is usually defined as nearby in time). It comes to a variety of conclusions including that TD(0) does not induce much generalization, that TD(0) does not induce as much generalization as supervised learning, and that the choice of optimizer and objective changes the behavior according to their generalization criteria in various ways.
Decision:
This paper should be rejected because (1) the motivation is unsubstantiated, (2) the main metrics used (“gain” at nearby states and difference since last visit) are of questionable importance, and (3) the conclusions are often vague and not informative.
Main argument:
Motivation:
- The paper motivates the need for an evaluation of “gradient update generalization” by claiming that it is related to sample complexity and brittleness of RL algorithms. While I agree that this is plausible, there is nothing empirical or theoretical to support this claim in the paper or in the references. This is a significant problem since this assumed connection underlies everything in the paper.
- Also, it is this sort of generalization that differentiates the function approximation setting (where there are no convergence guarantees for TD without strong assumptions) from the tabular setting (where there are convergence and sample complexity guarantees).
Metrics:
- The main metric used is TD gain on temporally nearby states. The TD gain is defined as the reduction in the squared TD error at some state s’ when an update is applied at some other state s. Note this metric does not capture all update generalization, but only update generalization as it effects the TD error.
- It is not evident, nor supported by the paper, that improvements in this metric at nearby states necessarily improve performance of the algorithm. This is especially true when there is a tradeoff between improvement at very nearby vs. somewhat nearby states, it is not clear which behavior is preferable (and this behavior seems to occur in experiments). As a result there is no clear way to use this metric to determine which algorithms are preferable.
- The other metric used in the paper is the change in value function at a state since the last time that state was sampled from the buffer. It is also not clear whether this measurement is necessarily important for the same reasons as above.
Conclusions:
- In general the results are presented in plots that do not give clear implications and are difficult to read. I understand that we cannot expect completely clean results on such empirical questions, but the results would be potentially much more convincing and clear if the hypotheses were clearly stated and then one plot could summarize the result with respect to each hypothesis. For example, the gain plots are difficult to compare and interpret clearly and the memorization plots do not give such clear results (eg. 3a and 3d look visually fairly similar). Another example is the comparisons between optimizers, which are fine to have as a specific point in one section, but do not need to be in every plot.
- The main result claimed in the paper is that there is little generalization from TD(0) when compared to supervised learning. This seems to be born out by the difference between figure 1a and 2a, but the difference in scale makes it a bit difficult. It is also not clear how to quantify this result or what the implications are.
- The plots are averaged across all states over all of training and all environments, while this is somewhat rationalized in figure 11, I am worried that this may be covering some additional complexity/ambiguity in the results.
- The result about TD(lambda) seems to be born out by figures 3e and f and 5, but is also unsurprising since the objective explicitly averages across temporally nearby states. Again it is not clear how this temporal consistency of updates should be interpreted in terms of the goals of the RL algorithms.
At a higher level, the paper feels like a solid preliminary set of experiments rather than a paper organized around a clear motivating idea with clear hypotheses to test. The results would become more interesting if the metrics used could be connected back (either empirically or theoretically) to an objective. For example, does generalization in the sense defined in the paper give better performance at value estimation or return maximization? Can these results be quantified in a more direct way than the plots presented in the paper?
Additional feedback:
- The plots all have different scales which makes them difficult to compare
- Using “distance” to refer to the relative tilmestep of samples in the replay buffer is confusing (distance cannot be negative)
- There are strange visual artifacts (horizontal lines) in Figures 3, 5, 10, 12, 13, and 14
- Sections 3.7 and 3.8 very briefly present results that seem to distract from the main thread of the paper
- Using one network architecture across all experiments seems like it may have a significant impact on the results. I understand that the architecture chosen is standard and testing different architectures is time consuming, but making broad claims about the algorithms based on only one architecture is potentially dangerous.
- I assume that the replay buffer is always being sampled uniformly, but I could not find this detail in the paper.


We present the ADaptive Adversarial Imitation Learning (ADAIL) algorithm for learning adaptive policies that can be transferred between environments of varying dynamics, by imitating a small number of demonstrations collected from a single source domain. This is an important problem in robotic learning because in real world scenarios 1) reward functions are hard to obtain, 2) learned policies from one domain are difficult to deploy in another due to varying source to target domain statistics, 3) collecting expert demonstrations in multiple environments where the dynamics are known and controlled is often infeasible. We address these constraints by building upon recent advances in adversarial imitation learning; we condition our policy on a learned dynamics embedding and we employ a domain-adversarial loss to learn a dynamics-invariant discriminator. The effectiveness of our method is demonstrated on simulated control tasks with varying environment dynamics and the learned adaptive agent outperforms several recent baselines.

Summary:
The submission considers the problem of imitation learning when the dynamics of the expert are not known to the agent and the dynamics of the agent may change frequently. It is however assumed that the agent has access to a parameterized simulator that can simulate the expert dynamics. The parameters for the simulator are not known but are assumed to be drawn from a known distribution.
The proposed method is based on GAIL but uses several modifications:
- A contextual policy is trained that also takes the dynamics-parameters as additional input. At each iteration, a new environment is sampled for performing the policy-rollout.
- A "posterior" prediction network is trained to maximize the likelihood of the parameters that were used for the different roll-outs. This network is used for the test case, where the true dynamics of the agent are not known.
- The discriminator might use features of the state-action input that correlate with the corresponding dynamics. Classifying based on such features may be undesirable because the discriminator might no longer produce useful rewards. In order to address this problem, an additional head is added to the discriminator that outputs a prediction of the dynamic parameters. The prediction error is trained by backpropagation, however by flipping the sign of the gradient at the last shared layer, the features of the discriminator are optimized to be unsuited for predicting the dynamic parameters (the technique is known as Gradient Reversal Layer).
- A VAE-based method for learning latent dynamic parameters is proposed, by training a conditional VAE to reconstruct the next state, where the current state and action are provided as context to the encoder and decoder.
Contribution:
One of the strong-points of the submission is the fact that it features several different, orthogonal contributions. I also think that the considered problem setting is relatively interesting. However, also when considering real applications such as robotics, I am not convinced that explicitly modeling the dynamic changes is necessary. Some existing imitation learning methods focus on a setting where the dynamics of the expert may differ from the agent, but the dynamics of agents do not change. This setting does not require dynamic-contextual policies and seems to be applicable to typical robot applications.
Soundness:
The different components of the proposed methods seem reasonable to me. They do not come with (and arguably do not require) new derivations but seem rather like pragmatic solutions for the encountered problem.
The optimization problem (Eq.2) seems to be formulated slightly inaccurate, because the last term should in my opinion not depend on theta. If I understand correctly, the policy should not maximize the likelihood of the dynamics posterior.
The contrastive regularization loss needs to be better motivated. A high KL in the first term may not necessary be bad, for example, if the confidence in the prediction of (s_0, a_0, s'_0) is lower compared to (s_1, a_1, s'_1). If a similar regularizer has been used in prior work, such work should be referenced. Otherwise, it needs to be motivated.
Presentation/Clarity:
The presentation of the work is arguably the main weakness of the paper.
The submission does not seem polished. It contains a large amount of typos. Figure 2 is confusing and adds little compared to the text description. Also the structure could be improved. For example, the submission introduces the posterior loss and outlines the algorithm before describing the individual components.
The paper uses some techniques such as conditional VAEs [1] or contextual policy search [2]
are used but not described / referenced.
Evaluation:
I like that the different aspects of the proposed method are also evaluated individually. The ablations with respect to the adaptability and GRL are crucial. The evaluation of the performance could be improved. PPO and UP-True use the true reward function, so the only real competitor is a naive GAIL-baseline that uses randomized dynamics during training. I'm not aware of prior work that considers the exact same setting as the manuscript. However, one of the main arguments for inverse reinforcement learning is the claimed generalizability of a reward function as opposed to a policy. I see that learning a new policy after each change in the dynamics may be too costly in some settings. However, comparisons to methods such as AIRL that aim to learn reward functions that are robust to changes in the dynamics would be highly interesting.
[1] Sohn Kihyuk, Honglak Lee, and Xinchen Yan. “Learning Structured Output Representation using Deep Conditional Generative Models.” Advances in Neural Information Processing Systems. 2015.
[2] Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters.  A survey on policy search forrobotics.Foundations and Trends in Robotics, 2(1–2):328–373, 2013.
Comparison: How about IRL, e.g. AIRL?
What about state-only GAIL?
Typos:
Equations are not properly integrated into the sentences (missing punctuations)
"domains, as oppose to one domain."
"Inspired by to GANs"
"and generates a environment"
Figure 5a (legend): "dyanmics"
"that can generalized across"
Algorithmbox: "A environment", "and Generate environment"
"is achieved through 1) allowing"
"the policy is mainly concerned with the end-effector of the latent parameters"
Question:
What are the network architectures?
According to line 10 of the algorithmbox only the current trajectory is used for updating the dynamics posterior. Why?


We propose to use a meta-learning objective that maximizes the speed of transfer on a modified distribution to learn how to modularize acquired knowledge. In particular, we focus on how to factor a joint distribution into appropriate conditionals, consistent with the causal directions. We explain when this can work, using the assumption that the changes in distributions are localized (e.g. to one of the marginals, for example due to an intervention on one of the variables). We prove that under this assumption of localized changes in causal mechanisms, the correct causal graph will tend to have only a few of its parameters with non-zero gradient, i.e. that need to be adapted (those of the modified variables). We argue and observe experimentally that this leads to faster adaptation, and use this property to define a meta-learning surrogate score which, in addition to a continuous parametrization of graphs, would favour correct causal graphs. Finally, motivated by the AI agent point of view (e.g. of a robot discovering its environment autonomously), we consider how the same objective can discover the causal variables themselves, as a transformation of observed low-level variables with no causal meaning. Experiments in the two-variable case validate the proposed ideas and theoretical results.

The paper proposes a method of discovering causal mechanisms through meta-learning, by assuming that models transfer faster if their causal graph is correct.
For a possible causal graph, for adaptation to one interventional dataset, the samples are iteratively revealed and the log likelihood of the next sample is measured, after which the parameters are updated using one step of gradient descent on that sample. The sum of these log likelihoods is the ‘online likelihood’ and a measure of speed of adaptation. The parameters are initialised using maximum likelihood estimation on the train dataset.
The meta learning procedure then at each episode samples an interventional distribution and an interventional dataset from that distribution. It then performs a gradient based update of the belief over graphs based on the difference between the online likelihoods of each graph on that dataset.
The meta-learning approach appears to be a novel contribution. The authors provide a theoretical argument by counting ‘effective parameters’ to suggest why models using the right causal model obtain a higher online likelihood. Additionally, they prove that the gradient updates to the graph belief are easy to compute and converge. The method is validated with several synthetic experiments which discover the direction of the arrow between two random variables, each either continuous or discrete. Furthermore, they successfully experiment with the combination of learning a representation of a raw data to two random variables with learning the causal direction. The paper is very well written and most claims are carefully proven.
I recommend a weak rejection for this paper, because:
1) The empirical validation is not strong enough, as no real dataset is used, only toy datasets. The toy experiments themselves could also be more extensive.
2) I am unconvinced of two of the theoretical claims made: (A) the fact that the expected gradients in Prop 1 are 0, implies that the right causal graph has better online likelihood and (B) that the method is easily extensible to more than two random variables [Appendix E].
Supporting arguments:
1.1) Fig D.1 suggests that, in the experiments using continuous random variables, the training dataset alone is sufficient to discover the true causal model, under the assumption of independent additive noise, as is done in e.g. [1]. I find it plausible to believe that the training curve on the training dataset alone already makes it possible to disambiguate the causal from anti-causal model. A similar pattern is shown by the authors themselves in Appendix B on discrete variables.
For finite training data, the models are distinguishable, while for infinite training data they are not [Fig B.1]. The exact same holds for finite and infinite interventional data [Fig 1].
Are these experiments then good benchmarks for causal discovery based on intervention when the causal model can already be inferred from the non-interventional training data?
1.2) The simplicity of the representation learning setup doesn’t convince that the method is applicable to more real-world settings with more complicated encoders. Additionally, some important details on this experiment are missing (see below).
1.3) All experiments show the effect of intervention on the cause p(A). No experiments are given for intervention on p(B|A). Does the method then still work?
1.4) No experiments for more than two random variables are performed in this paper.
2.A) Prop 1 shows that the expected maximum likelihood gradient for one conditional probability distribution is zero if the graph is correct and that CPD is not intervened on. Subsequently a claim is made that this effectively reduces the number of parameters and thus that adaptation is faster. However, the zero-expectation gradient may still be non-zero and even large on the small intervention sample. It is unclear to me why they therefore can be excluded in the number of parameters. Furthermore, whether the online likelihood is large will depend not only on generalisation, but also on the training convergence, since not empirical risk minimization is used, but SGD with a fixed number of steps. Thus, even though the authors prove that the method will converge to the causal graph with lowest online likelihood, it is unclear why this is necessarily the correct causal graph.
2.B) In appendix E it is mentioned that cycles can occur in causal models. However, it is unclear why factorization (76) is still correct in the cyclic case. Perhaps I am misunderstanding, but it seems to me that p(x) = \prod_i p(x_i | x_{Pa_i}) only makes sense for a DAG. Hence, how would the online likelihood be computed for cyclic models?
Suggestions for improvement:
- Could you explain in what realistic settings we would have access to data from a large number of different interventional distributions?
- Could you show a plot similar to Fig 1, but with online likelihoods? Such a plot may be more indicative of the ideal episode length than Fig 1.
- Could you provide details on the representation learning experiment? In particular: (1) is \theta_D different in the train dataset and each interventional dataset? (2) How do the gradients of theta flow through the meta-update steps?
- A reference to Appendix B appears missing in main text.
[1] Nonlinear causal discovery with additive noise models. Hoyer et al 2009


In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

###  Summary
- The paper tries to understand what learning principles can lead to representations that allow zero-shot learning/generalization.
- It explores the impact of two properties -- locality and compositionality -- on ZSL performance.
- To encourage locality (and also compositionality since they can overlap), the paper uses an auxiliary loss at an earlier layer (when the receptive field of features is small) to predict local attributes.
- For interpreting local features, the paper computes MI between the global feature map of one image and local features from a different image of the same class and visualizes it using a heatmap.
- It uses normalized TRE to measure compositionality (Normalized to take into account that some methods are biased towards learning more compositional representations.)
- It compares supervised, unsupervised, and self-supervised representation learning methods and studies the impact of locality and compositionality on ZSL performance for these methods.
### Decision and reasons
I vote for a weak accept. However, I think it's a good paper and does a lot of things right. I'm willing to increase my score if the authors can convincingly answer my questions.
Positives:
1- The proposed zero-shot learning from scratch setting is a step in the right direction for focusing on uncovering general learning principles.
2- Locality and compositionality are sensible goals for good representations. The paper defines both and explores their importance with clear and novel experiment-designs and metrics. The experiments are also well conducted.
Negatives:
1- The paper makes a lot of observations, but sometimes does not even try to explain some unexpected observations.
2- CMDIM is presented as a self-supervised algorithm but seems to require class labels.
### Supporting arguments for the reasons for the decision.
Positives:
1- I agree with the paper that ZSL setting is more about uncovering learning principles and less about constructing practical systems that can do well in zero-shot settings. In a practical setting, it doesn't make much sense to zero-shot learning anyway. I also agree that current ZSL work is focusing too much on pushing the state-of-the-art using pre-trained imagenet features and missing the actual goal of the problem setting (I wouldn't say we shouldn't do that at all, however. Some learning principles may require a significant amount of data to learn good representations for zero-shot learning, and imagenet pretraining is a good proxy for that). This paper acts as a good reminder that ZSL research should be done keeping in mind the goal of ZSL.
2- The paper first defines locality and compositionality. It then uses interesting and meaningful metrics for empirically testing if these properties correlate with zero-shot performance. I found the experiment designs to be clever (such as computing parts F-1 score using boolean maps, visualizing MI between local and global features of images from the same class, etc).  I think this paper will act as an important reference for motivating future work on learning more modular representations.
Negatives:
1- The paper doesn't try to explain the possible reasons behind some observations. For example, why does locality not correlate with ZSL performance of generative models? Why does LC loss hurt performance for CMDIM?
2- CMDIM draws positive samples from other images of the same class. As a result, it's not a truly unsupervised learning method. The direct comparison of CMDIM to DIM seems unfair.
### Questions
Q1-  Computation of the local classification and attribute auxiliary loss is a bit unclear. How is it assured that a certain attribute is present in the local feature when computing the auxiliary attribute-loss?
Q2- Did the authors try a baseline in which AC is also trained on the global representation in the context of Figure 2? The extra information about the parts may be the reason behind better ZSL performance, and a global AC could improve ZSL performance without improving Parts F1 score (Although a more likely outcome is that a global AC would increase both the parts F1 score and the ZSL accuracy.)
Q3- For VAEs, the local F-1 score does not correlate with ZSL performance. This seems to contradict the idea locality leads to better ZSL performance. Is there an explanation for this? The paper just glances over this by calling this 'interesting.'
Q4- Can CMDIM be considered a self-supervised algorithm? Doesn't it need class labels to draw positive samples from the same class?
### Other minor remarks
- "Formally, f(x) \elem R is compositional if it can be expressed as a combination of the elements of ..."
The definition would be more clear if the authors could mention some reasonable combination operators here (such as weighted average etc).
- "However, this choice in architecture does not guarantee locality, as CNN representations could only contain “global” information, such as the class or color of the object, despite having a limited receptive field. "
I'm not sure what this means. Why are CNN representations restricted to 'only' global information?
### Update after author's response
The author's response fixes some minor clarity issues. I think this work is a good contribution and should be accepted. I've updated my score accordingly.


Monte Carlo Tree Search (MCTS) has achieved impressive results on a range of discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains. In this work, we introduce TPO, a tree search based policy optimization method for continuous environments. TPO takes a hybrid approach to policy optimization. Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only a few action samples from the policy distribution and defining a new loss function based on the trajectories' mean and standard deviations. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of samples and simulations. However, we observed that bootstrapping tree search with a pre-trained policy allows us to achieve high quality results with a low MCTS branching factor and few simulations. Without the proposed policy bootstrapping, continuous MCTS would require a much larger branching factor and simulation count, rendering it prohibitively expensive. In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all the environments. For example, in complex environments such as Humanoid with a 17 dimensional action space, we achieve a 2.5× improvement over the baseline algorithm.

This paper proposes Tree search Policy Optimization (TPO) algorithm for tasks with continuous action spaces. TPO works after a well trained policy can be obtained (PPO is used in the paper). After a well trained policy is obtained, TPO firstly uses the policy to do Monte Carlo Tree Search (MCTS), with at most 32 actions at each node (state) in the tree (to make MCTS work for continuous action spaces). Then after searching, policy is updated by matching the statistics of MCTS policy (mean and variance), since usually MCTS induces a better policy than the current sampling policy. Experiments on MuJoCo tasks show that TPO has performance improvement over PPO.
1. This paper is basically a special case of dual policy iteration methods (as mentioned), with some heuristics to make MCTS work in continuous action spaces. Unlike dual policy iteration, there is no theoretical justification why doing these heuristics are good/convincible.
2. As a paper focusing on experiments, the results are not enough, in the following senses,
a) There are other existing work of MCTS in continuous action spaces, but they are not mentioned, like
"Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty", Yee et al., 2016.
"Monte Carlo Tree Search for Continuous and Stochastic Sequential Decision Making Problems", Couetoux, 2013.
Without comparing with other baselines, the current experiments are not convincing to claim the proposed TPO method is a better choice over other existing methods.
b) The experiments only show application of TPO on PPO, which does not indicate that TPO can improve performances of other methods, like Soft Actor-Critic (SAC), and "Relative Entropy Regularized Policy Iteration", Abdolmaleki et al, 2018 (RERPI), which are known as better choices for Mujoco tasks. Actually, I was wondering how can TPO work with other true off-policy algorithms like the mentioned SAC and RERPI.
c) The learning results/curves seem to be not efficient. The maximum timestep is 4M. What is timestep? Is it the number of iterations? Does that mean the environment steps are totally 4M *32 = 128 M (since every step there are 32 envs in parallel). Please clarify this, and also show how many environment steps (totally how many number of actions have been taken) are there for PPO here. If their environment steps are not the same, then this comparison is not fair (TPO actually uses more actions). And if 128 M environments are used, then the learning is quite inefficient, for example, the final score for Ant of TPO is about 5000, while SAC achieves 6000 after 3 M environment steps.
3. The MCTS policy update stage (second stage) uses mean and variance matching to update policy Eq. (8). I did not see why this objective is good. There is no intuition or comparison to support it. For the Gaussian policy here, KL divergence between MCTS policy and the current policy induces another different matching of mean and variance. How is this objective compared with KL? Please use experiments to justify.
4. TPO works under restricted requirements (require well performed policies), which makes the claimed contribution of "making MCTS work for continuous action spaces" weak (nearly not hold). Actually, the experiments show that if rho = 0.1, there is no good enough policy, then the proposed method does not work at all. This means the contribution is "MCTS works for continuous action spaces in special cases, that a good policy can be used for sampling". The 32 branch factor is enough also because of this reason. The MCTS in discrete action space is guaranteed to converge (UCT algorithm). However, here there is no evidence that this proposed method will achieve similar results if it starts from initialization rather than well trained policies. From this perspective, I consider this method not really a method that "makes MCTS work for continuous action spaces."
5. What is the replay buffer size for MCTS? Are the trajectories in MCTS buffer going to be used again (or thrown away after calculate the mean and variance of MCTS policy)? I suppose the simulations will be just used for once and the MCTS tree will be thrown away (next iteration a new tree will be constructed). If this is the case, then the proposed method is not really a true "off-policy" method as claimed. It does not have the same sample efficiency as other off-policy methods (like SAC, DQN, DDPG), and it does not need to face the same difficulty like importance ratio corrections. Therefore, it is more a policy update step within dual policy iteration framework, rather than true off-policy learning (with replay buffer storing trajectories, and those trajectories will be reused for multiple times.)
6. TPO requires knowledge of maximum timestep, meaning it is not an any-time algorithm (unlike most existing algorithms).
Overall, this paper has no contributions on theory. And I found the experiments cannot show: 1) the proposed TPO is better than other baselines; 2) TPO can work with other methods, especially off-policy algorithms; 3) the learning efficiency, proposed objective/algorithm are questionable; 4) the proposed method is not really a method that makes MCTS work for continuous action spaces, except with restricted requirements.
=====Update=====
I have read the rebuttal and I keep my rating, since there is no revision to see any improvements.


Sample-selection approaches, which attempt to pick up clean instances from the noisy training data set, have become one promising direction to robust learning from noisy labels. These methods all build on the memorization effect, which means deep networks learn easy patterns first and then gradually over-fit the training data set. In this paper, we show how to properly select instances so that the training process can benefit the most from the memorization effect is a hard problem. Specifically, memorization can heavily depend on many factors, e.g., data set and network architecture. Nonetheless, there still exists general patterns of how memorization can occur. These facts motivate us to exploit memorization by automated machine learning (AutoML) techniques. First, we design an expressive but compact search space based on observed general patterns. Then, we propose to use the natural gradient-based search algorithm to efficiently search through space. Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efficient than existing AutoML algorithms but can also achieve much better performance than the state-of-the-art approaches for learning from noisy labels.

This paper focuses on the topic of learning from noisy -- or as they call it "corrupted" -- labels. Specifically this focuses on an approach where data selection -- ideally of cleaner/less noisy examples --  can help the learn model overcome data noise, akin to the approaches this builds upon (i.e, the Co-Teaching and MentorNet approaches). The specific idea here is to take an AutoML style approach to the problem  in particular to determine how many examples are selected in each mini-batch. The proposed method is based upon natural gradient based updates to the hparams (which was really the only feasible way to tackle this problem given the complex dependence on the hparams and a good choice). The experimental results using synthetic noise corruption are indicative of improved performance compared to the baseline techniques.
Overall while I thought the paper made for a very interesting read and showed some great promise I had some significant concerns as well.
On the plus side:
+ The empirical results on the simulated noisy data are quite positive/
+ The proposed method makes sense as does the search algorithm in the hparam space.
My main concerns with the work stem from the empirical study and choices made there. While I understand that other existing techniques like the Co-teaching and MentorNet approaches have used simulated noise to study the impact of performance of these robustness techniques, at some point I question their validity on real datasets. Noise patterns in real datasets hardly follow some set pattern and thus I hesitate to read much into results derived solely on synthetic datasets. Given that the goal of these techniques is to improve performance when training with real, noisy labeled data why not actually demonstrate performance on such benchmarks? For example, there are numerous datasets from domains like crowdsourcing that allow you to get "noisy" ratings for datapoints. Wouldn't a more compelling argument be derived by showing improved performance on such datasets?
Thus to summarize: I worry that the results derived solely on simulated noise may not be very indicative of performance in more realistic settings and would request the authors to consider providing evidence on more realistic datasets.
I also wanted to note that the paper exposition is lacking in some aspects and I needed to reread certain sections to make sure I understood them correctly. I think the paper would benefit from a good proofread not just from the grammar/spelling perspective (which there are multiple instances which could be improved) but also from the overall presentation and legibility perspective.
All this said: I want to clarify that this topic is not my research focus and hence I am uncertain as to how much findings on these simulated noise patterns carry over to real datasets and their associated noise patterns. If there is existing evidence indicating strong correlation, then perhaps my review may have varied.


ResNet structure has achieved great success since its debut. In this paper, we study the stability of learning ResNet. Specifically, we consider the ResNet block h l = φ(h l−1 + τ · g(h l−1 )) where φ(·) is ReLU activation and τ is a scalar. We show that for standard initialization used in practice, τ = 1/Ω( √ L) is a sharp value in characterizing the stability of forward/backward process of ResNet, where L is the number of residual blocks. Specifically, stability is guaranteed for τ ≤ 1/Ω( √ L) while conversely forward process explodes when τ > L − 1 2 +c for a positive constant c. Moreover, if ResNet is properly over-parameterized, we show for τ ≤ 1/Ω( √ L) gradient descent is guaranteed to find the global minima 1 , which significantly enlarges the range of τ ≤ 1/Ω(L) that admits global convergence in previous work. We also demonstrate that the over-parameterization requirement of ResNet only weakly depends on the depth, which corroborates the advantage of ResNet over vanilla feedforward network. Empirically, with τ ≤ 1/ √ L, deep ResNet can be easily trained even without normalization layer. Moreover, adding τ = 1/ √ L can also improve the performance of ResNet with normalization layer.

This paper shows the stability of ResNet for the output scale of resblock: $\tau < \sqrt{L}^{-1}$ and shows the explosion for $\tau > L^{-0.5+c}$. Based on this analysis, a linear convergence rate of the gradient descent for the squared loss is also shown. In addition, this paper empirically verifies the efficiency of the initialization of $\tau = \sqrt{L}^{-1}$ with/without batch normalization.
Contributions of the paper are summarized below:
- Provides a sharp characterization of the largest scale of outputs of resblocks: $\tau$.
- Enlarges the class of ResNets with the global convergence guarantee of the gradient descent w.r.t. the scale of $\tau$. (A related study [Allen-Zhu+2018b] focused on the scale of $\tau = 1/L$).
- Improve the depth dependence in the network width and the iteration complexity compared to existing studies.
Significance:
To theoretically justify the success of ResNets, this line of research is important, and I think this paper makes a certain contribution in this sense, although there seems to be much room for improvements regarding the depth dependence in the network width. (Unrealistically high over-parameterization: $m=n^8L^7$ is still required). In addition, a proposed technique (Theorem 1) to derive gradient bounds is surprising and beyond the intuition. Indeed, a simple way of using a natural spectral bound cannot explain the stability of ResNets with $\tau=L^{-0.5}$.
Clarity:
The paper is well organized but there are some technical concerns as described below.
Quality:
I think the quality of the paper can be improved.
- Maybe probabilities in some statements are not correct, for instance, a probability in Theorem 2 should depend on the value of $c$.
- It is better to specify the value of $c$ in Theorem 1. It may take an arbitrary small value at the expense of the probability.
- In the convergence analysis of the gradient descent (Theorem 5), the dependency on $\tau$ should be mentioned. Is there no effect on the convergence rate by the choice of $\tau$?
Minor comments:
- The network architecture is different from the standard ResNets. Usually, non-linear activation functions are applied after resblocks. Can theoretical analyses be extended to such a setting?
- Initialization scales of parameters in related studies, as well as $\tau$, should also be clarified for a fair comparison of $\tau$.
- In proofs, citations of theorems and lemmas seem incorrect, e.g., Theorem 1 may be used in the proof of Lemma 3 (and Theorem 3 too?) instead of Lemma 1.
- Typo In Theorem 1: for all $a > l$  --> for all $b > a$ (?).
- There is no definition of $h'_{i,l}$ in Lemma 4. Is it a perturbation of a hidden node h?
-----
Update:
I thank the authors for the response and hard work.
I would like to keep my score. Some concerns have been resolved. Moreover,  I think highly of a proof technique (Theorem 1). However, the advantage of ResNets is still unclear because high over-parameterization is required.


Graph neural networks (GNNs) have been widely used for representation learning on graph data. However, there is limited understanding on how much performance GNNs actually gain from graph data. This paper introduces a context-surrounding GNN framework and proposes two smoothness metrics to measure the quantity and quality of information obtained from graph data. A new GNN model, called CS-GNN, is then designed to improve the use of graph information based on the smoothness values of a graph. CS-GNN is shown to achieve better performance than existing methods in different types of real graphs.

Summary
The paper proposes two graph smoothness metrics for measuring the usefulness of graph information. The feature smoothness indicates how much information can be gained by aggregating neighboring nodes while the label smoothness assesses the quality of this information. The authors show that Graph Neural Networks (GNNs) work best for tasks with high features smoothness and low label smoothness by utilizing information from surrounding nodes which also tends to have the same label. Based on these two metrics, the authors introduce a framework, called Context-Surrounding Graph Neural Network (CS-GNN), that utilizes important information from neighboring nodes of the same label while reduce the disturbance from neighboring nodes from different classes. The results demonstrate considerable improvement across 5 different tasks.
Strength
The authors advocate for better understanding of the use of graph information in learning, which is both an important and interesting problem. Two graph smoothness metrics appears to be intuitive and reflect common situations in graph-based data (1) features from neighboring nodes contribute differently to target node representation (2) neighboring nodes information sometimes causing disturbance if node with different labels tend to be connected. The paper provides some theoretical analysis that supports this claim and thorough experiments that show the correlation between the two proposed metrics with the performance of GNNs.
Weakness
While the paper is reasonably readable, there is certainly room for improvements in the clarity of the paper. First, I would suggest the authors to avoid too much word repetition as well as long, obscure sentences. For example, the first sentence of section 2.2 can  be rewritten as “GNNs usually contains an aggregation step to collect neighboring information and a combination step that merges this information with node features.” The flow of the paper is also hard to follow and need some rearrangement. For instance, paragraph 3 of section 2.1 can be pushed until section 3.3. Another suggestion about the flow is to separate section 2.2 into two subsections for features smoothness and label smoothness (also the title of this section need to be refined). Finally, the results would be more clear if separated into different tables or subsections/paragraphs.
Questions
* Is there a particular reason for using the KLD instead of mutual information?
* In 2nd sentence of section 2.2, what does “node’s own information” mean? If it is the individual node’s features then why it naturally the representation vector h_v (which is aggregated with neighboring nodes?)


Neural networks with ReLU activation functions have demonstrated their success in many applications. Recently, researchers noticed a potential issue with the optimization of ReLU networks: the ReLU activation functions are positively scaleinvariant (PSI), while the weights are not. This mismatch may lead to undesirable behaviors in the optimization process. Hence, some new algorithms that conduct optimizations directly in the path space (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space, and it was shown that, SGD in the path space is superior to that in the weight space. However, it is still unknown that whether other deep learning techniques beyond SGD, such as batch normalization (BN), could also have their counterparts in the path space. In this paper, we conduct a formal study on the design of BN in the path space. According to our study, the key challenge is how to ensure the forward propagation in the path space, because BN is utilized during the forward process. To tackle such challenge, we propose a novel re-parameterization of ReLU networks, with which we replace each weight in the original neural network, with a new value calculated from one or several paths, while keeping the outputs of the network unchanged for any input. Then we show that BN in the path space, namely P-BN, is just a slightly modified conventional BN on the re-parameterized ReLU networks. Finally, our experiments on two benchmark datasets, CIFAR and ImageNet, show that the proposed P-BN can significantly outperform the conventional BN in the weight space.

This paper analyzes a reparametrization of the network that migrates from the weight space to the path space. It enables an easier way to understand the batch normalization (BN). Then the authors propose a variant of BN on the path space and empirically show better performance with the new proposal.
To study BN in the reparameterized space is well-intuited and a natural idea. Theorem 3.1 itself is interesting and has some value in understanding BN. However, the main contribution of the paper, i.e., the proposal of the P-BN, is not motivated enough. It is merely mentioned in the beginning of section 4 and it's not clear why this modification is better compared to conventional BN. This is not verified by theory either. By comparing theorem 3.2 and theorem 4.1, it seems P-BN even gives even worse upper bound of gradient norm.
Plus we don't actually care that much about the issue of gradient exploding since one could always do gradient clipping. The notorious gradient vanishing problem on the other hand, is not address in the theorems.
The formulation of the P-BN seems to be closely related to ResNet, since it sets aside the identity mapping and only normalizes on the other part. It would be better to have some discussions.
Also, the reparameterization and P-BN seems only to apply to fully connected layer from Eqn. (3-5) where they are proposed, but the experiments applies to ResNet. It would be better to describe the method in a broader sense. How would you do this P-BN in more complicated networks?
Finally, it's very unclear to me the value of Theorem 3.1 and the proof that takes almost one page in the main context. The assumption of diagonal elements of matrix w to be all positive is very restrictive and simply removes the effect of ReLU activations.
Therefore I think the paper has some room for improvement and is not very suitable for publication right now.


Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value estimator using a reinforcement signal of the reward obtained on a small validation set that reflects performance on the target task. We demonstrate that DVRL yields superior data value estimates compared to alternative methods across different types of datasets and in a diverse set of application scenarios. The corrupted sample discovery performance of DVRL is close to optimal in many regimes (i.e. as if the noisy samples were known apriori), and for domain adaptation and robust learning DVRL significantly outperforms state-of-the-art by 14.6% and 10.8%, respectively.

This paper proposes a meta learning approach based on data valuation for reinforcement learning tasks. The core idea is to train a second network (the data value estimator) in conjunction to a regular predictor network. The predictor is then trained with samples chosen via the data value estimation. The authors motivate this construction with the goal to filter out unreliable and corrupted data.
It's well established that RL poses a difficult learning problem, and as such the goal to improve the RL process is definitely a good one. To the best of my knowledge the approach proposed here is new. The exposition of the paper is also quite clear, and all parts of the approach are explained nicely. In addition, the submission contains a thorough evaluation of the method.
A central point for the method seems to be the validation data set which is used to train the data value estimator. The text emphasizes that this data set can be "small" several times, and the discussion and results of section 4.5 try to shed light here. However, Figure 5 indicates that a fairly large fraction of samples is needed to identify, e.g., more than 50% of the corrupted samples.
Another cricital aspect for meta-learning approaches such as this one is also the training time. RL is already expensive, so if the meta learning introduces a large factor, the training could quickly become infeasible. Here, the text gives a factor of about 3, which is noticeable, but not overly big. This still seems practical. Potentially, the estimator could also be reused (at least partially) for repeated training runs.
The tests in figure 2 are somewhat unituitive at first - I was expecting results of models trained on datasets with different samples being removed beforehand, rather than removing samples based on a trained estimator. However, this test makes sense on second sight, and e.g., the significant drop in performance after removing the most important samples indicates that the estimator was able to correctly identify a certain portion of data that is actually important for successful predictions. In addition to the noisy label and domain adaptation tests, this paints a positive picture. The method seems to yield useful (be it somewhat small) improvements in terms of learning performance.
One aspect that could be justified better in my opinion is the choise for a discrete representation for the data value estimator. Shouldn't the method likewise work with a continuous representation here? The paper explain how the discrete model is trained with quite some detail, but the choice itself could be motivated more clearly.
However, overall I think the method is interesting and yields nice performance improvements. It is described and evaluated in detail, so I think the paper could me included in the ICLR program.


We study learning control in an online lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, and capably condense broad experiences into compact networks, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. Under constrained computation limits, the agent must allocate its resources wisely, which requires the agent to understand both its own performance and the current state of the environment: knowing that its mastery over control in the current dynamics is poor, the agent should dedicate more time to planning. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By measuring the performance of the planner and the uncertainty of the model-free components, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times. We show that AOP gracefully deals with novel situations, adapting behaviors and policies effectively in the face of unpredictable changes in the world -challenges that a continual learning agent naturally faces over an extended lifetime -even when traditional reinforcement learning methods fail.

The authors study continual, lifelong learning. They suggest a new algorithm, named Adaptive Online Planning (AOP) that combines model-based planning with model-free learning. AOP decides how much additional planning is needed based on the uncertainty of the model-free learner and the performance of the planner. Experiments are carried out on three tasks, i.e. Hopper, Ant and a Maze.
This paper should be rejected. The main reason is that the experiments were only performed for 3 different seeds and are therefore not statistically relevant (see Henderson et al. "Deep reinforcement learning that matters." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.).
Besides the issue of significance of the results section, there are other concerns. Some of them are:
- Page 2: 'The dynamics model is updated immediately at world changes.' - Is this a reasonable assumption? Where does an accurate model come from? Given a perfect model, it is not surprising that a learner that is combined with such a model achieves a superior performance.
- Although the authors state that the 'ability to perform well in old tasks (backward transfer)' is important, they don't explicitly show their algorithm to achieve this goal. Backwards transfer might be included into the experimental section, but I could not find a statement that addresses this explicitly.
- I would like the authors to crisply define their use of the word 'online learning'. Does online learning simply mean to process each sample as it is available or does the term include real-time?
- How is \sigma_{thres} chosen? What is the influence of this parameter?
- The statement that 'AOP uses only 0 - 25% of the number of timesteps as MPC-8, but achieves generally comparable or stronger performance.' is wrong (see Fig 4, d and e). This statement is especially difficult, as results are only averaged over 3 runs.
There are furthermore a few minor concerns:
- the interval for \gamma should exclude 1 in this setting, as the return would otherwise be unbounded.
- In the background section, the authors confuse the definition of the return with reward.
- the term 'deep exploration' is used but not defined
- There are two figures between the subsection header for 4.4 and the text - this is highly confusing


We introduce Neural Markov Logic Networks (NMLNs), a statistical relational learning system that borrows ideas from Markov logic. Like Markov Logic Networks (MLNs), NMLNs are an exponential-family model for modelling distributions over possible worlds, but unlike MLNs, they do not rely on explicitly specified first-order logic rules. Instead, NMLNs learn an implicit representation of such rules as a neural network that acts as a potential function on fragments of the relational structure. Interestingly, any MLN can be represented as an NMLN. Similarly to recently proposed Neural theorem provers (Rocktäschel & Riedel, 2017), NMLNs can exploit embeddings of constants but, unlike NTPs, NMLNs work well also in their absence. This is extremely important for predicting in settings other than the transductive one. We showcase the potential of NMLNs on knowledge-base completion tasks and on generation of molecular (graph) data.

The following paper provides an extension to Markov Logic Networks(MLNs), by removing their dependency on pre-defined first-order logic rules.  This is handled via neural networks which are able to capture the statistical relations, so-called Neural Markov Logic Networks(NMLNs). As this is an implicit representation from the neural network, the rules act as potential functions on the MLN structure. As general MLN techniques are reliant on domain experts or exhaustive structure learning approaches, NMLNs are able to model more domains as provided in the work with knowledge-base completion tasks and generative modelling of molecules.
The primary contribution in this body of work is based on the observation that relational structure repeats regularities in the data, and where deriving the statistics of these regularities is what allows for improved accuracy in a model.  The proposed NMLN  is architecturally identical to MLNs with the difference being the addition of the potential function.
As defined by the paper, fragments are connected subsets derived from relational data. The authors derived sets of fragments with constants defined by values in the data and anonymized fragment sets with integer assignments. With potential functions sampled from the anonymized and the true value fragments. The objective is a search for a maximum-entropy distribution to model the data derived fragments. The neural network aspect comes in the form of the minimum-maximum entropy modelling with weights for given fragments being learned by minimising the entropy of the fragment potential function. Where the model also maximizes the log-likelihood related to the anonymized fragments. The intuition in this work is that by selecting the maximum entropy distribution while also minimizing it by selecting the most informative statistical information for it, we will derive an accurate probability distribution given the possible worlds.
Overall, the paper performs a well enough job explaining the technical aspects. It does a thorough job explaining the algorithmic detail in the main body, and the appendix provides clear and implementable pseudocode equations. It is not exactly clear to me why the anonymization of fragments is necessary, but the authors suggest this places a greater focus on the graph structure and minimizes the model acting differently with different constants. The min-max entropy modelling also appears to be a novel approach in terms of statistical relational modelling. The results also demonstrate the success of NMLNs modelling on relational data and KB completion.
Regarding the technical aspects, a few concerns are the claims that the domains for experimentation seem rather trivial since the smoking and nations dataset are common relational datasets, and the strength of this model is the ability to learn on other domains. This is possibly addressed with the molecule experiments, but more datasets would have helped in confirming the breadth of domains as claimed by the paper. It is also difficult to measure the success of their model with generative modelling as no baseline was present for the molecule experiments. I still view this paper as a positive contribution to MLN research, with a technique that is successful among the few experiments tested.
At the same time, however, given the emerging contributions in the area, such as relational neural networks (Poole et al.), although the technical development is sensible, I don’t find the contribution that technically novel, and seems somewhat straightforward. Given the preliminary investigations, this is a reject from my side.


We propose a new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory rollouts performed on a learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use a special type of uncertainty quantification by a stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model is addressed by next state predictions based on an uncertainty aware ensemble of dynamics models. The influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and a model-based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.

### Summary ###
This paper focuses on model based reinforcement learning (RL). Specifically, the authors consider the setting of combining model based and model free RL algorithms by using the learned dynamics model to generate new data for training the model free algorithm. In order to capture the uncertainty of the environment and the model, the author applied Baysian neural network to learn the dynamics of the environment. The authors approximated the true Bayesian inference process by keeping an anchored ensemble of neural networks, where the prior and posterior of network weights are approximately Gaussian. The ensemble of dynamics model is then used to generate data to train a PPO[1] based agent. In order to prevent the agent from exploiting the learned dynamics model, the authors propose a heuristic way of balancing the amount of real data and model generated data by comparing the rewards.
The authors evaluate the proposed algorithm on simulated robotic locomotion environments in MuJoCo, and the results show that the proposed method has better same efficiency compared to baseline methods in some environments.
### Review ###
Overall I think this paper presents an interesting idea in combing model based and model free RL algorithms. The idea is very well presented and authors include empirical evidence to support the proposed method. However I do find a number of shortcomings that need to be addressed.
Pro:
1. The idea for this paper is really well presented. The structure of the paper is well organized and the experiment results are easy to interpret.
2.  The authors provide a detailed description of the configurations and the hyperparameters for each experiments. Such description would be very helpful if the results in this paper are to be reproduced.
Con:
1. I’m not convinced about the magnitude of novelty in this paper. The proposed method seems very similar to ME-TRPO[2] and it seems to me that the novelty comes from the application of Bayesian ensemble techniques and the generated data ratio tuning heuristics. While these variations might be important for the final performance of the proposed method, the paper does not include any ablation study to further justify the importance of these variations.
2. I’m not convinced about some of the performance of some of the baseline methods presented in this paper. In this paper, MB-MPO[3] does not improve at all during training on Half-Cheetah environment. However, in the original MB-MPO paper, the algorithm does improve and the performance seems to be comparable to that of the proposed method in this paper.
3. The experiment results are not very strong for the proposed method. In 3 of 4 environments, the proposed algorithm does not show much advantage over the baseline algorithms. The only environment in which the proposed method shows significant improvement is Half-Cheetah, and I believe that the baseline algorithms might not be properly tuned in this environment.
4. The paper lacks certain baseline comparisons. There are many other model based RL algorithms developed recently, and it would be important to compare to these methods. Some examples would be ME-TRPO[2], SLBO[4] and MBPO[5].
The idea in the paper is well presented and carefully investigated. However, I am still not convinced about the novelty of the proposed idea and the magnitude of performance improvement. Therefore, I would not recommend acceptance before these problems are addressed.
References
[1] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
[2] Kurutach, Thanard, et al. "Model-ensemble trust-region policy optimization." arXiv preprint arXiv:1802.10592 (2018).
[3] Clavera, Ignasi, et al. "Model-based reinforcement learning via meta-policy optimization." arXiv preprint arXiv:1809.05214 (2018).
[4] Luo, Yuping, et al. "Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees." arXiv preprint arXiv:1807.03858 (2018).
[5] Janner, Michael, et al. "When to Trust Your Model: Model-Based Policy Optimization." arXiv preprint arXiv:1906.08253 (2019).


Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning. Here we employed a classic associative inference task from the memory-based reasoning neuroscience literature in order to more carefully probe the reasoning capacity of existing memoryaugmented architectures. This task is thought to capture the essence of reasoningthe appreciation of distant relationships among elements distributed across multiple facts or memories. Surprisingly, we found that current architectures struggle to reason over long distance associations. Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path. We therefore developed MEMO, an architecture endowed with the capacity to reason over longer distances. This was accomplished with the addition of two novel components. First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory. Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of 'memory hops' before the answer is produced. MEMO is capable of solving our novel reasoning tasks, as well as match state of the art results in bAbI.

Summary:
This paper presents a new task (paired associate inference), drawn from cognitive psychology, which requires linking many pieces of information together to make inferences with long range dependencies. Experimental results show that standard memory architectures fail on these tasks. To redress this, the paper proposes a new memory architecture with several new features that allow for much better performance on the paired associate task. Finally, the paper undertakes systematic experiments on more traditional domains like shortest path problems, showing that the new architecture achieves modest improvements.
Major comments:
Overall this is an interesting and useful work which uses a task from cognitive psychology to illuminate reasoning limitations in prior memory augmented neural networks. It then proposes several architectural and algorithmic fixes to improve performance. The resulting solution appears to work based on the experimental evaluation, and although this does not lead to substantial improvements in the state of the art on the bAbI dataset, it seems likely that these improvements could be useful for even harder task settings.
The new algorithmic improvements are justified mainly by their performance in experiments, which is completely acceptable if the experiments have been done to a high standard. Nevertheless, it would be wonderful if more insight could be gained about why exactly this configuration of architectural changes was selected and what they contribute. The ablation experiments go some way to addressing this question and perhaps the paper would benefit from a greater discussion of these results in the main text.
The experiments appear to have been done to a high standard, and it is promising that the method’s advantages are more pronounced as the task gets harder (eg by demanding longer chains of inference).
The paper could also be improved by more clearly describing how hyper parameters were selected for each experiment (including the different ablation studies). Why were hyperparamters sometimes chosen based on training loss and other times chosen based on validation loss? In general it would be useful to explain the selection procedure very carefully for each result, since it appears to have differed between evaluations.
The paper is reasonably easy to follow, but could be streamlined to enable more discussion of material that is currently in the appendix.
Typos:
ENM->EMN on pg2