Pre-training is transformative in supervised learning: a large network trained with large and existing datasets can be used as an initialization when learning a new task. Such initialization speeds up convergence and leads to higher performance. In this paper, we seek to understand what the formalization for pre-training from only existing and observational data in Reinforcement Learning (RL) is and whether it is possible. We formulate the setting as Batch Meta Reinforcement Learning. We identify MDP mis-identification to be a central challenge and motivate it with theoretical analysis. Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data. In challenging control tasks and without additional exploration on unseen MDPs, tiMe is competitive with state-of-the-art model-free RL method trained with hundreds of thousands of interactions. This work demonstrates that Meta RL from observational data is possible and we hope it will gather additional interest from the community to tackle this problem.

This paper studies the meta-RL problem in the off-policy, batch learning setting. Batch-RL is the setting in which a policy is learned entirely offline, that is, without interaction with the environment and given only trajectories collected by some policy. Compared to RL, Meta-RL involves the additional challenge of task-inference; the goal of Meta-RL is to train a policy that can generalize to a distribution of tasks (i.e. a distribution of MDPs), without actually being given a description of the task (unlike contextual multi-task policy learning). A simple approach for solving the meta-RL problem thus might be to first perform task inference by encoding data from some task into a task description, and then condition a contextual policy on this task description.
The authors state that the straight-forward application of such an idea in the Batch-RL setting fails due to an issue they term the MDP “misidentification” problem, wherein having multiple tasks in a single batch results in confusion between tasks. This issue really only arises in the setting where task inference is jointly learned with the multi-tasking contextual policy. Thus, they propose a stage-wise algorithm wherein first 1) the N tasks in some dataset of trajectories are learned by N separate policies, and then 2) these N separate policies are distilled into a single master policy, wherein the subpolicy enacted by the master policy is modulated by some sort of task description. The distillation procedure thus involves task description (i.e. mapping from task data to a task embedding) and supervised learning (i.e. imitation) of each sub policy when conditioned on respective task embedding.
* Pros
	* Demonstrates the effectiveness of a straightforward stage-wise approach for off-policy meta-RL (albeit without comparison to alternatives)
	* The method seems to perform well on the tasks considered, approaching the level of performance of SOTA model-free algorithms.
	* The approach is general insofar as it could presumably be used with other batch-RL methods (for first stage of algorithm), or even non-batch RL methods. In this sense, the main contribution might be seen as an approach for distilling multiple policies into a single policy in a way that allows for interpolating between them.

* Cons
	* Technical novelty: while they address a problem that has not yet seen much attention, their solution is combination existing solutions. I say this because I am not fully convinced that the more novel aspects of their distillation procedure (i.e. the auxiliary task) are absolutely necessary. I would be willing to change my stance on this, given evidence.
		* Ablations of i.e. the auxiliary task would help to clarify this. In all tasks considered, the transition function T does not change. Therefore, the function composite P(E(s_j, a_j)) should actually not learn any task-specific information when predicting s’_j. If the method works without the auxiliary loss, it is very similar to a stagewise version of PEARL, with Batch-RL.
	* Limited comparisons to relevant alternatives, simple baselines.	 Do not compare to anything other than SAC.
	* Only tasks considered are pointmass, hopper, and half-cheetah; other work (i.e. PEARL) has also been demonstrated on Ant in Mujoco.
	* Argues that it is more stable than variants that involve interaction with the environment / use critic loss for MDP identification, which is a somewhat unfair since in their case 1) they assume they have data that is good enough to learn a good policy in the batch-RL setting, and 2) the stability is by virtue of the fact that they do things in a stage-wise manner because batch-RL works, i.e. they are doing supervised learning without having to bootstrap.
	* Not sure if the use of term “pre-train” is appropriate, insofar as the test tasks are assumed to come from the same distribution as training tasks. It seems to be more about Batch-Meta-RL.
	* Not much attention given to attempts to solve the MDP mis-identification problem with simple solutions like giving side-information at training time (i.e. task-ID).
* Would be helpful to see:
	* More baselines
		* A simple task classifier. This would basically do a nearest neighbor lookup over the training tasks (given a test task), but this might perform well under the reward function.
		* Comparison to something like PEARL, or some method that does involve interaction with the environment -> this would help shed light on whether interaction is necessary for task inference in the the tasks they consider, and if so, such methods would in some sense be oracles.
	* Ablations
		* Remove auxiliary forward prediction loss: if it works without auxiliary loss, this is very similar to a stagewise version of PEARL, with Batch-RL.
		* How much data per task is needed?
	* Harder tasks, where the method can’t approach SAC (as one would expect for sufficiently challenging tasks)
	* Experiment where they study zero-shot generalization by considering disjoint parts of task space
		* Claim zero-shot generalization but in this case they study tasks where resets are not crucial for exploration needed for task inference

* Minor comments:
	* I found the discussion about inductive bias in RL at the end of section 3 (last few sentences of last paragraph) to be a bit vague.
I've given a weak reject mainly because 1) auxiliary loss has not been experimentally shown to be crucial and therefore the technical novelty may be relatively limited, and 2) more comparisons are needed, to alternatives or other baselines.  I would be willing to change my decision on this, given supporting evidence.


A general problem that received considerable recent attention is how to perform multiple tasks in the same network, maximizing both efficiency and prediction accuracy. A popular approach consists of a multi-branch architecture on top of a shared backbone, jointly trained on a weighted sum of losses. However, in many cases, the shared representation results in non-optimal performance, mainly due to an interference between conflicting gradients of uncorrelated tasks. Recent approaches address this problem by a channel-wise modulation of the feature-maps along the shared backbone, with task specific vectors, manually or dynamically tuned. Taking this approach a step further, we propose a novel architecture which modulate the recognition network channel-wise, as well as spatial-wise, with an efficient top-down image-dependent computation scheme. Our architecture uses no task-specific branches, nor task specific modules. Instead, it uses a top-down modulation network that is shared between all of the tasks. We show the effectiveness of our scheme by achieving on par or better results than alternative approaches on both correlated and uncorrelated sets of tasks. We also demonstrate our advantages in terms of model size, the addition of novel tasks and interpretability. Code will be released.

> What is the specific question/problem tackled by the paper?
This paper tackles a restricted multi-task setting where the task is known. The main contribution is a new architecture for training a task conditional model. The new architecture is reminiscent of an encoder-decoder-classifier with ladder (latent) connections, the decoder is conditioned on task ID. The claim is this is a type of modulation, it is unclear. Results on three multi-task datasets show that the proposed method is slightly better than compared methods and single task learning. There is no theory or loss function to analyze.
> Is the approach well motivated, including being well-placed in the literature?
In my opinion, this is lacking. The assumption that task ID is known is fairly severe. Unfortunately the prior works cited also have this restriction, whereas few papers under the topic of continual learning have removed this limitation. This assumption/drawback needs to be clearly mentioned in the paper and discussed if it is realistic? A related shortcoming is that the training data simultaneously comes from all the tasks, whereas prior work has looked at the more interesting setup where tasks arrive sequentially and incrementally.
- Reference [1] seems relevant and should be cited as it shows context dependent gating of tasks / modulation as well. Other missing references e.g. learning without forgetting (LwF) [2] and [3].
- There is not a clear explanation to think that this is modulation since the result is only passed through a residual connection. More importantly there is no discussion on these important issues. I found the writing to be brief and sketched.
- in the introduction, it would be good to define multi-task learning with the assumption made clear. It would be good to introduce what you mean by TD and BU clearly
- Another drawback is assuming the tasks being encoded as integers, whereas there might be a continuous task space with interpolation, or hierarchical task structure.
- "However, all of these works modulate the recognition network
channel-wise, using the same modulation vector for all the spatial dimension of the feature-maps." - why is this not enough? A nontrivial explanation or discussion is needed. Simply extending to W(t, y, x, ch) would increase performance by a little.
- how is the proposed model different from a conditional model like a task conditional classifier? Also in experiments.
- How is the proposed model different from an encoder-decoder? The impact of "modulation" is not clear.
- "We can scale the number of tasks with no additional layers." - task conditional classifier can also scale in this way to the number of tasks. This claim is not valid.
- Page 3: "uncorrelated gradients from the different tasks" - need not be uncorrelated, but still can be interfering
- next about Kendall (2018) and Sener (2018) - need to compare and contrast to them.
- Last para on page 3 seems not relevant.
- Modulation equations: this seems specific to CNNs. How would you extend this technique to beyond CNNs to recurrent units or even simpler MLPs? Modulation as a technique has been successfully applied in these architectures as well.
- "added to the input tensor X through a residual connection" - this is not clear at all. Are the residual connections not shown in Fig 1(d)?
- "it to be unfeasible due to their large dimensions" - can you explain please? later you say "To avoid the unfeasible computation burden of directly optimizing W"
- Fig 1d, would be good to mark the modulation arrows in a different color
> Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
Having said that, I like the experimental section even in the restricted setup. But a lot of details are missing. It is not surprising that there is slight increase in performance over channel modulation due to the increase expressivity.
- table 1: why is there degradation in the LL task across all methods? the introduction of an additional task seems to bring the performance back up. It seems to be a weakness of your method. Please improve the discussion. I'm inclined to think that the tasks are not uncorrelated, as claimed by the authors.
- table 1: how did you arrive at the number of parameters like 1.12x? Doesn't the separate BU and TD nets mean you have at least 2x parameters compared to single task? It seems the larger number of params in single task is mainly coming from the hidden layers?
- table 1: it would be fair for the comparison methods to have equal number of parameters as the proposed method.
- Missing experimental comparison to Kendall et. al. 2018
- Missing details about reproduction of results from Sener (2018)
- An important baseline would be to show image sensitive full tensor modulation without the new architecture. Similar to XdG.
- Another baseline should be a task-conditional classifier that takes task as input along with the image.
- ablation study: what are the auxiliary losses? I could not find any details.
- The third experiment with CUB seems to use a different loss function that the other methods. This is somewhat hard to evaluate.
- number of parameters are not reported for the CUB experiment
- "where only a single pixel is labeled as foreground, blurred by a Gaussian kernel" needs more details about the smoothing
- In the CUB experiment, due to lateral connections, the top-down result 224x224 image is not the only input to the BU2 classifier, the interpretability argument is weak.
- Why did you choose these 4 questions from CLEVR? There are many interesting types of questions that can be handled.


First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss. Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK). Different from typical second-order methods that have heavy computational cost in each iteration, GGN only has minor overhead compared to first-order methods such as SGD. We also give theoretical results to show that for sufficiently wide neural networks, the convergence rate of GGN is quadratic. Furthermore, we provide convergence guarantee for mini-batch GGN algorithm, which is, to our knowledge, the first convergence result for the mini-batch version of a second-order method on overparameterized neural networks. Preliminary experiments on regression tasks demonstrate that for training standard networks, our GGN algorithm converges much faster and achieves better performance than SGD.

Authors propose minimizing neural network using kernel ridge regression. (Formula 9 and Algorithm 1). Main difference of this method is compared to Gauss-Newton, is that it uses JJ' as curvature, which has dimensions b-by-by (batch size b), instead of J'J as curvature, which has dimensions m-by-m (number of parameters m).
When b is much smaller than m, this matrix is tractable to represent exactly. Related approach is taken by KKT (see Figure 1 of https://arxiv.org/pdf/1806.02958.pdf) which also replaces J'J with more tractable JJ'.
There is a long history of authors trying to extend second order methods to deep learning and and finding that curvature estimated on a small batch is extremely noisy, requiring large batches (see papers by Nocedal's group). Authors propose a method that estimates curvature from small batches. Given the history of failures in small-batch curvature estimation, the bar is high to show that small-batch curvature estimation works.
Bulk of the paper is dedicated to theoretical convergence and connections between concepts. Since the focus of the paper is on a new optimization method for deep learnning, I feel like convergence proofs can be moved to Appendix, and more of the paper should focus on practical aspects of the method. Also the connections to other concepts (ie, tangent kernel) are not essential to the paper and could be better left over for a tutorial paper.
I'm not convinced that their method works well enough to have practical impact.
- Their method seems to be limited to neural network with one output (ie, univariate regression task). This is a serious limitation and paper should highlight this more on this, given that vast majority of applications and benchmarks involve more than output variable.
- Practical implementation details are skimmed over. Section 3.3 brings up that to compute Jacobian, one needs to keep track of the output derivative on per-example basis. How is this accomplished? Modern frameworks like PyTorch and TensorFlow don't give an easy way to compute per-example derivatives efficiently.
- Experiments are performed on two tasks that are not well known in the literature. The choice is somewhat understandable given that their method performs for univariate regression, but also this makes it hard to evaluate whether the method works. SGD vs Gram-Gauss evaluation use parameter settings which are not comparable, so it's impossible to tell whether the improvement are due to better choice of hyper-parameters.
The changes needed to make this paper acceptable are extensive, and I would recommed a reject.
I would recommend authors attempt the following changes for future submission:
1. Make it work for multivariate regression. There's a conversion technique to represent multivariate regression in the same form as univariate regression (see Section 2.4 of "Rao, Toutenberg" Linear Models). Essentially it comes down concatenating o output Jacobians (o output classes) along the batch dimension.
2. Use this to evaluate the method on standard benchmarks like MNIST and CIFAR and show that it doesn't cause a significant worsening in quality. Given that similar approach (KKT paper) found bigger improvement on RNN task, an RNN task may be useful.
3. Give more details on implementation. How was Jacobian calculation implemented? Which framework? How was the per-example computation made tractable? Making small-scale experiments reproducible through anonymous github submission would also help


In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influence on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental results on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.

This paper proposes a neural network architecture that provides an agent-agent based embeddings that are used for actions that directly affect specific agent. Proposed architectural choice exploits (implicitly assumed) independence of some actions wrt. observations of agents that it is not directly affecting. Authors perform experiments in minigame of SC2 (controlling small armies) and in NeuralMMO (analogous setup).
It is worth stressing, that apart from conducting typical analysis, authors also put forward more qualitative hypotheses, that are then verified by custom experiments. I highly value this way of conducting research, as it provides way more insights into the method than just pure return plots, often encountered in similar papers.
Major comments
- Please be more explicit in the abstract, when talking about the experiments conducted. Claiming "Experimental results on StarCraft II and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures." could be misinterpreted as providing a solution that can play SC2 better than state-of-the-art methods, while the actual result is to be able to micromanage small fights in this environment well, and thus it would be fairer to say "on StarCraft II Minigame" or more explicitely "StarCraft II small scale fight simulation" or anything else in this spirit. These are great research domains, there is no need to overclaim the results.
- A proposed architecture assumes (to some extent) an independence of effects of action-agent pairs (action aimed at player i is conditionally independent from observation of agent j). There is still a way to represent it, but only by bypassing the proposed architecture, and expressing this behaviour directly in e^i. While this sort of decomposition makes sense for various heavily localised problems (such as micro-management of fights in games like SC2 or NeuroMMO) it looks like a potential limitation, and assumption, that is not articulated in the text. Have authors performed any analysis looking into this phenomenon?
- It is unclear, how ASN compares or relates to the architecture used in OpenAI Five network (https://d4mucfpksywv.cloudfront.net/research-covers/openai-five/network-architecture.pdf) which arguably should be used as a baseline, given that it follows a similar idea of producing per-agent embedding, that influences part of action space, that is directly related to influencing other agents (targeting opponents). Despite not being peer reviewed, it is a well known, and clearly described method, which should at the very least be acknowledged as prior work.
- Some equations are hard to parse, e.g.:
-- page 5, sum_i=1^b [(y_t^tot - Q_tot(s,a; theta))^2]  <- "i" is never used in the equation (I guess it changed meaning, and is no longer agent identity, but rather batch index?)
-- page 4, what does expectation over "t" mean? In the introduced notation, expectation is taken wrt. control policy pi_i, interpreted as a joint probability distribution over agent actions and environment state transition. Can this be unified?
Minor comments
- "When one of the arm units dies" -> "When one of the army units dies"
- details of ASN-PPO could be safely moved to the Appendix, as it is a direct substitution regular policy updates with (2), rather than a separate contribution that requires in-depth description (and given that paper is quite long, it could use a bit of message "sharpening")
I am happy to revisit (and increase) the rating assigned, once authors address the comments above, especially the relation to OpenAI Five architecture.


ABSTRACT AI today is far from being able to write complex programs. What type of problems would best teach computers programming, and how should such problems be generated? We suggest programming puzzles as a domain for teaching computers programming. A programming puzzle is a short program for a Boolean function f (x) with the goal of finding an input that makes f return True. Puzzles are objective in that one can easily test the correctness of a candidate solution, in contrast to other common specifications for program synthesis like input-output pairs or English problem descriptions. To address automatic puzzle generation, we propose a GAN-like algorithm called "Troublemaker" which can generate puzzles adaptively targeted at any given puzzle-solver. Rather than generating a single dataset of puzzles at random, it generates a diverse set of puzzles that are difficult for the solver. In our experiments, Troublemaker learns to generate challenging problems for a variety of state-of-the-art puzzle-solving techniques.

This paper proposes a trainable 'puzzle' program synthesizer that outputs a program f with a specific syntax. These 'puzzles' are structured as boolean programs, and a program solver solves the puzzle by finding an input x such that f(x) = True.  The authors motivate this task by making a case that puzzles of this sort are a good domain for teaching computers how to program.
The paper is fairly clear overall. There is some repetition in the early parts, so this could be restructured a bit, but these are minor points. A more significant restructuring however, is that this work would benefit from the related work being present the beginning of the work. Since this work is so similar in many ways to previous work I think the overall clarity of the paper would be improved, and the contributions clearer, if the work was better situated with respect to related work.
The experiments demonstrate that the trainable puzzle generator is able to produce harder (i.e. takes longer time to solve) puzzles than a random or probabilistic generator of the same grammar. While this does show that the program generator is learning something useful, these results are insufficient to show the utility of this approach in any real context. It seems the most interesting solver to assess is a trainable solver. Yet only 1 of the 4  solvers they assess is trainable. I know the authors make a point that they are not putting forward any new solver algorithms. That ok, however, taking existing trainable solvers and assessing how they perform with this guided puzzle generation vs. some other puzzle generation approach is a critical empirical study. Furthermore, it would be helpful to have more discussion of the baseline methods of generating puzzles. When the trainable puzzle solver was originally proposed, how was it trained? Where did the  data come from? How does that compare to this approach. I am not very familiar with this literature, and I imagine this paper would be of interest to folks outside the program synthesis space, so it would be very helpful to better explain this (also we note about related work). There are several additional empirical analysis that could be added to improve this work. For example, for the trainable solver, a plot of (training time) vs (time to solve puzzle) would be interesting. Beyond looking at training time, does a solver trained with the guided puzzle generator end up being a 'better' solver in some way? Are the resulting puzzles harder but still being solved? Is there a way of quantifying the 'hardness' of a puzzle? Perhaps a proxy like size? Then it would be cool to plot (training time) vs (approx puzzle hardness) to demonstrate that . the puzzle generator is really developing a reasonable curriculum.
Finally, there is a bunch of related work that I think is missing. Again, I'm not super familiar with this work, but I think there is a lot of curriculum learning stuff within RL that seems super relevant. Of particular relevance is the Alice/Bob framework from "Intrinsic motivation and automatic curricula via asymmetric self-play" seems very similar to the work at hand.  Something that is interesting in the Alice/Bob framework that could be transferred over here is the notion of the generator wanting to make a puzzle hard, but not too hard, i.e. make it just outside the solvers current capabilities.
Overall my assessment is that this paper doesn't quite meet the standard for ICLR. My two major critiques are (1) the related work is seriously lacking making it difficult to situate this work in a broader context. The authors also seem to miss the entire curricular learning literature. (2) The empirical evaluations are lacking. In particular, more thorough analysis of how the generator is behaving, the type of curriculum it learns, and the resulting impact this has on a trainable solver all are missing. Furthermore, more focus on trainable solvers would improve this work.
I'm not an expert in this area so it is possible I misjudged the significance of this work. I'm certainly open to revising my assessment if the authors are able to address (2) in a meaningful way.


While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging -ambiguous future or error propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies. To capture spatiotemporal dynamics in videos, we combined LH-STM with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods.

The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a "previous history" term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper.  The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames).
Overall I believe there are serious flaws with the paper that prevent acceptance in its current form.
First, I believe the paper starts from the wrong assumption, namely that current video prediction models are limited by their capacity to limit the propagation of errors and to capture complex dynamics. Instead, it is well known that the main difficulty for longer term video prediction is to manage the increasing uncertainty in future outcomes. Stochastic models such as SVG-LP or SAVP are currently the state-of-the-art in video generation, with deterministic models not being able to generate more than a few non-blurry frames of video. While the authors mention that they do not focus on future uncertainty here, it is not clear how the proposed model helps to generate better longer-term videos when it does not deal with what actually makes long-term video generation difficult. In addition, it's misleading to claim that current models produce high quality generations for "only one or less than ten frames", especially without defining high quality. Models such as SVG [1] or SAVP[2] can produce non-blurry videos for 30-100 frames for the BAIR dataset, for example.
The experiments are missing 1) SVG as a baseline, 2) metrics that correlate with human perception such as LPIPS or FVD [3] and 3) qualitative samples that compare to stochastic models. Deterministic models can achieve very high PSNR/MSE/SSIM scores but produce very bad samples, as these scores are maximized by blurry predictions that conflate all possible future outcomes. This is highly apparent when looking at samples, and metrics that correlate better with human perception are usually better to compare video prediction methods. Comparisons to SAVP are found in Table 1 and 3 but there are no figures comparing samples from this model to the proposed model. The samples from the proposed model on the BAIR Push dataset for example (found in the appendix) are of significant lower quality than those reported from SAVP or SVG, and at the same time they are not longer-term than the predictions from these models. Consequently, the experimental section does not correctly assess how this model can generate better longer-term prediction than current models and it also does not give an accurate assessment of the model with respect to the current state-of-the-art.
To sum up, the paper does not adequately address how the proposed model allows for longer-term video generation. It is missing critical qualitative comparisons to state-of-the-art models such as SVG and it is unclear how the proposed model is different from a ConvLSTM with attention on previous hidden states.
[1] Stochastic Video Generation with a Learned Prior. E.Denton and R. Fergus. ICML 2018
[2] Stochastic Adversarial Video Prediction. Lee et al. Arxiv 2018
[3] Towards Accurate Generative Models of Video: A New Metric & Challenges. Unterthiner et al. Arxiv 2018
--- Post-discussion update ---
The authors have addressed a number of points raised by the reviewers and I'm raising my score to a weak reject from a reject. There are important remaining issues with the experimental section and the conclusions reached from their results, and therefore I still think the paper is below the acceptance bar.


In human perception and cognition, a fundamental operation that brains perform is interpretation: constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence. The problem of interpretation is well matched to an early and often overlooked architecture, the attractor network-a recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics. We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture with a novel training loss, activation function, and connectivity constraints. We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution. We argue that this architecture is better motivated than ever-deeper feedforward models and is a viable alternative to more costly sampling-based generative methods on a range of supervised and unsupervised tasks.

This paper studies and evaluates variations on attractor networks (which have recurrence and converge to a fixed point) trained end-to-end by SGD on image completion or super-resolution tasks. Several loss functions are discussed and compared.
The only quantitative results against published state-of-the-art results (on  the super-resolution task) show that the proposed approach wins on some measures and tasks and loses on others (in fact it wins 25% of the time, within a pool of 4 methods...). While the authors claim superior performance against other recent attractor networks, no quantitative comparison was found in the paper. There were quantitative comparisons against the "convolutional denoising VAE" but that model was not previously proposed by other authors to be competitive nor was it defined properly (and it is not clear from its name and the description whether this is really a VAE,  since VAEs are generative models and are not trained to denoise). Overall, I am concerned that the experimental  part of  this paper does not warrant the conclusions of superiority. That being said, I like the approach and I believe that these issues can be fixed but I'd like to see those revisions before accepting the paper.
The experiments on toy problems like the Bar Task are not necessary. Better use the place for real comparisons against published state-of-the-art benchmark baselines. Otherwise (like with your own implementation of the CD-VAE) it is plausible that the comparisons will be biased (one usually works harder on our own method than on improving someone else's architecture for our task).
Similarly, the experiments on MNIST are not very informative and you might want to push them to appendices. Also, a more interesting comparison would have been with the convolutional version, where convnets do a LOT better than 1.5% error.
The comparison against Liao et al is not fair because they don't use convolutions, which we know can make a huge difference.
In addition to the experimental results  issues, there are several places in the paper which are unclear or possibly wrong. See minor points below.
One theoretical issue which bothers me is with the "loss" L_{\Delta E}. There is no guarantee that it is lower-bounded, i.e. it can diverge which would not be appropriate for sure.
The training procedure which ends up being used is TD(1) but that procedure (in the context of the setup of the paper) has not been explained at all. It is really necessary to clarify that.
It should be clarified that the parameters are tuned by gradient descent on the loss (using backprop through time), or not?
Minor points
The last paragraph of section 2 seems strange to me. Many ANs are also EBMs so the critique of EBMs (as  if this did not apply to ANs) probably needs to be removed or reconsidered. Also the last sentence of the paragraph is clearly wrong,  since EBMs can easily handle inputs (by opposition to the outputs which are to be predicted or constructed), e.g. see conditional Boltzmann machines or the ANs in the style of Equilibrium Propagation - Scellier et al 2017 (which by the way should probably be mentioned in the review, along with follow-up work).
The 'bipartite' structure is not clear. What are the two "parts" when you have more than 2 layers? Clarify.
Note that the proposed sweep is inefficient compared to the one proposed for Deep Boltzmann Machines (Salakhutdinov et al 2009) which alternates between odd and even layers. The latter is also more biologically plausible than a feedforward-feedback sweep (consider that the differences in update times of different layers are very different depending on their depth, with your proposed scheme).
Define CBAN.
Please clarify how many iterations are needed for convergence in the various experiments. My own experience with ANs suggests that convergence can be quite slow, which is problematic both in terms of computational cost (compared to standard feedforward methods with reasonable depth) and in terms of memory. Hence the sentence in the conclusion stating that the computational cost of CBANs is no greater than DNNs is not clearly true until you demonstrate those aspects quantitatively.


iclr2020_LLQL
Deep reinforcement learning (RL) algorithms can learn complex policies to optimize agent operation over time. RL algorithms have shown promising results in solving complicated problems in recent years. However, their application on real-world physical systems remains limited. Despite the advancements in RL algorithms, the industries often prefer traditional control strategies. Traditional methods are simple, computationally efficient and easy to adjust. In this paper, we propose a new Q-learning algorithm for continuous action space, which can bridge the control and RL algorithms and bring us the best of both worlds. Our method can learn complex policies to achieve long-term goals and at the same time it can be easily adjusted to address short-term requirements without retraining. We achieve this by modeling both short-term and long-term prediction models. The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value. The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.

This paper proposes a Locally Linear Q-Learning (LLQL) method for continuous action control. It uses a short-term prediction model and a long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously. The problem the paper seeks to solve is important. However, this paper has several issues.
First, there seems to be an over-claim of the contribution. The proposed method is more like hybrid of model-based and model-free RL method. Specifically, the short-term prediction model is in fact the linearized dynamic system with system parameters modeled by deep neural networks, while the long-term prediction model is in fact different (state- and action-) value functions. For this reason, it is probably unnecessary to name them as “short-term network” or “long-term network”, since they are simply the system model (or the model-based part) and the value functions (the model-free part).
Second, the proposed method is not sufficiently evaluated. It is only evaluated on the toy Mountain Car task (and the Crane system in supplementary material). In order to justify the performance of an RL algorithm for continuous action space, it should be at least evaluated on the set of MuJoCo tasks.
Third, the proposed method is not sufficiently compared with different baselines. In Figures 3-5, the proposed LLQL algorithm is never compared to any baseline method, leaving it open whether it is actually better than earlier methods like DDPG. In Table 1, LLQL is compared to DDPG (a model-free method), and is shown to achieve better performance. However, this seems to be unfair because the proposed method is in fact a model-based RL algorithm. Therefore, it should at least compare to other model-based algorithms (and also other riche set of safe-exploration RL methods).
Other comments:
•	In eqn. (6), \gamma should be \gamma^{i-k}?
•	In the paragraph after (8), “Q-learning algorithms (16)…” is referring to a wrong equation (16) for Q-learning. Or probably the authors are not using the correct format to cite the reference. (This seems to happen repeatedly in later part of the paper such as as in the paragraph between (14) and (15).) It confuses the equation number and the reference number.
•	More explanation should be given about d(x_k|\theta^d) and h(x_k|\theta^h) after (15). The meaning of them has never to defined before.


Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks. However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently. On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer. In this paper, we attempt to bridge this gap with "Hierarchical Accumulation" to encode parse tree structures into selfattention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task. It also yields improvements over Transformer and Tree-LSTM on three text classification tasks. We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.

This paper extends transformers by enabling them to incorporate hierarchical structures like constituency trees structured attention with hierarchical accumulation. In particular, they modified the architecture of transformers such that they can learn phrase-level attention scores, and use them in the final assigned task of text classification or language translation.
They also explored a couple of variants of their proposed architecture: Hierarchical embeddings and Subnet masking which helped them outperform SOTA methods including Tree-LSTM (similar to this paper in principle, except that it was designed for LSTM-based models and not transformers).
The paper is well written and well-augmented with supportive figures. Particularly, Figure1 was very helpful in understanding all the complexities of the proposed model. Further, the authors justify the utility of the proposed approach covering different aspects of evaluation including comparative studies with baselines, ablation studies, phrase vs token-level attentions, training-time analysis.
A limitation of this work is high inference cost. As the results indicate, parsing trees from text is the most costly step in the entire framework, and consequently, the inference time of proposed model will still be much higher than transformers. Hence, this work might still not be applicable to low-latency constraint scenarios.
Other Comments:
1) I did not fully understand why it would be better to mask out the non descendants in subnet masking approach. Why shouldn't a phrase node seek attention from tokens outside its scope? Probably the answer lies in the the way these trees are constructed. Nevertheless, it would be useful to provide some intuition with examples to motivate subtree masking.
2) In Equation(5), the subscripts "i-1" should be replaced with "i"? Otherwise it will be sensitive to ordering of non-terminal nodes in N, and also Figure 1 wouldn't make sense.


In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given. The structure of clean and noisy data is modeled by a graph per class and Graph Convolutional Networks (GCN) are used to predict class relevance of noisy examples. For each class, the GCN is treated as a binary classifier learning to discriminate clean from noisy examples using a weighted binary cross-entropy loss function, and then the GCN-inferred "clean" probability is exploited as a relevance measure. Each noisy example is weighted by its relevance when learning a classifier for the end task. We evaluate our method on an extended version of a few-shot learning problem, where the few clean examples of novel classes are supplemented with additional noisy data. Experimental results show that our GCN-based cleaning process significantly improves the classification accuracy over not cleaning the noisy data and standard few-shot classification where only few clean examples are used. The proposed GCN-based method outperforms the transductive approach (Douze et al., 2018) that is using the same additional data without labels.

The paper makes a significant attempt at solving one of the practical problems in machine learning -- learning from many noisy and limited number of clean labels. This setting is presumably more practical than the setting of few-shot learning. Noisy labels are often abundantly available and investing in methods that can take the noise into account for building a discriminative model is quite timely.
To be honest, the theoretical contribution of the paper is limited.  The authors make use of the nearest neighbour graph obtained from a reduced-dimensional set of features to compute the weights of the noisy labels that must guide the predictive model. From this perspective, the paper seems like an application of existing tools (such as CNN, graph convolutional network and binary classification). However, that does not undermine the superior results the authors have received in the novel application they have targeted. I appreciate the effort that went validating these ideas with real-world datasets.
In future, I would like to see a joint approach to such training, where the function g(), the nearest neighbour graph loss and the classification loss are all tied in the same objective function and are optimized jointly.
The paper has few really minor grammatical errors and typos. Please fix those before uploading the final draft.


Solving tasks in open environments has been one of the long-time pursuits of reinforcement learning research. We propose that data confusion is the core underlying problem. Although there exist methods that implicitly alleviate it from different perspectives, we argue that their solutions are based on task-specific prior knowledge that is constrained to certain kinds of tasks and lacks theoretical guarantees. In this paper, Subjective Reinforcement Learning Framework is proposed to state the problem from a broader and systematic view, and subjective policy is proposed to represent existing related algorithms in general. Theoretical analysis is given about the conditions for the superiority of a subjective policy, and the relationship between model complexity and the overall performance. Results are further applied as guidance for algorithm design without task-specific prior knowledge.

This paper introduces a new framework for reinforcement learning (named subjective reinforcement learning) which aims to resolve some of the inherent problems with RL in open environments.  The authors posit that one problem with RL in open settings is “data confusion”, which they describe as being situations where there are external factors (e.g. timeframe) that affect the action space differently.  They propose a “subjective reinforcement learning framework” which, as I understand it, can be described as an ensemble of traditional MDP’s subject to external factors k.  The paper evaluates how this subjective policy compares to traditional MDP’s in terms of theoretical bounds on performance.
Although this learning framework seems like a potentially interesting future research direction, I tend to lean towards rejection for the reasons that: (1) the theoretical analysis is difficult to follow and there is sometimes a lack of clarity throughout the paper, (2) it isn’t very clear how easy this framework would be to implement aside from the theoretical guarantees and there aren’t any experiments or proofs of concept that would demonstrate the feasibility or practicality of the proposed framework in a real scenario, (3) the paper would benefit from more discussion of how their work differs from related techniques (like hierarchical RL, various forms of meta-learning, etc.).
Here are some more concrete points:
(1) The feasibility of this framework in a real-world scenario seems a bit hard to imagine and a strong use-case or proof-of-concept would be very helpful.  I liked that this paper provided some theoretical analysis for guiding the design of these systems.  However, it seems like these claims would also benefit from detailed empirical analysis and experiments.  Without empirical results, I feel a bit skeptical about how straightforward it would be to implement such a system or whether it would really be significantly useful in practice.  Similarly, though there are theory-based suggestions for how to optimally design such a system, it might be difficult to implement this system with optimal hyperparameters in a real-world use-case and the challenges in doing so are not really addressed.
(2) In spite of claims that this method is able to be trained without domain knowledge, it seems like domain knowledge would still be necessary for things like determining what external information (K) is available and necessary, determining the appropriate N_S, etc.  It may be helpful for the authors to explain a bit more about how these things can be determined in a truly agnostic way.
(3) It seems like there should be more discussion of the difference from hierarchical reinforcement learning.  In practice, hierarchical RL also can be used in similar ways to what’s described here.  As the authors point out, hierarchical RL is not necessarily splitting into submodels that handle data confusion problems, it seems like that is a constraint that could be added into a hierarchical framework’s design.
(4)  I appreciate that the authors provide detailed theoretical analysis, but it can sometimes be confusing and difficult to follow.  I had some trouble evaluating the correctness of several of the proofs.  It may benefit from re-writing with more concise definitions of all of the variables and more clearly stated assumptions about observable information.  Here are some points of confusion for me:
    - It seems like certain letters (eg. A or K or N_s vs N_d) are being overloaded as variable names with different fonts.  I realize that this is somewhat unavoidable, but I would recommend that the authors try to disentangle the naming a bit for improved clarity.
    - In theorem 1, you stated "the number of all possible data samples tends to infinity when the total number of samples N_d approaches infinity".  This proposition seemed confusingly worded to me.  Maybe I am misunderstanding the wording, but it seems like possibly a tautology?
    - I’m not sure I understand what is meant by “fake subjective policies” in Theorem 1.  Could you explain what is meant by that and the intuition here a bit more?
    - It’s still unclear to me how K (that is, the external information) is being collected at any given timeframe.  Are you assuming that the necessary types of external information corresponding to K (your examples are ‘state history, out-of-MDP task encodings, samples from related tasks, etc.’) have been pre-determined?  If so, how could the most-appropriate type of external information be chosen in a practical way?
I also noticed a few (very minor) grammar errors that authors may want to fix, though they did not affect my review:
- page 1: "tasks in open environments poses difficulties" --> "tasks in open environments pose difficulties"
- page 1: "Problem is that both..." --> "The problem is that both..."
- page 2: "we propose a novel framework named as Subjective reinforcement" --> "we propose a novel framework named Subjective reinforcement"
- page 4: "should contain no data confusion" --> "should not contain data confusions"


Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs, but also reduces the variance in gradients. We evaluate BQNs on MNIST, Fashion-MNIST, KMNIST and CIFAR10 image classification datasets. compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).

This paper proposed Bayesian Quantized Neural Networks (BQN), which is a rough Bayesian treatment for quantized neural networks (QNN). The advantages the BQN over QNN is that it provides better uncertainty estimates as well as accuracy. The disadvantage is that BQN loses the memory efficient property of QNN, that is, BQN in general still needs to store real-value parameters, which parameterize the binary weights. The authors compare BQN to EQNN and show that BQN outperforms the latter.
In general, the paper is well organized and fairly readable, though the first 2 sections could have been more concise. I appreciate the careful description of notations.
The topic of Bayesian formulation and inference in the context of QNN is interesting. I like the analysis provided for handling small/medium/large fan-in for Bayesian QNN.
One of my concerns is that some description in the provided theorems (as well as the proof) is not very rigorous and sometimes misleading. For example, Theorem 3.2 claims that there is an analytic form for Bayesian inference in softmax layers. One would expect that it is either an exact integration over the distribution of h, or a valid bound. Unfortunately, it is neither. Looking at the proof, it seems it is a rather coarse approximation. Note that in the Appendix, inequality in Equation (42) and Equation (47) takes different direction, meaning that this is not a valid bound for the error term L. The authors may want to adjust the description to make it more rigorous and avoid confusion.
The paper describes BQN in very general terms. It is unclear what distributions are used to parameterize the weights in the experiments. Do you use Bernoulli distributions to parameterize the binary weights {-1, 1}, and learn the real-valued parameters for the distributions during training?
Theorem 3.1 is straightforward. The rough approximation provided in Theorem 3.2 is interesting. It would be interesting to compare this with Gaussian natural-parameter networks, which bypass the softmax problem using multi-logistic regression for classification, where the convolution of Gaussian and sigmoid functions has a closed form.
In terms of related work, references such as natural-parameter networks (NPN) and its variants (one of which specifically handles the softmax case) are missing [1, 2]. Note that, similarly to BQN, NPN also provides a general probabilistic (Bayesian) framework for sampling-free learning/prediction. Theorem 3.3 is also covered in Section 3.2 of [1].
This brings me to another of my concerns. The only baseline provided is E-QNN from three years ago. There are various state-of-the-art baselines, either on QNN or BNN. It would certainly make the paper much stronger if such baselines are included, especially when the authors decide to take up the full ten-page space in this submission.
Minor:
P2: bf Bayesian -> Bayesian
P2: theta that predicted -> theta that predict
P6: Lemma 4.2 -> Theorem 4.2
P6: by with -> with
Assuming most of my concerns can be addressed during the feedback period, I tend to give a ‘weak accept’ to this submission.
Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018


Trained generative models have shown remarkable performance as priors for inverse problems in imaging. For example, Generative Adversarial Network priors permit recovery of test images from 5-10x fewer measurements than sparsity priors. Unfortunately, these models may be unable to represent any particular image because of architectural choices, mode collapse, and bias in the training dataset. In this paper, we demonstrate that invertible neural networks, which have zero representation error by design, can be effective natural signal priors at inverse problems such as denoising, compressive sensing, and inpainting. Our formulation is an empirical risk minimization that does not directly optimize the likelihood of images, as one would expect. Instead we optimize the likelihood of the latent representation of images as a proxy, as this is empirically easier. For compressive sensing, our formulation can yield higher accuracy than sparsity priors across almost all undersampling ratios. For the same accuracy on test images, they can use 10-20x fewer measurements. We demonstrate that invertible priors can yield better reconstructions than sparsity priors for images that have rare features of variation within the biased training set, including out-of-distribution natural images.

Update: I have read the other reviews and the author response and have not changed my evaluation.
Recent work has shown that GANs can be effective for use as priors in inverse problems for images such as compressed sensing, denoising, and inpainting. A drawback is that GANs may have the problem of inexact reconstruction, and strongly reflect the biases in the training set yielding poor performance on out-of-distribution data. This paper shows that the exact inverses available to normalizing flow models and their broad assignment of likelihoods allows for better reconstructions especially on out-of-domain data.
As far as I know, this is the first work to use normalizing flows for inpainting and compressed sensing. The approach and application is very natural, although it’s a bit surprising that using the likelihood as a prior term directly did not work very well. The results of this work show that invertible generative models have utility for inverse image problems even when the quality of raw samples is substantially below GANs. In my opinion the main advantage in this method is not on having low reconstruction error on observed pixels, which becomes less of a problem for more powerful GAN models, but rather the good performance on out of domain data which is somewhat surprising. The authors are reasonably thorough, testing their model on a variety of problem settings and perform ablation studies on hyperparameters.
As additional baselines for compressed sensing and denoising, it would be good to compare to the Deep Image Prior since there is effectively no out-of-distribution input for this untrained model and it performs well with moderate image corruption. Additional discussion about the two could be useful, as for the Deep Image Prior a similar patter is observed where denoising requires explicit regularization (early stopping or gradient noise for DIP) but image completion and compressed sensing do not. Also, there have been many improvements to DCGAN over the years that might ameliorate the problems that were observed in reconstruction, but I don’t fault the authors much for this as it can be difficult training models like StyleGAN even at 64x64 sizes.
It might also be interesting to know whether the good performance on out-of-distribution inputs is due to the exact invertibility or the log-likelihood objective, although I would guess that it is the latter. On way to test this would be training the GLOW model with an adversarial objective instead of NLL as done in [1].
Minor Comments:
Figure 4 would probably be better with a logarithmic scaling for # of measurements
I did not understand the comment about sublevel sets of the data misfit term being inverse images of cylinders, maybe this could use some elaboration.
[1] https://arxiv.org/abs/1705.08868


Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. These properties make it a natural fit to guide the training of interactive agents as it could ease recurrent challenges in Reinforcement Learning such as sample complexity, generalization, or multi-tasking. Yet, it remains an open-problem to relate language and RL in even simple instruction following scenarios. Current methods rely on expert demonstrations, auxiliary losses, or inductive biases in neural architectures. In this paper, we propose an orthogonal approach called Textual Hindsight Experience Replay (THER) that extends the Hindsight Experience Replay approach to the language setting. Whenever the agent does not fulfill its instruction, THER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, THER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER.

This work attempts to learn instruction following agents without the requirement for a paired instruction, behavior corpus and instead simply relying on environment interaction and using a form of hindsight relabeling to learn how to relate language and behavior.
Introduction:
It thus leads to the following questions: are we eventually making the rein-forcement learning problem harder, or can we generate learning synergies between policy learningand language acquisition? -> it really doesn’t seem like the point of instruction following is that. It seems like you want instruction following so that you can communicate novel objectives to your agent, with the promise of generalization. Maybe reword?
The motivation for using HER makes sense, but maybe a bit more would be useful to describe why we need text here and not just regular HER.
I think this line “triggers alearning synergy between language acquisition and policy learning” is pretty confusing and not really adding too much value. Would remove.
Overall motivation makes sense, do language grounding in an interactive way much more effectively by leveraging hindsight relabeling but to get around the circular problem of hindsight generation leverage a model of successful behaviors seen thus far. This is a pretty neat thing to do!
Textual HER:
“We emphasize that such signals are inherent to the environment, and an external expert does not provide them” -> I do not think this is true. Rewards do not magically show up in the environment, they have to be provided. I get what you’re trying to say but this statement is very often not true. Please revise.
Conceptual question: what happens if none of the random trajectories are successful coz the reward is so sparse? Wouldn’t this be prohibitive? Importantly, I would be curious to understand how the number of entries in D affects the relabeling function m_omega and how this can be good or bad depending on the schedule of training. For instance if the D is very small at the start, it’s not going to be very good at doing the relabeling and might be erroneous.
Experiments
“Surprisingly, even highly noisy mappers, with a 80%noise-ratio, still provides an improvement over vanilla DQN-agents” -> do you know why?
The synthetic noisiness that you introduce is from a different distribution than the type of noisiness you’d expect from just having very few successful trajectories to train m_omega right? How can we evaluate that?
If we consider the number of successful trajectories obtained just by accident, is it even 1000 or 5000 as required to train the m_omega. If this is a negative feedback loop couldn’t it just keep getting worse because we are using erroneous m. Should we use some notion of uncertainty or something to know when to relabel with m? Or does it happen always
I generally like Section 4.2 -> nicely motivated!
“We emphasize again that it is impossible to have an external expert to apply HER in the general case” -> why??
Can the authors introduce other baselines. For instance the recent paper from Yiding Jiang, Chelsea Finn, Shane Gu and others might be a start. Consider corpus based instruction following would be another. Maybe these can be compared in terms of the number of instructions that need to provided to it? But I think for a successful ICLR paper, we would need 1-2 more meaningful baselines.
“Finally, we observe a virtuous circle that arises.” -> is there some mechanism to ensure that this is a virtuous cycle and not a vicious one? Couldn’t we just have horrible label corruption and then everything goes bad?
How easily would this scale to more temporally extended tasks in minigrid which have larger grids and more challenging tasks which are harder to solve in the sparse reward case?
Can we analyze whether the language goal space has some favorable generalization properties over a state based goal space as typical HER uses?
The language analysis in Section 4.4 is quite insightful and shows the good performance of the instruction generator over time.
How would this fare as language got more ambiguous and multimodal and the instruction generator had a harder time as well as HER might generalize more poorly?
Related Work
Fu et al (From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following) might be relevant for instruction following as well, and some of Karthik Narasimhans work.
Learning interactively with language would also be related to Co-Reyes et al (Guiding Policies with Language via Meta-Learning)
Yiding Jiang’s recent work would also be relevant (https://arxiv.org/abs/1906.07343)
Overall I like the formulation, and it seems pretty useful for instruction following. But we need more comparisons, and a little more motivation on when this thing might become degenerate coz of the m labeling. Perhaps even a discussion/experiment on the uncertainty measure might be helpful.


Recent few-shot learning algorithms have enabled models to quickly adapt to new tasks based on only a few training samples. Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose a method that focuses on regression tasks. Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of sparsifying basis functions. This enables using a few labelled samples to learn a good approximation of the entire function. We design a Basis Function Learner network to encode the basis functions for a task distribution, and a Weights Generator to generate the weight vector for a novel task. We show that our model outperforms current state of the art meta-learning methods in various regression tasks.

The paper proposes a regression approach that, given a few training (support) samples of a regression task (input and desired output pairs), should be able to output the values of the target function on additional (query) inputs. The proposed method is to learn a set of basis functions (MLPs) and a weight generator that for a given support set predicts weights using which the basis functions are linearly combined to form the predicted regression function, which is later tested (using the MSE metric) w.r.t. the ground truth. The method is trained on a large collection of randomly sampled task from the target family and is tested on a separate set of random tasks. The experiments include:
* sinusoidal wave prediction from a few samples
* MNIST and CelebA inpainting from a set of known pixel values (in 28x28 and 32x32 resolution respectively)
* additional experiments on heat equation and 2D Gaussian distribution task in Appendix
The experiments show that the proposed approach outperforms the other methods on the sinusoidal wave toy problem, and yet performs less good then than Kim et al. 2019 on MNIST and CelebA.
I propose to reject the paper in its current form, and consider the following negative points for further improvement:
1. The posed problem is not really few-shot learning, in (now classical) few-shot learning, such as few-shot classification on benchmarks such as miniImageNet, CUB, tieredImageNet, CIFAR-FS, FC100, etc. the meta-training is done on a disjoint set of categories and testing is done on a completely new set of categories unseen during training. The gap between disjoint visual categories is very large, and it does not come close to being tested on a different from training samples sinusoidal wave or different set of hidden pixels in inpainting on the same (seen during training) set of classes (where the basis function module could learn a set of basis functions for every class). In the proposed setting, I think a better definition would be "learning a structured regression" from a set of sample points to a function, and not few-shot regression.
If the authors would like to keep the "few-shot flavour", I would suggest re-formulating the experiments, and meta-train on some set of classes (e.g. inpainting over digits 0 to 4) and meta-test on a different set of classes (e.g. inpainting over digits 5 to 9). This partially holds for faces as they are all mostly different categories (different people), but in 32x32 resolution and MSE metric, I don't think they are sufficiently different.
2. I would expect stronger results on the more realistic MNIST and CelebA experiments (although as suggested in 1. the setting there should be different), currently it does less well then existing method.
3. An emerging important class of few-shot regression problems is few-shot object detection, where the bounding box coordinates of objects location need to be regressed. There are several papers and benchmarks in this space, and it will help the current paper to test on this challenging family of problems. Please see the following papers for benchmarks and settings:
* LSTD: A Low-Shot Transfer Detector for Object Detection, Chen et al. 2018
* RepMet: Representative-based metric learning for classification and one-shot object detection, Karlinsky et al. 2019
* Few-shot Object Detection via Feature Reweighting, Kang et al. 2019


Multi-agent reinforcement learning is a particularly challenging problem. Current methods have made progress on cooperative and competitive environments with particle-based agents using communication and centralized training. Little progress has been made on solutions that could operate in the real world with interaction, dynamics, and humanoid navigation strategies. In this work, we make a significant step in multi-agent models on simulated humanoid navigation by combining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning. Specifically, we develop a partial parameter sharing approach wherein the lower level of the hierarchy is shared enabling learning using decentralized methods. This drastically reduces the overall parameter space in the multi-agent problem and introduces structure in the optimization problem. We build on top of prior foundational work in learning goal-conditioned policies to learn low-level physical controllers for balance and walking. These lower-level controllers are task-agnostic and can be shared by higher-level policies. Overtop of these goal conditioned policies, we can train decentralized heterogeneous policies for multi-agent goal-directed collision avoidance. Surprisingly, our results show that with this combination of methods, RL techniques can be used for finding strong policies. A video of our results on a multi-agent pursuit environment can be seen here 1 .

Summary:
This paper looks at the MARL problem in high-dimensional continuous control settings. To improve learning in this multi-agent setting, they propose to pre-train a lower-level policy that takes as input foot-step goals and is executed for a fixed number of timestep, thereby simplifying both the learning and exploration.
I'm a bit unsure of how to evaluate this paper. On the one hand, I believe it has several contributions:
- Proposing a new MARL - continuous control environment
- Proposing a new lower-level policy for high-demensional continuous control environments, including how to learn it
- Using it to perform MARL in this environment
On the other hand, it is hard to say what the _main_ contribution is, which in turn makes it difficult to evaluate whether the experimental evaluation is sufficient:
Clearly, a main part of the paper is the work done to construct the hierarchical setup, including goal space, observation space and reward functions. However, this work, as far as I can tell, is separate from the MARL problem. Furthermore, there are several similar ideas already published, so comparison against those (for example by J. Peng, N. Heess or J. Mere) either as argument or even better as experiment, would be helpful to evaluate the quality of the proposed hierarchy.
On the other hand, there is the application of the hierarchical setup to the MARL problem. However, as far as I can tell, there is no difference between applying such a hierarchy to the MARL case and to the single agent problem. Especially if the lower-level component of the hierarchy is pre-trained in a non-MARL setup, it can just be seen as part of the environment from the point of view of the MARL training, offerring limited new insight into MARL.
I believe in the second paragraph of 4.1 the authors provide some insight into this matter, however, I have to admit I do not understand this paragraph:
- Why does temporal correlation reduce the non-stationarity of the MARL problem?
- Why does structured exploration reduce the number of network parameters that need to be learned?
- Why does partial parameter sharing make it easier for each agent to estimate other agents potential changes in behavior?
In summary, I think this is interesting work, but a clearer explanation of the relationship between HRL and MARL, as well as a clearer main argument, supported by experimental evidence, would greatly improve this paper.
Edit:
Thank you for your response.
Unfortunately, I don't feel like it sufficiently addresses my questions and concerns.
I do apologize if my original comment wasn't clear regarding the contribution part of the paper. What I was trying to say is not that I didn't see the individual contributions of the paper, but instead that the paper does multiple things simultaneously, without comparing against the relevant baselines for any of the individual contributions.
Regarding my questions: I understand where the temporal correlation is coming from in an HRL setting. However, what was not clear to me is how this reduces the non-stationarity of MARL.
I also understand that HRL can reduce the number of parameters, but I don't see how structured exploration reduces the number of parameters.
And lastly, I also can see how parameter sharing can simplify the learning, but I still don't see how it would allow agents to estimate the behaviour change of other agents easier.
I feel like in the paragraph in questions, a lot of causes and effects are mixed up and more careful descriptions of the benefits of the algorithm would help.
I want to re-iterate that I think that the submitted work by the authors is impressive and can provide valuable insights, but I believe it requires more work and more relevant baselines.


Despite an ever growing literature on reinforcement learning algorithms and applications, much less is known about their statistical inference. In this paper, we investigate the large-sample behaviors of the Q-value estimates with closed-form characterizations of the asymptotic variances. This allows us to efficiently construct confidence regions for Q-value and optimal value functions, and to develop policies to minimize their estimation errors. This also leads to a policy exploration strategy that relies on estimating the relative discrepancies among the Q estimates. Numerical experiments show superior performances of our exploration strategy than other benchmark approaches.

This paper studies the asymptotic properties of action value function $Q$ and value function $V$. Specifically, the authors assume that we can collect $n$ data points $\{(s_t,a_t,r_t(s_t,a_t),s_{t+1})\}_{t=1}^{n}$. Based on the collected data, the authors calculated the sample mean of the unknown reward function $\widehat r_n$ and transition probability $\widehat{P}(\cdot|s,a)$. They further defined an estimator $\widehat Q_n$, which is the fixed point of the empirical Bellman operator $\widehat{\mathcal{T}}_n$ derived from $\widehat r_n$ and $\widehat P_n$. The authors proved under certain assumptions that $\widehat Q_n\rightarrow Q^*$ almost surely. Based on this argument, they also derived a similar convergence of value function $\widehat V_n$ in distribution. Confidence intervals can also be established based on these results.
The proof of convergence in distribution is established based on the following idea: $\widehat r_n$ and $\widehat P_n$ converge to $r$ and $P$ by central limit theorem. By examining the Jacobian matrix of $\widehat{\mathcal{T}}_n$ with respect to $\widehat Q$, it is found that the implicit function $\widehat{ \mathcal{T}}_n \widehat Q -\widehat Q=0$ can be solved in small neighborhoods of $\widehat r_n, \widehat P_n$ and $\widehat Q_n$ as
$$
\widehat{Q}_n = \phi(\widehat{r}_n, \widehat P_n) \qquad\text{(implicit function theorem)}
$$
Then by Slutsky theorem, $\widehat Q_n = \phi(\widehat r_n, \widehat P_n)$ also converges to $Q^*$ in distribution. The proof is straightforward and easy to follow. However, I have some concerns about the above analysis.
1. The data $\{(s_t,a_t,r_t(s_t,a_t),s_{t+1})\}_{t=1}^{n}$ are collected by executing a fixed policy for n steps, which means the data points are not i.i.d. Therefore, the convergence of  $\widehat r_n$ and $\widehat P_n$ by central limit theorem may not hold ground, which further leads the later proofs potentially problematic.
2. The $\phi(\cdot,\cdot)$ function in implicit function theorem only exists in neighborhoods of $\widehat r_n, \widehat P_n$ and $\widehat Q_n$. It is unclear from the current proof that $\widehat Q_n$ fall into the same neighborhood of $Q^*$. Therefore, it needs to be justified that the limit point is $Q^*$.
3. Even for the empirical estimator $\widehat Q_n$, it is still the fixed point of $\widehat{ \mathcal{T}}_n$, which is hard to compute or solve. Thus Algorithm 1 may be inefficient in practice.
Other comments:
The font and margin of this paper does not conform the format requirement of ICLR.
What is $\sigma_R(i,j)$ in Theorem 1 and Corollary 1? It seems to be undefined.
I found it hard to read when the authors vectorize all the matrices and tensors to $N$ dimension vectors and $N\times N$ matrices. In particular, the rearranged notations such $\mu((i-1)m_a+j)$, $\tilde P^{\pi}((i-1)m_a+j,(i’-1)m_a+j’)$ are much more complicated that $\mu(i,j)$ and $\tilde P^{\pi}(i’,j’|i,j)$. Similarly, the mapping $F(Q’,r’,P’)$ defined in the proof of Theorem 1 can be represented by the Bellman operator $\mathcal{T}$ defined on page 3.
====post rebuttal==
I read other reviewers' comments and the author's response. I do not think the authors have addressed all my concerns. I will keep my rating.


Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprehensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize different parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigitMNIST, NYUv2 and SUN RGB-D datasets where we obtain competitive results.

The submission argues that when training the multiple objectives in a multi task learning framework, orthogonalizing gradients is beneficial toward reducing task competitions and efficient allocation of the learning capacity in the parameters.  This is demonstrated experimentally and subsequently a second order method is used for incorporating this in training.
Positives:
1. The concept is sensible, intuitive, and simple. There has a been quite few works (Sener and Koltun 2018) that similarly argue analyzing/regularizing the gradient direction when training multiple task objectives is beneficial. This submission reinforces those.
2. The implementation of the concept using a second order method is sensible and simple.
Weaknesses:
1. Missing comparisons: As reiterated above, several existing works have augmented training of neural networks with terms that are based on directions of gradients of multiple objectives. Some of them, e.g. Sener and Koltun 2018 or Du et al 2018, are very related. Though the submission cites them, no experimental comparison or convincing verbal critique of the differences are provided. The experiments could have included baselines that correspond to the concepts proposed in those prior papers to support the novelty of this submission. If the authors believe the concept of those papers are basically the same as theirs, then this submission should change to an analysis paper rather than appearing to pitch a new regularization term. I would have found an analysis paper as valuable as one with a novel method, but the stance should be clear.
2. I found the experiments too toy to be convincing. The MultiDigitMNIST is artificial and with limited benchmarking value as its construct doesn't really reflect the construct of multi task learning in the real world (e.g. as in the multi task vision datasets). NYUv2 dataset is more realistic, but the reported results are not clear to show significant and meaningful differences (see table 2). Particularly since the NYUv2 dataset has certain biases with imperfect ground truth from the kinect sensor which questions if differences in 0.001 range are meaningful. The authors can consider more recent and comparatively more reliable multi label datasets like Taskonomy for benchmarking.
3. Inline with the above comment, qualitative results or any other method for convincing that the achieved results (on datasets other than MultiDigitMNIST) is meaningful seems crucial.
Further comments:
4. The single task baselines in table 2 often perform inferior to the multi task baselines. This suggests the NYUv2 dataset could be too small to learn individual networks, whereas many other multi task papers often find single task baselines are hard to beat if they're not starved of parameters (e.g. see Standley 2018 "Which Tasks Should Be Learned Together in Multi-task Learning?"). Please clarify and/or consider using large enough datasets that allow you to benchmark under both high data and low data regimes.
5. why there is no single task baseline in table 1?
6. why CosReg is not included in figure 3 plot?
7. A closer look at the recent works on analyzing task competitions in multi task learning could be useful, and probably supportive of the concept of this submission. For instance standley 2018 Which Tasks Should Be Learned Together in Multi-task Learning seem to suggest that tasks that are related under transfer learning setting did not help each other in multi task setting. Their observation seems related to the pitch of this paper that orthogonal gradients (ie tasks with dissimilar updates) can be optimized better.
8. In page 4 authors state that computing the regularization term for all layers is complex, hence they do that only for the last layer. This seems okay to me, though I would have found an experiment demonstrating the consequences of this simplification useful (e.g. am experiment showing which layer to pick and a one-time expensive experiment demonstrating that picking one layer vs more layers is not too damaging).
-------
Comments after rebuttal:
I read the rebuttal and appreciate authors responses and the attempted improvement. It helped. I think the submission has the potential to be ultimately a useful read for the community, but at this stage more work/revision that wouldn't fit rebuttal time constraints would be needed to achieve that. I still think an experimental comparison with the related methods would be more convincing than verbal.
The added experiment on SUN RGB-D dataset is appreciated, but all results appear too close (table 2) to suggest one should adopt the proposed method. Also SUN RGB-D limits the tasks that could be learned together in a multi-task experiment given the limited number of labels per datapoint. I would have found using more recent  datasets with higher number of labels (rather than forcing to use depth+semantics which doesn't have to be a good mix, see Standley 2018) and more significant quantitate improvements more convincing.
Overall I lean toward not accepting the current submission but acknowledge the potential value in resubmission after further work.


We propose and investigate the design of a convolutional layer where kernels are parameterized functions. This layer aims at being the input layer of convolutional neural networks for audio applications. The kernels are defined as functions having a band-pass filter shape, with a limited number of trainable parameters. Building on the literature on this topic, we confirm that networks having such an input layer can achieve state-of-the-art accuracy on several audio classification tasks. We propose new filter functions and explore the effect of different parameters on the network accuracy and learning ability. This approach reduces the number of weights to be trained and enables larger kernel sizes, an advantage for audio applications. Furthermore, the learned filters bring additional interpretability and a better understanding of the data properties exploited by the network.

This paper proposes to specify the first layer of a CNN for audio applications with predefined filterbanks from the signal processing community. Those latter are only specified by a limited number of parameters, such as the bandwidth or the central frequency of the filter, and those parameters are then optimized through the standard back-propagation algorithm. Some accuracy improvements are obtained on non trivial datasets.
I think there are a lot of interesting ideas and the numerical improvements seem consistent with the method. However, I find that this study would benefit of more careful comparisons to understand which particular component is responsible for some of their success! Also, I think some relevant papers are missing in the introduction.
Pros :
- Good numerical performances.
- Interesting study of the impact of predefined filters; an analysis at the end of the paper(bandwidth, principal frequency chosen by the algorithm) is shown, which is a positive aspect of the paper.
Cons :
- Several attempts to employ hybrid architectures (as defined in the text) have been already proposed. References to hybrid architecture from Mallat's group are missing, e.g.:
 https://arxiv.org/abs/1809.06367/ https://arxiv.org/abs/1605.06644 . Another line of work concerns the steerable filters, which is another manner to parametrized the filters and learn them (e.g.: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Jacobsen_Structured_Receptive_Fields_CVPR_2016_paper.pdf ) Another manner could be to directly learn the filters as wavelets: https://arxiv.org/pdf/1811.06115.pdf . I agree some of those references are only considering images, but those methods are definitely not specific to them.
- Comparing the number of parameters of hybrid and non hybrid architectures is meaningless in this setting, as in all the experiments, the number of parameters of the layers above the first layer are kept identical: one only sees the difference due to the first layer, whose kernel is indeed relatively high-dimensional.
- Also, my understanding is that the general pipeline is slower: indeed, a parameter update aims to compute @f/@w=@f/@x*@x/@w. The computation of the term @f/@x is unavoidable and is identical to its non-hybrid counter-part. However, @x/@w might be sometimes higher because the computations can involve potentially more complex functions(e.g., exponential, cos, sin contrary to linear functions). Would you mind to clarify this thought?(A small fair timing comparison would be welcome!)
- Furthermore, the improvement in performances is clearly thanks to those a-priori incorporated. As stated in the text, many works propose to initialize the CNN with a specific filter bank. Have the authors tried to compare their performances if the first layer is simply initialized with those filters and then freely evolve? I feel this is missing and would make the claim of the paper stronger. If this has been already done, please highlight it in the text.
- Abstract: it is claimed that this technique leads to a training speedup (i.e., less epochs) but I do not understand where this is shown.
- Section 3: Sometimes(e.g., AudioMnist), the hybrid training pipeline is quite different from the original implementation, for instance, because of the use of ADAM when the original implementation was using SGD. Did the use of a different optimizer affect the performances?(e.g., SGD?)
- Section 3.4: Why not comparing with data augmented settings?
Suggestions of improvement:
- I would have liked to see a littlewood-paley plot (eg, the sum of the modulus of the filters in the frequency domain) to understand better the distribution of the filters in the Fourier domain, in particular w.r.t. the high-frequency.
- "the output maybe too redundant"(page 5) - I don't understand why this would be an issue? In this case, the network should decide which coefficients to discard if the classifier is good enough, shouldn't it?
Post-discussion:
R1 made several relevant comments about the technical novelty and my concerns weren't fully solved. Thus I decided to maintain my score.


Deep Reinforcement Learning (DRL) has led to many recent breakthroughs on complex control tasks, such as defeating the best human player in the game of Go. However, decisions made by the DRL agent are not explainable, hindering its applicability in safety-critical settings. Viper, a recently proposed technique, constructs a decision tree policy by mimicking the DRL agent. Decision trees are interpretable as each action made can be traced back to the decision rule path that lead to it. However, one global decision tree approximating the DRL policy has significant limitations with respect to the geometry of decision boundaries. We propose MOËT, a more expressive, yet still interpretable model based on Mixture of Experts, consisting of a gating function that partitions the state space, and multiple decision tree experts that specialize on different partitions. We propose a training procedure to support non-differentiable decision tree experts and integrate it into imitation learning procedure of Viper. We evaluate our algorithm on four OpenAI gym environments, and show that the policy constructed in such a way is more performant and better mimics the DRL agent by lowering mispredictions and increasing the reward. We also show that MOËT policies are amenable for verification using off-the-shelf automated theorem provers such as Z3.

The paper proposes a method (MOET) to distillate a reinforcement learning policy  represented by a deep neural network into an ensemble of decision trees. The main objective of this procedure is to obtain an "interpretable" and verifiable policy while maintaining the performance of the policy. The authors build over the previously published algorithm Viper (Bastani et al, 2018), which distillates deep policies into a single decision tree using the DAGGER procedures, i.e. alternation of imitation learning of an expert policy and of additional data-sampling from the newly learned policy. In the VIPER algorithm decision trees are chosen because their structured nature allows to formally prove properties of the policy they represent when the environments dynamics are known and expressible in closed form.
The main contribution of the paper is the adaptation of VIPER to ensembles of trees, specifically to mixtures of experts, and an adaption of the mixture of experts algorithm to non-differentiable experts. The mixture of experts is extended to decision trees with an em-like two step procedures that alternatively trains the experts and their input dependent weighting function. The main reason for extending VIPER to multiple trees is to increase the representation power of the procedure and obtain more faithful approximations of the original policy. The choice of using a input-dependent linear mixture of trees model instead of a more classical aggregation procedure like random-forests or xg-boosts is instead due to the assumed interpretability of input-dependent linear combinations.
The results in the paper seems to demonstrate comparable or improved performance of MOET over viper, but also show the systematic inability of both methods in imitating the actual neural network policy in all the environments but CARTPOLE. The authors conclude showing that the formal methods proposed to verify the policy in (Bastani et al, 2018) can be extended to MOET as they only rely on the choice of using decision trees as a target model.
On overall the paper creates a consistent narrative grounded on the notions that decision trees and linear models are interpretable, neural networks are not, individual decision trees can only create limited decision boundaries. The authors frame their technical contribution (em-like training of decision trees mixtures of experts) as necessary for creating more sophisticate decision boundaries through ensembling, while keeping the interpretability of the model (the ensembling method is linear and the components are decision trees).
While the narrative is consistent and consequent technical work seems valid the paper has some flaws that put it in my opinion slightly below the acceptance threshold:
1) The notions that linear models and decision trees have the "interpretable" propertiy is simply assumed as a known fact in the scientific field, while it is actually a largely discussed point of contention. See as an example "Lipton 2017, The Mythos of Model Interpretability". The authors should explain specifically what they mean by interpretability and how the properties of decision trees and linear mixtures of experts are consistent with their definition.
2) On the overall paper (similarly to the narrative used for Viper) the authors speak of interpretability as if it was referred to the original deep neural network model while they only interpret the surrogate model. The lack of a formal definition of interpretable masks the problem of interpreting a model through the interpretation of a surrogate model, which would rely on some proof that links whatever is demonstrated on the surrogate to hold for the original model.
3) The presentation of the results focus on the improvement of MOET with respect to VIPER. As a results it is difficult to compare the performance of both procedures to the original neural network model. For Pong and Acrobot the surrogate models do not achieve comparable performance, nor seems to be good imitations of the original policy, the reasons and the implication of this fact are not discussed.
4) There should be a better connection with the knowledge distillation/model compression literature and in general the important idea that the same computation can be represented by different algorithms. (Born Again Trees, Model Compression, Distilling the Knowledge in a Neural Network, Policy Distillation, Born Again Neural Networks).  There is also some related novel literature mapping neural network policies and word models to minimal hidden-markov models that could be of interest to the authors (Learning Finite State Representations of Recurrent Policy Networks, Learning Causal State Representations of Partially Observable Environments).