With the rise in employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, timeseries data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in a novel way by proposing TSInsight 1 where we attach an auto-encoder with a sparsity-inducing norm on its output to the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty. The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves as a feature attribution method to boost interpretability. In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction. In contrast to most other attribution frameworks, TSInsight is capable of generating both instance-based and model-based explanations. We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy. Furthermore, we analyzed the set of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction. The obtained results advocate that TSInsight can be an effective tool for the interpretability of deep time-series models.

The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.
* Pros
	* The work provides extensive comparison to a battery of other methods for model prediction interpretation.
	* The method is conceptually simple and is easy to implement. It is also general and can be applied to any prediction model (though this is more property of the sparse auto-encoder).
	* Despite its simplicity and generality, the method is shown to perform well on average, though it sometimes performs significantly worse than simple baselines.
* Cons
	* The method itself is not explained very well. The authors use language such as “attach the auto encoder to the classifier”, which is a bit vague and could mean a number of things. It would be helpful if they provided either a formal definition of the model or a architectural diagram.
	* Though the quantitative evaluation is not entirely flattering, the authors should not be punished for providing experiments on many datasets. That said, if their contribution is then rather one of technical novelty, i.e. a sparse-autoencoder-based framework for time series interpretability, it would be helpful for them to
		* More formally define their framework / class of solutions
		* Provide a more in depth study of possible variants of the method (this is elaborated on in the “Questions” section)
		* More strongly argue the novelty of their method
	* The authors provide a discussion on automatic hyper-parameter tuning that seems a bit out of place in the main method section, since it is not mentioned much thereafter and is claimed to not bring benefits.
	* The qualitative evaluation made by authors is rather vague:
		* "Alongside the numbers, TSInsight was also able to produce the most plausible explanations”

* Additional Remarks
	* Why not train things jointly? Does this have to be done post-hoc? The authors state that they “should expect a drop in performance since the input distribution changes” -> so why not at least try fine-tune and study the effect of training the classifier with sparse representations end-to-end? Exploring whether things can be trained jointly, or in other configurations, might allow the authors to frame their work as more of a general technical contribution.
	* It would be nice to have the simple baseline of a classifier with a sparsity constraint, i.e.
		* I.e. ablate the reconstruction loss
I’ve given a reject because 1) the explanation of the method is not very precise and could be greatly improved, 2) the quantitative evaluation is not sufficiently convincing, given the lack of technical novelty), and 3) the qualitative evaluation is hand-wavy.


There have been recent efforts for incorporating Graph Neural Network models for learning fully neural solvers for constraint satisfaction problems (CSP) and particularly Boolean satisfiability (SAT). Despite the unique representational power of these neural embedding models, it is not clear to what extent they actually learn a search strategy vs. statistical biases in the training data. On the other hand, by fixing the search strategy (e.g. greedy search), one would effectively deprive the neural models of learning better strategies than those given. In this paper, we propose a generic neural framework for learning SAT solvers (and in general any CSP solver) that can be described in terms of probabilistic inference and yet learn search strategies beyond greedy search. Our framework is based on the idea of propagation, decimation and prediction (and hence the name PDP) in graphical models, and can be trained directly toward solving SAT in a fully unsupervised manner via energy minimization, as shown in the paper. Our experimental results demonstrate the effectiveness of our framework for SAT solving compared to both neural and the industrial baselines.

This paper investigates the well-studied problem of solving satisfiability problems using deep learning approaches. In this setting, the authors propose a neural architecture inspired by message passing operations in deep probabilistic graphical models. Namely, the architecture takes as input a CNF formula represented as a factor graph, and returns as output a set of soft assignments for the variable nodes in the graph. The internal layers of the architecture consist of propagation, decimation and prediction steps. Notably, decimation operations take an important role in learning non-greedy search strategies. Besides PDP operations, the architecture incorporates parallelization and batch replication techniques. The learning model is trained in a non-supervised way, using a cumulative (discounted) log-likelihood loss that penalizes the non-satisfying assignments returned by the algorithm.
Overall, the paper is relatively well-written and well-positioned with respect to related work. However, it is difficult to accept the paper in its current state: as explained below, it is difficult to understand how the PDP architecture effectively solves SAT problems, and experiments are not really conclusive.
Although the idea of using general message-passing techniques for learning to solve SAT problems is relevant, the overall architecture left me somewhat confused. In the SAT problem, we have a CNF formula, say $F$, and the goal is to predict whether $F$ is satisfiable or not. In the former case, the solver is required to additionally supply a model of $F$, that is, an assignment of variables to values satisfying $F$. However, unless I missed something, the PDP architecture returns as output a set of $T$ “soft assignments” for each input SAT instance, which leads to two major concerns:
* There is no final decision (SAT/UNSAT), so how can we predict the satisfiability of an instance $F$ with just a set of $T$ assignments? I guess that the PDP model will predict SAT (resp. UNSAT) if the resulting loss is small (resp. large) enough, but this is very unclear.
* Furthermore, the output set consists of “soft” assignments, as defined by (4). But a solver should return a "discrete" assignment mapping variables to values in $\{0,1\}$. So, how can we convert soft assignments to discrete ones? Are the authors using a rounding method?
The experimental results are a bit confusing too. First of all, which generator has been used for random instances ($4$-SAT) and pseudo-industrial instances? For the sake of reproducibility, this should be mentioned in the revised version of the paper. Furthermore, it seems that at first sight, PDP is competitive with Glucose, as illustrated in the left part of Figure 1. Notably, the performance of Glucose degrades as the ratio $M/N$ grows. But this is not surprising because its timeout is set to 3s. However, the right part of Figure 1 is telling another story: apparently, Glucose can solve all instances in less than 10s. So, for the sake of fairness, it would be legitimate to report curves (left part) using 10s per instance: this would highlight the behavior of PDP with respect to modern SAT solvers on random instances using reasonable timeouts for the UNSAT part. The idea of making experiments on pseudo-industrial instances is interesting, but the PDP algorithm trained on those instances (i.e. PDP-modular) is rapidly degrading as the ratio $M/N$ increases. In fact, the difference between PDP-4SAT and PDP-modular is not statistically significant for UNSAT instances. Finally, Glucose is clearly dominating PDP on pseudo-industrial instances, as it can solve all instances with just a timeout of 2s per instance.


We present a new approach for efficient exploration which leverages a lowdimensional encoding of the environment learned with a combination of modelbased and model-free objectives. Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty. We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space. One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy. We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.

The paper proposes an approach to exploration by utilizing an intrinsic reward based on distances in a learned, evolving abstract representation space. The abstract space is learned utilizing both model-free and model-based losses, and the behaviour policy is based on planning combining the model-free and model-based components with an epsilon-greedy exploration strategy. Learning the abstract representation space itself is based on a previous work, but the contribution of this paper is the utility of it to design the reward bonus for exploration by utilizing distances in this evolving representation space.
As it stands, I am leaning towards rejecting the paper, for the following reasons.
(1) while the idea proposed is interesting, the current work rather explores it in a limited manner which is unsatisfactory.
(2) I think the presentation of the bonus itself -- novelty search (Section 4), which is the core of the paper, is rather unclear. (3) The assumption of deterministic transition dynamics may be ignored in favour of games which seem to be our benchmarks, but the results presented for the control tasks, Table 1, are not statistically significant, and the paper is missing details about the architecture/sweep for the baselines experimented with.
(4) Parts of the paper is rather unclear/feels disconnected -- for instance, the interpretable abstract representation bit; this was a loss in the original work, and seems to be just mentioned arbitrarily here while the loss isn't really used (unless it is used, and not mentioned in the paper).
(5) Overall, the proposed reward bonus is a heuristic whose specific design choice isn't statistically shown to be useful (Ablation in Appendix), and the empirical results comparing to other methods are underwhelming.
Here are my main points of concern which I hope the authors address in the rebuttal:
(1) Designing reward bonuses to induce exploratory behaviour in the agent has seen a surge of publications in the Deep RL literature in recent years. The key property all these methods aim for is a bonus that pushes the agent to the boundaries of its current "known region", and then rely on the stochasticity due to epsilon-greedy to cross that boundary -- pushing this boundary further. While this is different from exploration to reduce uncertainty, it is nonetheless a reasonable approach leading to competitive policies when evaluated in deep RL. But a characteristic all these bonuses aim for is that they fade away with time -- for instance count-based bonus are inversely proportional to visit counts, or prediction error bonuses go to 0 as the prediction becomes more accurate. But what do these novelty bonuses converge to? Is it just a stationary value based on consecutive loss parameter (in which case the hope is they don't affect the external reward scale, they just shift it uniformly)?
(2) What exactly are the nearest neighbours? Is it a search based on the data in the buffer or is it a notion of temporal neighbours?
(3) If it's temporal, why would there ever be biased for some states -- "We do this in order..novel states".
(4) I was completely unable to understand the section in the Appendix which is making a case for the ranked weighting. If you have a succinct explanation for the heuristic it'd be great.
(5) Further, as a heuristic it is mentioned that l2 norm may not be effective if the dimensionality of the representation space is increased. So why the heuristic? I think it either needs more empirical validation, or a theoretical justification.
(6) While the evaluation scheme used in the paper to quantify the exploration of the behaviour policy is interesting -- y-axis of plots in Figure 4 for the Labyrinth task -- why/what exactly is the role of Figure 2? Is the interpretability loss used here? Is it to reason for utilizing e-greedy instead of a purely-greedy behaviour? I think this is a little unclear, and can be better clarified. Further, the distinction of primary and secondary features is interesting, but their clear demarcation is rather questionable in more complicated domains -- in the abstract space.
(7) Do you have a hypothesis for why the 1-step value functions are not sufficient for decision making in this simple domain -- labyrinth - with the abstract representations?
(8) If model-based algorithms get more steps to learn shouldn't model-free too? I'm not sure I understand the reasoning for the experiment design choice.
(9) Whats the architecture used for Bootstrap DQN? It needs to have multiple heads -- but based on the current architecture that doesn't seem likely.
(10) Are the extrinsic rewards ignored in learning -- "only focus on intrinsic rewards" (Section 6.2.2)? If they are for the proposed method, are they for the competitors too? If so why, and what is the reward for Bootstrap DQN?
(11) I think the Discussion section raises interesting points about interpretability and metric learning, but I do think the conclusions drawn are a little inflated.
(12) The ablation study in Section D of the Appendix is not statistically significant -- so why is wighted reward useful? Please comment.
(13) How would stochasticity in transition dynamics affect the abstract representation space? Discussing this would be very interesting.
(14) Learning curves for the control tasks?
Comments about typos/possible points of confusion:
(1) The last para in Section 6.1 -- discusses "open" labyrinth heat map, then what do we mean by learning the dynamics of the wall? There is no wall in open, right?
(2) In Section 4 -- I think x_{t+1} is an estimate from the unrolled model -- \hat{x}_{t+1}? Further, it would be helpful to mention that it is an estimate based on the learned model.
(3) n_freq is used in the pseudocode in the main paper -- but no mention of it to explain it is made in the main.
(4) Contrasting the work to existing literature would be useful (in the Related Work section; as opposed to summarizing existing work).
(5) buffered Q network --> target networks?


Modern complex sequential decision-making problem often requires both lowlevel policy and high-level planning. Deep hierarchical reinforcement learning (Deep HRL) admits multi-layer abstractions which naturally model the policy in a hierarchical manner, and it is believed that deep HRL can reduce the sample complexity compared to the standard RL frameworks. We initiate the study of rigorously characterizing the complexity of Deep HRL. We present a modelbased optimistic algorithm which demonstrates that the complexity of learning a near-optimal policy for deep HRL scales with the sum of number of states at each abstraction layer whereas standard RL scales with the product of number of states at each abstraction layer. Our algorithm achieves this goal by using the fact that distinct high-level states have similar low-level structures, which allows an efficient information exploitation and thus experiences from different high-level state-action pairs can be generalized to unseen state-actions. Overall, our result shows an exponential improvement using Deep HRL comparing to standard RL framework.

This paper proposes a new kind of episodic finite MDPs called "deep hierarchical MDP" (hMDP). An L-layer hMDP can be *roughly* thought of as L episodic finite MDPs stacked together. A variant of UCRL2 [JOA10] is proposed to solve these hMDPs and some results from its regret analysis are provided.
Pros:
1. The essential result (Theorem 4.1) on the regret bound of the proposed algorithm seems correct. I have not checked the proofs in detail but in part because it does not seem surprising and that a precise assessment is hindered by many typos (see Min2 and Con2).
Cons (in descending order of their weights in my decisions):
1. The proposed hMDPs do _not_ seem to capture important features or challenges in hierarchical RL. My understanding is that the transitions in hMDPs work _like_ a clockwork (more on this in Mis6), the algorithm interacts with the sub-MDPs at each layer in turns according to their fixed horizons H_l's. This structure is very rigid temporally and seems to exclude the mentioned example of autonomous driving: the number of decision steps between intersections would be fixed.
2. There are many (typographical) errors in both the text and mathematical expressions. Some of them are more severe than others hindering understanding.
3. Possible as a consequence of Con2, some quantities defined seem unclear or incorrect at worst. For example, the "standard regret" defined in (2) is an expectation, not a random variable as in convention.
4. There are some notable deviations from similar settings in prior works. They might be worthwhile innovations but their significance or motivations is omitted. For example, the rewards in hMDPs are defined as a function of the full state, i.e. in general not decomposable to rewards on the states of each layer, yet the analogy for hMDP is "L levels of episodic MDPs."
A non-exhaustive list of obvious mistakes/typos:
1. In the title, "Provably" -> Provable.
2. In the abstract, "often both" -> often requires both.
3. In Organization, "theoremm" -> theorems.
4. In Section 2, “between exploration” -> between exploration and exploitation.
5. Above Section 3, "carried" -> carried out.
6. Below (1), "amount reward" -> amount of reward.
7. The definition of horizon H is incorrect. Consider H_1 = 2 and H_2 = 3, the algorithm will interact with the sub-MDPs in the following order within one episode: 1, 1, 2, 1, 1, 2, 1, 1, 2. There are 9 steps not 6 = 2 * 3 as defined.
8. Section 3.3, "able accumulate" -> able to accumulate.
9. Section 3.3, the definition of V_h^\pi, there should be not \max.
10. (5), "H" -> H - h.
11. Section 6, "tabular R" -> tabular RL.
12. In References, "Posterior sampling for reinforcement learning: worst-case regret bounds" -> Optimistic posterior sampling for reinforcement learning: worst-case regret bounds.
13. In References, "Temporal abstraction in reinforcement learning" should be cited as a PhD thesis.
Some other possible errors/inconsistencies:
1. Related work listed regret bounds from prior works (the presentation closely mirrors that of [JABJ18]) assume an episodic MDP with non-stationary transitions, i.e. P_t ≠ P_{t'} in general. However, in 3.1 the transitions are stationary. Relatedly, regardless of the stationarity of the transitions, there may not be an optimal _stationary_ policy in an episodic MDP contrary to the claim in the paper.
2. Indexing seems inconsistent near the top of page 3. The initial state is s_0 but the trajectory starts with s_1.
3. Near the top of page 3, V_h^\pi and Q_h^\pi should sum from h'=h, not h'=1. I assume that the authors intend to define h-step values (to appear in the Bellman equations).
4. Section 3.3, what are the k's in the equations?
5. (6), what is n(k-1, e)?
Minor (factored little to none in my decision):
1. The claim in Introduction that some games "do not require high-level planning" while others do is highly speculative and vague. Note that any policy can be written a function with codomain in the primitive actions. In fact, many people thought to solve a game like chess or Go requires some temporal hierarchy (opening, mid-game, and end-game).
2. The comparison to running UCRL2 on hMDP ignoring the given structure seems weak. Given the knowledge of the particular clockwork-like structure of hMDP at each layer (horizons, states, actions), the natural attempt would be run O(L) copies of UCRL2, one for each sub-MDP (under different terminating states of the immediately lower sub-MDP). Frankly, in my understanding, that seems to be roughly what the authors propose as the solution (thus the results unsurprising). Moreover, it is not immediately clear that UCRL2 can apply to the proposed setting of hMDP without checking regular conditions like communicating (diameter being finite).
3. The claim that RL with options “can be viewed as a two-layer HRL” needs much elaboration if not correction. Note that in the former, primitive actions are always taken in the original MDP at consecutive steps.
4. There is a limited relevance to deep learning or deep RL central to the themes at ICLR, i.e. the general issue of representation. This work may be more suitable for other general ML venues.
Some suggestions
I agree with the authors' sentiment that our theoretical understanding of hierarchical RL is relatively limited. I applaud the authors' effort to address this limitation. But judging from this aim of advancing our theoretical understanding, I think the paper may be improved by
1. better articulating the motivations for hMDPs (concrete examples would help)
2. contextualizing hMDPs with respect to other well-known models such as semi-MDPs (technical and precise comparison would help).
To put it in a different way, it is unclear to the readers why we want to solve this special class of hMDPs and what does hMDPs have to do with the general issues in hierarchical RL. Technically, I feel that assuming episodicity seems against the spirit of hierarchical RL where subtasks are often delimited by their subgoals instead of durations.
In conclusion, I cannot recommend accepting the current article.
(To authors and other reviewers) Please do not hesitate to directly point out my misunderstandings if there is any. I am open to acknowledging mistakes and revising my assessment accordingly.
Post-rebuttal update:
Thank you for replying to my review and incorporating some of my suggestions into your revision. However, I found many concerns (and mistakes) unaddressed, such as Mis7. The use of driving in Manhattan as an example troubles me because even stopping for a traffic light seems to disrupt the fixed temporal hierarchy of decisions. In conclusion, I will maintain my recommendation.


Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful. This has not been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The results of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training.

Summary:
The goal of the paper is to improve generalization of RL agents to a set of known transformations of the observation.
The authors propose to explicitly include a term into the PPO loss function that incentivizes invariance to transformations of the environment which should not change the policy, in their case changing textures of walls.
The idea itself is straightforward but worth exploring, although it does make the fairly strong assumption that one has access to or knowledge of the transformation function of the environment that can be applied to the observations.
Overall, the paper is clearly written and easy to read.
I'm currently recommending rejection based on the experimental evaluation as I don't believe that, in their current form, they sufficiently show the utility of the method (see detailed comments below).
However, I'm happy to change my rating if some or all of my comments and questions are addressed.
My main concerns are with the experimental evaluation:
- I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results.
-I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 task is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set (for example in the appendix).
- Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? I.e. when training on 10 envs, does the transformation function produce only observations from those 10 or potentially from all 500 in the 'maximum' training set? Having access to all 500 would explain why the success rate for IR is constant across all number of training set sizes.
- I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large.
Edit:
Thank you for your response and your comments.
Overall I think this is interesting and very promising work. Consequently, I am raising my score to "weak reject". Below, I discuss several points how I think the paper could be improved.
Additional environment: I understand that Coinrun requires a significant amount of compute and that's not easily affordable for everyone. However, I do strongly believe that this work would profit a lot from additional experiments. For future versions of the paper, I do think Coinrun would be very interesting, but alternatively, maybe the Multiroom environment used in [1] might be an easier alternative? (With transformations being symmetry transformations?). Just an idea.
More environments leading to a deterioration of performance: Unfortunately, I still don't understand how more data can lead to more overfitting. I believe it would be useful to investigate this further, either to avoid an unfair comparison to PPO if there's some training instability that can be easily avoided. Or alternatively, if this result holds, this could be a very useful insight as well if the authors can explain why this is the case.
Good and constant performance of IR: I still find it surprising how constant the IR performance is independent of how many levels are being used. Providing more data/ablation studies to better support this result and provide further insight into IR would strengthen the paper a lot. Additionally/alternatively: Submitting the code for reviews to inspect would help here as well.
>> "We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .
In particular, if you can show some further support for this claim, I think this would strengthen the paper a lot.


As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action. Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our approach generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation on the relative expected reward of the action to be explained. The second downweights irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare our approach with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.

[score raised from weak reject to weak accept after rebuttal - on a more fine-grained scale, I would rate this paper now an accept (7), but not a strong accept (8), however since this year's scale is quite coarse, I'll stick with a score of 6]
Summary
The paper proposes a new perturbation-based measure for computing input-saliency maps for deep RL agents. As reported in a large body of literature before, such saliency maps are supposed to help “explain” why a deep RL system picked a certain action. The measure proposed in the paper aims at combining two aspects: specificity and relevance, which should ensure that the saliency map highlights inputs that are relevant for a particular action to be explained (specificity), and this particular action only (relevance). The paper shows illustrative examples of the proposed approach and two previously proposed alternatives on Chess, Go and three Atari games. Additionally the paper evaluates the method and the two previous alternatives on two interesting chess-tasks: chess-puzzles where human players were shown to be able to solve puzzles faster and with higher accuracy when given the proposed saliency map in addition, and evaluation against a curated dataset of human-expert saliency maps for 100 chess puzzles.
Contributions
i) Novel, perturbation-based saliency measure composed of two parts. Main idea of specificity is (similar to Iyer et al. 2018) to use State-Action value function (Q-function) with a specific action, instead of State-Value function only. Main idea of relevance is to “normalize” by taking into account change in Q-value for all other actions as well. Both parts are combined in a heuristic fashion.
ii) More objective/reproducible assessment of how saliency maps produced by different methods overlap with human judgement of saliency of pieces in chess. To this end: two experiments with human chess players/experts (puzzle solving, and expert-designed saliency maps).
Quality, Clarity, Novelty, Impact
The paper is well written and most sections are easily understandable, though for some parts it might help if the reader is quite familiar with Chess/Go. The proposed saliency measure seems to address some shortcomings of previously proposed measures - my main criticism is that the construction of the measure seems rather ad-hoc and heuristic. It would have been great to formally define specificity and relevance (e.g. in an information-theoretic framework as Sparse Relevant Information) and then derive a suitable measure that is shown to satisfy/approximate the formal definitions. At least, there is one ablation study that justifies parts of the heuristic construction to some degree.
The proposed measure seems to do reasonably well on board-game domains, in particular chess. However it might also be the case that the measure does particularly well for generating saliency maps for Stockfish (which is the agent that happens to be evaluated in the chess domain), which might be quite different from previously reported methods that have been designed for deep neural network RL agents. The illustrative examples on Atari and Go do not allow for a statistically significant judgement of the quality of the proposed method.
On a conceptual level, a bigger issue (of many saliency methods in general, but the criticism also applies to the paper) is that the “explanations” drawn from saliency maps are rarely evaluated rigorously. The paper makes a nice attempt by trying to establish some “ground-truth” saliency in chess using humans to increase the degree of objectivity, which I greatly appreciate. However, it remains unclear whether explanations that happen to coincide with human notions of saliency really are of higher quality for assessing how an artificial system makes its decisions. The main goal of explainability/interpretability methods must be to come up with testable hypotheses that tell us something about how the artificial system makes its decisions in novel/unseen situations. The goal is not to explain a decision mechanistically after the fact (which is trivial, given a deterministic, differentiable feed-forward system), but to come up with non-trivial explanations that extrapolate/generalize. Specificity and Relevance are probably important ingredients of such explanations, but I think it’s important to formalize them properly first. Currently I am in favor of suggesting a major revision of the work, but I am happy to reconsider my decision based on the rebuttal and other reviewers’ assessment. Having said that, I do want to reiterate that I think it is great that the authors included some ablation studies and measures of “usefulness” of the saliency method.
Improvements / Major comments
i) Formally define specificity and relevance (e.g. as sparsity and compression?). Ideally derive a saliency measure based on these formal definitions.
ii) Show how saliency maps (of the same situation) change when producing explanations for different actions. I assume that currently the illustrative examples only show the action with the highest Q-value, what changes when using e.g. a less likely action?
iii) Show that the saliency map stays roughly constant for seemingly irrelevant perturbations. In particular, using the chess-dataset with expert annotations, apply various kinds of perturbations to non-salient pieces (e.g. removing them, swapping them for other pieces and potentially moving them in irrelevant ways) and see whether the AUC stays roughly constant.
iv) Apply non-relevant perturbations to the salient pieces. I.e. take the same puzzle/scene and move it across the board, does the saliency-map move in (roughly) the same way.
v) The saliency method might be particularly suited for Stockfish (whose action-value estimates might be strongly influenced by human saliency and chess theory). See whether the method still produces good results for other chess agents (ideally trained without human heuristics or data). If this is hard to do for chess, think about a different application where this is easier.
vi) Add a section that discusses current shortcomings and caveats with the method, and saliency maps for explainability in general.
vii) (Experimental details) For each domain, please explain the perturbations that you used (removing pieces in Chess/Go, blurring pixels in Go, anything else e.g. replacing pieces?). In all experiments, was it always the action with the highest Q-value that was being explained?
Minor comments
a) Table 1: (add error-bars) What is the variance across players? Are the results for the proposed method statistically significant?
b) Chess saliency dataset. Are the expert saliency ratings binary? Why not have multiple degrees of saliency?
c) Would the Greydanus et al. 2018 approach deliver similar results when using a threshold to cut off low-saliency inputs?
d) Why is the saliency map in 3.4 binary (pieces are either salient or not)? How was the binarization threshold chosen? What would the non-binary saliency maps look like?
e) Please provide all experimental details (in the appendix) that are necessary to reproduce the experiments. Referring to a code-repository is not a replacement for describing the methods in detail.


Knowledge graph has gained increasing attention in recent years for its successful applications of numerous tasks. Despite the rapid growth of knowledge construction, knowledge graphs still suffer from severe incompletion and inevitably involve various kinds of errors. Several attempts have been made to complete knowledge graph as well as to detect noise. However, none of them considers unifying these two tasks even though they are inter-dependent and can mutually boost the performance of each other. In this paper, we proposed to jointly combine these two tasks with a unified Generative Adversarial Networks (GAN) framework to learn noise-aware knowledge graph embedding. Extensive experiments have demonstrated that our approach is superior to existing state-of-the-art algorithms both in regard to knowledge graph completion and error detection.

This paper proposes to provide a novel noise-aware knowledge graph embedding (NoiGAN) by combining KG completion and noise detection through the GANs framework. More specifically, NoiGAN repeatedly utilizes a GAN model to 1) approximate the confidence score for facts identifying reliable data (discriminator) and 2) generate more challenging negative samples (generator). Then, it uses this confidence score and negative samples to train a more accurate link prediction model. The authors validate the proposed model through several experiments.
This paper reads well and the results appear sound. I personally find the idea of incorporating confidence score into a link prediction model to achieve a more accurate model very interesting. Furthermore, the provided experiments support their intuition and arguments outperforming considered baselines.
As for the drawbacks, I find the baselines considered in this work outdated missing many SOTA and related works in link prediction and noise detection [1,2, 3, 4, 5]. Further, I believe this work needs more experimental results and an ablation study capturing different aspects of the presented method. My concerns are as follows:
•	Considering the existing reverse relation issue in FB15K and WN18, I suggest conducting the experiments on the FB15K-237 and WN18RR from [6] instead.
•	I suggest considering more recent link prediction models as baselines.
•	I am wondering if the only difference between NoiGAN and KBGAN [7] is incorporating the confidence score in the link prediction loss?
•	Considering the fact that NoiGAN repeatedly retrains GAN and link prediction model, I suggest providing a comparison of computational complexity.
•	I am wondering if NoiGAN can only work with pre-knowledge of noisy triples in KG? If not, why didn’t you report NoiGAN performance with 0% noise in Table 3?
•	I find utilizing few examples to evaluate the power of discriminator in distinguishing noisy triples (Table 4) not satisfactory at all. I suggest experimenting with more data and providing the per-relation breakdown performance of the discriminator.
On overall, although I find the proposed model quite novel and interesting, the paper needs more experimental results to validate the idea.

[1] Pinter, Yuval, and Jacob Eisenstein. "Predicting Semantic Relations using Global Graph Properties".
[2] Nathani, Deepak, et al. "Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs".
[3] Balažević, Ivana, Carl Allen, and Timothy M. Hospedales. "TuckER: Tensor Factorization for Knowledge Graph Completion".
[4] Sun, Zhiqing, et al. "Rotate: Knowledge graph embedding by relational rotation in complex space".
[5] Pezeshkpour, Pouya, Yifan Tian, and Sameer Singh. "Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications".
[6] Dettmers, Tim, et al. "Convolutional 2d knowledge graph embeddings.", AAAI-18.
[7] Liwei Cai and William Yang Wang. “Kbgan: Adversarial learning for knowledge graph embeddings”.


We present a modular neural network architecture MAIN that learns algorithms given a set of input-output examples. MAIN consists of a neural controller that interacts with a variable-length input tape and learns to compose modules together with their corresponding argument choices. Unlike previous approaches, MAIN uses a general domain-agnostic mechanism for selection of modules and their arguments. It uses a general input tape layout together with a parallel history tape to indicate most recently used locations. Finally, it uses a memoryless controller with a length-invariant self-attention based input tape encoding to allow for random access to tape locations. The MAIN architecture is trained end-to-end using reinforcement learning from a set of input-output examples. We evaluate MAIN on five algorithmic tasks and show that it can learn policies that generalizes perfectly to inputs of much longer lengths than the ones used for training.

This paper proposes a new variation of modular neural networks. Specifically, they proposed a new architecture for the neural controller that selects which module to use given the current state and history computations, and evaluated the architectures for five symbolic tasks. I'm leaning towards rejection because:
1. The story is confusing. Instead of framing algorithm induction as a program synthesis task, the paper told the story from the modular networks's view. This is quite confusing as most of work in this line of research usually assumes the modules are neural networks and are learned, while this paper uses user provided modules. This should then be framed as doing program synthesis with pre-specified primitive functions. This work hence is quite close to NPI [1]. A RL version of NPI is also recently proposed [2].
2. Lack of novelty. Continuing from the previous point, it seems it is not easy to distinguish the main idea of this work and the work of NPI. There are two differences. There's an architectural difference about how one deals with history. NPI uses a LSTM and this work uses the last time step's computation configuration as input to a FNN. It is unclear why one is better than the other? Such modification is also lack of motivation, unexplained in the paper. Another difference is about how one designs the tasks/environments. This work uses a working tape while NPI uses programming traces.
3. Lack of baseline comparisons. In the experiment section, no previous work are evaluated and compared to the proposed method, leaving one unknown how well the proposed method compares with previously known work.
That being said, there are still some merits in the paper. The currently known work for training NPI with RL [2] does not support argument predictions, while this work does. I do not know if this is due to the interface/environment that's designed for the tasks in the paper, but I recommend the authors to rewrite the story and clarify these differences more clearly. Also how general is this environment? For harder programming tasks, can one still uses tape? How scalable is this?
Minor point
- How do you use FNN to produce all actions of the controller? Do you use a different heads for different action output? Or do you use an autoregressive architecture? Have you tried both and compared?
[1] Neural Programmer-Interpreters.
[2] Learning Compositional Neural Programs with Recursive Tree Search and Planning.


In order to efficiently learn with small amount of data on new tasks, meta-learning transfers knowledge learned from previous tasks to the new ones. However, a critical challenge in meta-learning is the task heterogeneity which cannot be well handled by traditional globally shared meta-learning methods. In addition, current task-specific meta-learning methods may either suffer from hand-crafted structure design or lack the capability to capture complex relations between tasks. In this paper, motivated by the way of knowledge organization in knowledge bases, we propose an automated relational meta-learning (ARML) framework that automatically extracts the cross-task relations and constructs the meta-knowledge graph. When a new task arrives, it can quickly find the most relevant structure and tailor the learned structure knowledge to the meta-learner. As a result, the proposed framework not only addresses the challenge of task heterogeneity by a learned meta-knowledge graph, but also increases the model interpretability. We conduct extensive experiments on 2D toy regression and few-shot image classification and the results demonstrate the superiority of ARML over state-of-the-art baselines.

################################################################################
Summary:
The paper provides a interesting direction in the meta-learning filed. In particular, it proposes to enhance meta learning performance by fully exploring relations across multiple tasks. To capture such information, the authors develop a heterogeneity-aware meta-learning framework by introducing a novel architecture--meta-knowledge graph, which can dynamically find the most relevant structure for new tasks.
################################################################################
Reasons for score:
Overall, I vote for accepting. I like the idea of mining the relation between tasks and handle it by the proposed meta-knowledge graph. My major concern is about the clarity of the paper and some additional ablation models (see cons below). Hopefully the authors can address my concern in the rebuttal period.
################################################################################
Pros:
1. The paper takes one of the most important issue of meta-learning: task heterogeneity. For me, the problem itself is real and practical.
2. The proposed meta-knowledge graph is novel for capturing the relation between tasks and address the problem of task heterogeneity.
Graph structure provides a more flexible way of modeling relations.
The design for using the prototype-based relational graph to query the meta-knowledge graph is reasonable and interesting.
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results,  to show the effectiveness of the proposed framework. The newly constructed Art-Multi dataset further enhances the difficulty of tasks and makes the performance more convincing.
################################################################################
Cons:
1. Although the proposed method provides several ablation studies, I still suggest the authors to conduct the following ablation studies to enhance the quality of the paper:
(1) It might be valuable to investigate the modulation function. In the paper, the authors compare sigmoid, tanh, and Film layer. Can the authors analyze the results by reducing the number of gating parameters in Eq. 10 by sharing the gate value of each filter in Conv layers?
(2) What is the performance of the proposed model by changing the type of aggregators?
2. For the autoencoder aggregator, it would be better to provide more details about it, which seems not very clear to me.
3. In the qualitative analysis (i.e., Figure 2 and Figure 3), the authors provide one visualization for each task. It would be more convincing if the authors can provide more cases in the rebuttal period.
################################################################################
Questions during rebuttal period:
Please address and clarify the cons above
################################################################################
Some typos:
(1) Table 7: I. no sample-level graph -> I. no prototype-based graph
(2) 5.1 Hyperparameter Settings: we try both sigmoid, tanh Film -> we try both sigmoid, tanh,
Film.
(3) parameteric -> parametric
(4) Table 2: Origninal -> original
(5) Section 4 first paragraph:  The enhanced prototype representation -> The enhanced prototype representations
Updates: Thanks for the authors' response. The newly added experimental results address my concerns. I believe this paper will provide new insights for this field and I recommend this paper to be accepted.


The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills to find a solution to new problems. In this paper, we propose a novel deep reinforcement learning-based skill transfer and composition method that takes the agent's primitive policies to solve unseen tasks. We evaluate our method in difficult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible or exhibits high sample complexity. We show that our method not only transfers skills to new problem settings but also solves the challenging environments requiring both task planning and motion control with high data efficiency.

What is the specific question/problem tackled by the paper?
This paper addresses the hierarchical RL problem of combining multiple primitive policies (pi_1, …, pi_K) into policies for more complex tasks. Given a number of primitive skills and a new task within an environment, the paper aims to learn to pick and combine the primitives as needed to solve the new task. This problem statement is interesting and the method performs well on difficult tasks.
However, I argue for rejecting this paper because it lacks meaningful contributions to the field. I do not see how the method presented in the paper is more than RL over  hand engineered action spaces that are better for the tasks. While this improves results, we already know that for any task, there is some best action space for performing that task. This is why most HRL work aims to also find the primitive policies in additional to composing them. Is this any better than option-critic if the options are hardcoded to be the primitives? The experiment state that the option-critic method did not work, but did you give it access to the same primitives?
Summary:
The method presented in the paper is as follows: at each state s_t, the K primitives are queried for their action a_k ~ pi_k(s_t). Then, a biRNN reads in the actions in order from 1 to k. In parallel, the state and a goal are encoded by a network named “Decoder”. The encoded state and the hidden states of the RNN are used to output an attention weight over each primitive. Finally, the output action is the weighted combination of all the actions. The encoders and attention weights are trained with RL.
This method is evaluated on several mujoco tasks, such as making a cheetah jump hurdles by combining “forward” and “jump” primitives. Each environment has predefined primitives such as “forward” “left” “right” etc.
This method is compared against HIRO, which does not have access to the primitive policies. It is not surprising that hand engineering primitives helps performance.
Is the approach well motivated?
The general idea behind the approach is well motivated: using primitive skills to learn complex skills is a useful goal. The details of the method are strange.
I would like to see a better motivation and empirical justification for the biRNN. Why should the primitive’s action be encode in order? The ordering of the primitives is arbitrary and constant: a fully connected network could be used, or the attentions could be output entirely independently per primitive.
In fact, I do see not why the primitives’ actions need to be encoded at all. It would be much simpler for the encoder to look at (s_t, g_t) and output a discrete probability over the K primitives.
The ablations in 5.2 are for outputting actions directly rather than mixture weights. The paper would benefit from ablations where mixture weights are output but without the biRNN or without passing in the primitive’s actions.

Is the method well placed in the literature?
The main idea of predicting weights over multiple experts is not novel (see "Adaptive mixtures of local experts” from 1991). In the context of RL literature, we can interpret the primitive skills as actions directly, and then the method is performing basic RL over a better action space (the better actions being “go left”, “jump” etc. We can also interpret these as options, but unlike options a single primitive is not followed for multiple time steps with a termination condition. Functionally, this is equivalent to regular RL using domain knowledge to engineer the action space.


It has been observed (Zhang et al., 2017) that deep neural networks can memorize: they achieve 100% accuracy on training data. Recent theoretical results explained such behavior in highly overparametrized regimes, where the number of neurons in each layer is larger than the number of training samples. In this paper, we show that neural networks can be trained to memorize training data perfectly in a mildly overparametrized regime, where the number of parameters is just a constant factor more than the number of training samples, and the number of neurons is much smaller.

This paper proves that one can design a (shallow)neural network that with a mild amount of overparametrization (e.g. the number of datapoints n is roughly less than d^2 in Theorem 1), a second-order method can reach a global minimum. In general, I think this is an interesting direction. However, I have some doubts regarding the comparison to prior work (especially regarding the results on the two-layer network), as well as some technical details that need some clarification. Regarding the quality of the writing, it’s in general ok but there are lots of grammatical mistakes, the authors should pay more attention to this. I’m not giving a high score for now but I will reconsider my review once I hear back from the authors.
Comparison to Oymak & Soltanolkotabi 2019: my understanding is that this paper prove convergence to a global minimum for a neural network where the number of parameters is only twice the number of datapoints so aren’t your results “worse” in that sense? The text in the related work seem to say the opposite, so I’m rather confused by your statement, please clarify.
Landscape: I think this discussion is largely missing in the paper but another way to prove the same result would be to focus on showing that the loss surface is “well-behaved”. In fact the paper by Soltanolkotabi et al (see reference below) proves this for a similar network with a quadratic activation function and arbitrary data. Essentially, they show that such network obeys the following two properties:
There are no spurious local minima, i.e. all local minima are global.
All saddle points have a direction of strictly negative curvature
My understanding is that they prove this for a parametrization regime where n=d^2, isn’t this the same regime as proved in your results?
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks.
Three-layer neural net
1) This network uses activation function of the form x^p. You wrote “for some constant p”, is there any specific lower bound on p? I’m also surprised that one would allow large values of p as such functions have a saddle at x=0 with a large region with low gradient magnitudes around it.
2) Limitation quadratic activation function. You say “To address this problem, the three-layer neural net in this section uses the first-layer as a random mapping of the input“. How is this helping with changing the activation function?
3) You have to fix part of the weights of the network, this seems to be a limitation of the analysis that should be more clearly highlighted and better contrasted to what has been done in prior work.
4) I think it would be interesting for the reader to focus more on the three layer network (instead of the two-layer one whose analysis is rather simple) and provide a more detailed proof sketch.
Paper organization
The most interesting result of the paper is the one about the three-layer network but the entire analysis is relegated to the appendix. I feel it would be worth trying to provide a rough proof sketch in the main paper to highlight the difficulty of the analysis.
Proof Lemma 2
Alternatively to the current proof, couldn’t you differentiate the second term (2/n \sum_j (\sum …)^2 ) w.r.t. z and show it is zero at the argmax_z? Isn’t this what your result says?
Experiments
Consider repeating the experiments a few times and showing the average.
Minor: Low loss vs perfect fitting: I couldn’t find any discussion about this but the authors seem to assume that a zero loss directly implies that the training data is fit perfectly. In the case n=d, there exists a function that yields a perfect fit of the data and therefore a larger network should be able to represent this function. Perhaps it would be worth writing this down.
More minor comments
- Figure experiments: please use a log scale
- Formula top of page 16: should be z_k on the left and right of the term inside the bracket
- missing citation top of page 24: “e.g. in”


While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the meta-knowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the meta-knowledge or task-specific learning. We formulate this objective into a Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on two realistic task-and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework.

Summary
========
This paper introduces a mechanism for gradient-based meta-learning models for few-shot classification to be able to adapt to diverse tasks that are imbalanced and heterogeneous. In particular, each encountered task may have varying numbers of shots (task imbalance) and even within each task, different classes may have different numbers of shots (class imbalance). Further, test tasks might come from a different distribution than the training tasks. They propose to handle this scenario by introducing three new types of variables which control different facets of the degree and type of task adaptation, allowing to decide how much to reuse meta-learned knowledge versus new knowledge acquired from the training set of the given task.
Specifically, their newly introduced variables are: 1) the factor for learning rate decay (a scalar) for the inner-loop task adaptation optimization which allows to not deviate too much from the global initialization when insufficient data is available, 2) a class-specific learning rate (one scalar per class) that allows to tune more for under-represented classes of the training set, 3) a set of weights on the global initialization (one scalar per dimension) that can down-weigh each component if it’s not useful for the task at hand (e.g. if test tasks have significantly different statistics than training tasks did).
The values of these variables are predicted based on the training set of the task: the support set is encoded via a hierarchical variant of a set encoding (where pooling is done using higher order statistics too instead of simply averaging). The resulting encoded support set is the input to the network that produces the values for the three sets of variables discussed above. Each new variable is treated in a Bayesian fashion: a prior is defined over it (Normal(0,1)), which is updated by conditioning on the training set to form a posterior for each given task. Specifically, each of the above variables is represented by a Gaussian whose mean and variance are the learnable parameters that are produced by the network described above.
Experimentally, this method outperforms others on a setting of imbalanced tasks (the shot is sampled uniformly at random from a designated range). The gain over other methods is large in particular when evaluated on out-of-distribution tasks (coming from a different dataset) and when the imbalance is large.
Comments (in decreasing order of importance)
========================================
A) The Bayesian framework helps because it offers an elegant way to use a prior. In the deterministic version, was any effort made to resemble the effect of that prior? For example, one can define a regularizer that penalizes behaviors that ignore the meta-knowledge too much (e.g. too large values for \gamma, or for the class-specific learning rates etc). Albeit more ‘hacky’, if these regularization coefficients are tuned properly, they might result in a similar effect to that of having a prior. A fair comparison to the deterministic variant should include this.
B) I think that \gamma and z can be merged into a single set of parameters? In particular, imagine a per-dimension-of-\theta learning rate. This would then be large for a dimension when there is a larger need for adapting that dimension of \theta. In the case of large training sets, this can be large for all dimensions, recovering the behavior of a large \gamma. For the case of diverse datasets, this would behave as the current z (updates a lot the dimensions of \theta that are irrelevant for the given task due to the dataset shift).
C) Meta-Dataset (https://arxiv.org/abs/1903.03096) is a recent benchmark for few-shot classification that introduces both of what is referred to here as task imbalance and class imbalance and also is comprised of heterogeneous datasets and evaluates performance on some held-out datasets too. The current state-of-the-art on it (as far as I know) is CNAPs [1] which employs a flexible adaptation mechanism on a per-task basis but is fully amortized (performs no gradient-based adaptation to each task) and makes no explicit effort to tackle imbalanced tasks as is done here. I’m curious how this method would compete in that setup. It definitely seems to be a strong candidate for that benchmark!
Less important
=============
D) which dataset is used in Tables 4 and 5? I assume it’s Omniglot (due to the numbers being in the 90s) but it would be good to say this explicitly.
E) In section 5.2, expressions such as x5 and x15 are used to characterize the degree of imbalance of a task. How exactly are these computed? Does x5 mean that the largest shot is 5 times larger than the smallest shot? It would be good to explicitly state this.
F) In the Related Work section, in the Meta-learning paragraph there is a sentence that’s not accurate: “Metric-based approaches learn a shared metric space [...] such that the instances are closer to their correct prototypes than to others”. This sentence does not describe all metric based approaches. It describes Prototypical Networks (Snell et al) but not, for example, Matching Networks (Vinyals et al) nor many others that like Matching Networks perform example-based comparisons and don’t aggregate a class’ examples into a prototype.
In a nutshell
===========
I think this work is a useful contribution for moving towards a more realistic setting in few-shot classification. It captures some desiderata of models that can operate in more realistic settings and outperforms previous models in those scenarios. My comments above are mostly suggesting improvements and clarifications but I am inclined to recommend acceptance of this paper.
References
=========
[1] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.


Imitation learning has emerged as a powerful strategy for learning initial policies that can be refined with reinforcement learning techniques. Most strategies in imitation learning, however, rely on per-step supervision either from expert demonstrations, referred to as behavioral cloning (Pomerleau, 1989;1991) or from interactive expert policy queries such as DAgger (Ross et al., 2011). These strategies differ on the state distribution at which the expert actions are collected -the former using the state distribution of the expert, the latter using the state distribution of the policy being trained. However, the learning signal in both cases arises from the expert actions. On the other end of the spectrum, approaches rooted in Policy Iteration, such as Dual Policy Iteration (Sun et al., 2018b) do not choose next step actions based on an expert, but instead use planning or search over the policy to choose an action distribution to train towards. However, this can be computationally expensive, and can also end up training the policy on a state distribution that is far from the current policy's induced distribution. In this paper, we propose an algorithm that finds a middle ground by using Monte Carlo Tree Search (MCTS)  to perform local trajectory improvement over rollouts from the policy. We provide theoretical justification for both the proposed local trajectory search algorithm and for our use of MCTS as a local policy improvement operator. We also show empirically that our method (Policy Optimization by Local Improvement through Search or POLISh) is much faster than methods that plan globally, speeding up training by a factor of up to 14 in wall clock time. Furthermore, the resulting policy outperforms strong baselines in both reinforcement learning and imitation learning.

[Summary]
This paper proposes POLISH, an imitation learning algorithm that provides a balance between Behavioral Cloning (BC) and DAgger. The algorithm reduces the mismatch between the target policy and an expert policy on states obtained from starting at the target policy's state distribution and following the expert policy for a time segment of t steps. The claim is that a suitable t will keep the training states close to the target policy's state distribution and avoid the compounding errors that arise when the agent drifts away from its training distribution. The paper also explores the possibility of policy optimization by replacing the pre-defined expert policy in POLISH with a policy derived from Monte Carlo Tree Search. Theoretical and empirical analyses in the paper studies the effect of t and MCTS planning in POLISH on policy improvement.
[Decision]
A clear study of the dimension between BC and DAgger is a useful contribution to the literature and an algorithm that effectively solves the distributional shift problem in BC will be of high practical value. However, the results in this paper do not support the claim that a reasonable time segment length in POLISH alleviates this problem. The theory shows a bound on the performance of the target policy that varies with t, but it is not clear if a suitable t is better than the two extremes, i.e., BC and DAgger. The experiments section is limited and, on two out of the three tasks, there is no considerable difference between the performance of POLISH, BC, and DAgger. I am leaning towards rejecting this paper.
[Explanation]
In Section 5, Theorem 1 shows the effect of t, the length of time segments, on the performance and the target policy by providing a bound. If this theorem is motivating a middle ground between BC and DAgger, it needs to show that a value of t other than the extremes will maximize the bound. The bound in the paper consists of a positive term and a negative term, both of which grow with t. It is then concluded that a balance point will maximize the overall bound. I do not see how it follows that this balance point is a middle ground and not an extreme value. If, for example, the negative term grows faster than the positive term, then DAgger (t=1) will be have the best performance according to this theorem.
It is not clear how the bound in Theorem 1 is comparing the performance of algorithms with different values of t. For a fixed policy, one can obtain different bounds by choosing different values of t. These bounds will have different values of \epsilon. The state distribution for \epsilon is the distribution of states visited in a limited time segment (which depends on the target policy, the expert policy, and the segment length). Using \epsilon (or \epsilon_i) in the bound drops the relationship between \epsilon and the length of time segments, and hides the fact that different algorithms are minimizing different errors.
The equality in Eq 2 is not obvious to me. J(\pi) is and expectation under the discounted visit distribution. For example, if gamma is small, then the states in the start state distribution will have a higher weight in J(\pi) while the right-hand side sums over the expected performance in all time segments equally. I believe the later terms in the sum should also be discounted.
Does changing t also affect the MCTS step and the expert policy obtained from it? If so, it is possible that a suitable t will result in better performance because the MCTS step finds a stronger expert policy, and not because the imitation learning step better reduces the error.
Does the MCTS step use the perfect simulator or a learned model in the experiments? If POLISH, unlike PPO, has access to the MDP, the comparison of these two methods is not fair.
In Fig 2 (a) and (c), the curves for t=1, t=32, and t=1000 overlap through most of the training process. This is not conclusive evidence that a sweet spot for t results in better performance. An experiment on a simpler setting with more runs may elucidate the effect of t on the performance.
In 6.3, what does the reward improvement after running MCTS with the current policy precisely mean, and how does this correspond to the first term in the bound, i.e., the sum of the expected performance of \pi_* over time segments?
[Minor comments]
- I suggest adding the process of obtaining an expert policy through MCTS to Algorithm 1. It is hard to understand the process without a clear step-by-step description.
- How is t=32 chosen for the experiments in Fig 2?
-------------------
After rebuttal: I have read the authors' response and the other reviews. The rebuttal addresses my questions and concerns about clarity of presentation. However, I am still not convinced by the evidence in this paper. On two out of the three environments, the performance of the three values of t is not much different through the training and this is not because of the advantage over PPO. On Ant, for example, the curves for t=32 and t=1000 are not even one std apart through most of the training.


With massive possible synthetic routes in chemistry, retrosynthesis prediction is still a challenge for researchers. Recently, retrosynthesis prediction is formulated as a Machine Translation (MT) task. Namely, since each molecule can be represented as a Simplified Molecular-Input Line-Entry System (SMILES) string, the process of synthesis is analogized to a process of language translation from reactants to products. However, the MT models that applied on SMILES data usually ignore the information of natural atomic connections and the topology of molecules. In this paper, we propose a Graph Enhanced Transformer (GET) framework, which adopts both the sequential and graphical information of molecules. Four different GET designs are proposed, which fuse the SMILES representations with atom embedding learned from our improved Graph Neural Network (GNN). Empirical results show that our model significantly outperforms the Transformer model in test accuracy.

This paper focuses on the retrosynthesis prediction problem which to my understanding is the factorization of a target molecule into simpler structures. Previously, retrosynthesis prediction has been tackled as a language translation problem by using a Seq2Seq algorithm called "Transformer". This sequential approach was possible because molecules can be expressed as strings using the following format: Simplified Molecular-Input Line-Entry System (SMILES).
Despite its good performance, language translation methods ignore the graphical structure of molecules. This paper proposes to add a graph neural network in front of the Seq2Seq/Transformer network to exploit the graphical structure, hence the name “Graph Enhanced Transformer (GET)”. The Graph Neural Network considered in this work also uses an attention mechanism similarly to Graph Attention Networks.
The main contribution is the addition of a Graph Neural Network to a Seq2Seq model for retrosynthesis prediction which provides state of the art results.
From a machine learning / representation learning perspective, I do not consider the paper is innovative enough. Even so, the paper shows strong results for retrosynthesis prediction, I guess it would be a good fit in a chemistry conference.
Things to improve:
There are plenty of works related to graph Autoencoders and Graph generative models that are not mentioned in the related work. It would be great to dedicate a section for them. Here some examples:
- Li, Y., Vinyals, O., Dyer, C., Pascanu, R., & Battaglia, P. (2018). Learning deep generative models of graphs.
-Simonovsky, M., & Komodakis, N. (2018, October). Graphvae: Towards generation of small graphs using variational autoencoders.
-De Cao, N., & Kipf, T. (2018). MolGAN: An implicit generative model for small molecular graphs.
-You, J., Ying, R., Ren, X., Hamilton, W. L., & Leskovec, J. (2018). Graphrnn: Generating realistic graphs with deep auto-regressive models.
All experiments are compared to author’s re-implementations of other works. It would be interesting to directly report other work accuracies.


In this paper, we introduce Symplectic 1 ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system, given by an ordinary differential equation (ODE), from observed state trajectories. To achieve better generalization with fewer training samples, SymODEN incorporates appropriate inductive bias by designing the associated computation graph in a physicsinformed manner. In particular, we enforce Hamiltonian dynamics with control to learn the underlying dynamics in a transparent way, which can then be leveraged to draw insight about relevant physical aspects of the system, such as mass and potential energy. In addition, we propose a parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data is embedded in a high-dimensional space or we can only access velocity data instead of generalized momentum. This framework, by offering interpretable, physically-consistent models for physical systems, opens up new possibilities for synthesizing modelbased control strategies.

In this paper, the authors propose a framework for learning the dynamics of a system with underlying Hamiltonian structure subjected to external control. Based on the extended equations of motion, the authors suggest how to apply NeuralODE in a way that makes use of the prior information that the unconstrained system is Hamiltonian and subjected to a control term. For a range of tasks, the authors then demonstrate that the proposed SymODEN framework can learn the dynamics and recover the known analytical solution, and that they can derive a controller that allows them to drive the system to a target configuration.
I find this paper very interesting and the formulation elegant. In particular, I appreciate that the paper is pretty much self-contained and that the authors derive the theory from first principles. However, there are a couple of points (listed below) that should be addressed prior to publication to improve the clarity of the paper, and to help the reader to fully appreciate the depth of the experimental section. If these points were addressed in sufficient detail, I would be willing to increase the score:
1) Reading the abstract and the introduction, I got the impression that SymODEN can be applied to any physical system while the method is in fact only applicable to systems governed by Eq. (11). I think it’s worth mentioning in the text that not every physical system follows (constrained) Hamiltonian Dynamics.
2) You mention that a controller is designed based on the learned dynamics. For example, in the first sentence in Sec. 2.2: ‘Once the dynamics of a system have been learned, it can be used to synthesize a controller to maneuver the system to a reference configuration q*’. I think it is important to specify here which dynamics you are referring to, i.e. the constrained or unconstrained dynamics. Later on it becomes clear that you use constant-u training data and that u is part of the input to the model, so it has to be constrained dynamics, but at this stage it is still unclear to the reader.
3) In Eq. (11), you set du/dt = 0 without motivating this restriction. Can you provide at least one sentence on why this is an interesting choice and back it up with a reference? Furthermore, the sentence above Eq. (11) is broken and needs fixing.
4) Tasks (general): You consider a range of tasks and I appreciate that you start with a simple and intuitive system. However, I think a bit more guidance throughout the tasks section would be very helpful. In particular, I would suggest that you provide a short (one or two sentences) summary at the beginning of each task to say what exactly it is that you are trying to test or demonstrate. Furthermore, I would suggest showing the summary of  results, i.e. Sec. 4.2, after you introduce the individual tasks rather than before.
5) Task 2: This task addresses multiple things in one go: Initially, you demonstrate that you can recover the results of Task 1 without access to the generalised momenta, and explain why this can only be done up to a constant scaling factor. This is very interesting and clear. However, then you jump straight into the controller and things become a little unclear because, at this stage, it still seems that the dynamics are the same as in Task 1, i.e. unconstrained. Please add a sentence or two for clarification. Another important aspect that is not commented on at all is the behaviour of u(t) in Eq. (27). In particular, isn’t u(t) expected to satisfy du/dt = 0 based on Eq. (11)? The results in Fig. 6 suggest that this is not the case (see time interval [2, 6]). This seems like an interesting and surprising behaviour, especially because SymODEN was only trained with constant-u training data. I would appreciate if the authors could comment on this. I would also suggest to add horizontal lines to Fig. 6 to indicate the expected results.
6) Task 4: Why did you not explain this task in a dedicated section like you did for all other tasks?
7) Symplectic: Since both the method and the title of the paper contain the word ‘symplectic’, it would be good if you explained what the term actually means.
8) ‘Our results show that incorporation of such physics-based inductive bias can provide knowledge about relevant physical properties (mass, potential energy) and laws (conservation of energy)...’. To me, this statement is slightly misleading. You did not demonstrate that SymODEN ‘provides knowledge’ of laws of the system; energy conservation (for u = 0) as a law is hard-coded into your network. The specific value of the energy can be inferred but that I would consider a physical property. I would suggest to change the wording to reflect this clearly.
9) Introduce the acronym ODE much earlier than in Sec. 3.1.
10) Model training: What happens if you use unseen initial conditions rather than the ones in the training data? Perhaps you could add a comment to clarify.
11) There are many typos and grammar mistakes in the paper. Please revise it carefully. To give you a few examples:
‘are both reformulation’ -> ‘are both reformulations’
Sec 2.1: Decide on whether you use plural or singular for ‘dynamics’ and be consistent.
‘on a equal footing’ -> ‘on an equal footing’
‘beyond classical mechanics, the Hamiltonian’ -> ‘Beyond classical mechanics, Hamiltonian …’
‘Hamiltonian is same as’ -> ‘Hamiltonian is the same as’
‘represents potential energy’ -> ‘represents the potential energy’
‘trajectory actually converge to’ -> ‘trajectory actually converges to’
‘a ODE solver -> ‘an ODE solver’.
‘Lagrangian and Hamiltonian formulation’ -> ‘Lagrangian and Hamiltonian formulations’
‘assume that q and p evolves’ -> ‘assume that q and p evolve’
‘translational coordinate’ -> ‘translational coordinates’
‘naive baseline model approximate’ -> ‘naive baseline model approximates’
etc.
*************************************
The authors addressed my comments and answered  my questions clearly.  I therefore increased the score. *************************************


Point clouds, as a form of Lagrangian representation, allow for powerful and flexible applications in a large number of computational disciplines. We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong flickering, and easy solutions to suppress this flickering can result in undesirable local minima that manifest themselves as halo structures. We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach.

The paper tries to learn temporally stable representations for point-based data sets and focus on varying size and dynamic point sets, and demonstrate its usefulness in the context of super-resolution. To deal with a difficult target that dynamically moves and deforms over time with variable input and output size, they take a novel temporal loss function for temporally coherent point set generation and siamese network setup for temporal loss calculation. Their novel temporal loss is based on EMD to minimize differences between an estimated point cloud and a desired super-resolution point cloud. The discussion and evolution on multiple loss functions are mostly well done. Except spatial loss is considered, taking the ground truth acceleration and estimated velocity into account is beneficial to this task. Their main contribution is taking permutation invariant loss terms and a siamese training setup and generator architecture, enabling improved output variance by allowing for dynamic adjustments of the output size, and identifying a specialized form of mode collapse for temporal point networks.
They perform an empirical study of their temporal loss function on the generated data set and apply the proposed method to some complex 3D models to conclude the superior performance of temporal loss formulation in contrast to previous work.

Overall, this paper has some significant points on point cloud super-resolution, with the caveat for some clarifications on the theory and experiments. Given these clarifications in an author response, I would be willing to increase the score.

When the input moves slowly enough, the point cloud can be considered static. Can the proposed temporal loss outperform other works under this condition?

Only one previous work PU-Net based on PointNet++ is compared in the paper, I would like to see more discussion on applying the proposed temporal loss with other point-based algorithms.

I am very curious about the effect on different choices of weighting terms hyperparameters in temporal loss and predefined upsampling factor r.


It is well-known that overparametrized neural networks trained using gradientbased methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume that the activation function is ReLU or they depend on the minimum eigenvalue of a certain Gram matrix. In the latter case, existing works only prove that this minimum eigenvalue is non-zero and do not provide quantitative bounds which require that this eigenvalue be large. Empirically, a number of alternative activation functions have been proposed which tend to perform better than ReLU at least in some settings but no clear understanding has emerged. This state of affairs underscores the importance of theoretically understanding the impact of activation functions on training. In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks. A crucial property that governs the performance of an activation is whether or not it is smooth:
• For non-smooth activations such as ReLU, SELU, ELU, which are not smooth because there is a point where either the first order or second order derivative is discontinuous, all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data.
• For smooth activations such as tanh, swish, polynomial, which have derivatives of all orders at all points, the situation is more complex: if the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training. But if the dimension is large and the data satisfies another mild condition, then the eigenvalues are large. If we allow deep networks, then the small data dimension is not a limitation provided that the depth is sufficient.
We discuss a number of extensions and applications of these results.

Summary: The authors of the paper examine how different activation functions affect training of overparametrized neural networks. They do their analysis in a general way such that it includes most activation functions such as ReLU, swish, tanh, polynomial, etc.
The main point of their analysis is that they examine a matrix called the G-matrix which is described in equation (2), and this (positive semi-definite) G-matrix can determine the rate of convergence to zero of the training error. Namely, the minimum eigenvalue of the G-matrix is inversely proportional to the time required to reach a desired amount of error (Theorem 3.1 (Theorem 4.1 from Du et al. (2019a)).
The main results separate into two cases: (1) activation functions with a kink (i.e. if the activation function is NOT in C^{r+1}, the space of r+1 continuously differentiable functions, for some finite r) and (2) smooth activation functions.
In the first case, the authors show that the minimum eigenvalue of the G-matrix is large, i.e. bounded away from zero after a few assumptions.
In the second case, the authors show that polynomial activations have many zero eigenvalues, and sufficiently smooth activations such as tanh or swish have many small eigenvalues, if the dimension of the span of data is sufficiently small.
The author’s initial problem setup works on a one-hidden-layer neural network where only the input layer is trained, but provide some extensions in the appendix.
The authors also provide some empirical experiments: one synthetic data, and on CIFAR10. The synthetic data experiments agreed with theory, but the experiment on CIFAR10 did have some gap between theory and experiment, although the CIFAR10 with ReLU experiment agreed with theory.
Stengths: I appreciate the author’s effort in providing needed theoretical analysis on how activation affects training error for deep neural networks. The authors also provide an extensive appendix that provide seemingly full proofs of the theorems (although this reviewer did not go into detail for most of the appendix). The authors also provide experiments that confirm the theory and also provide examples highlighting the gap (which this reviewer sees as a strength).
Weaknesses: A clear weakness of this paper is that the appendix is too long. The authors do provide a proof sketch of the main results and refer to the appendix, but I would have liked to have seen a more focused paper. Having extensions of the main results is nice, but it’s sometimes unclear what is being extended. Perhaps a list of extension would make clear what’s in the appendix.
Other comments: (i) I’d like to an explanation to why it’s called the DZXP setting.
(ii) On page 4, when explaining the M matrix, I think it should be grad_W F (instead of L)
(iii) On page 4, I think there is more to assumption 1, after looking at Allen-Zhu et al. (2019) (https://arxiv.org/pdf/1811.03962.pdf page 4, footnote 5)
(iv) the wording from page 4, “the matrix G^(t) does not evolve from its initial value G^(0)” is a bit awkward. Do you mean G^(t) does not change much from G^(0), and as to goes to infinity then G^(t) goes to G^(0)?


Before we saw worldwide collaborative efforts in training machine-learning models or widespread deployments of prediction-as-a-service, we need to devise an efficient privacy-preserving mechanism which guarantees the privacy of all stakeholders (data contributors, model owner, and queriers). Slalom (ICLR '19) preserves privacy only for prediction by leveraging both trusted environment (e.g., Intel SGX) and untrusted GPU. The challenges for enabling private training are explicitly left open -its pre-computation technique does not hide the model weights and fails to support dynamic quantization corresponding to the large changes in weight magnitudes during training. Moreover, it is not a truly outsourcing solution since (offline) pre-computation for a job takes as much time as computing the job locally by SGX, i.e., it only works before all pre-computations are exhausted. We propose Goten, a privacy-preserving framework supporting both training and prediction. We tackle all the above challenges by proposing a secure outsourcing protocol which 1) supports dynamic quantization, 2) hides the model weight from GPU, and 3) performs better than a pure-SGX solution even if we perform the precomputation online. Our solution leverages a non-colluding assumption which is often employed by cryptographic solutions aiming for practical efficiency (IEEE  SP '13, Usenix Security '17, PoPETs '19). We use three servers, which can be reduced to two if the pre-computation is done offline. Furthermore, we implement our tailor-made memory-aware measures for minimizing the overhead when the SGX memory limit is exceeded (cf., EuroSys '17, Usenix ATC '19). Compared to a pure-SGX solution, our experiments show that Goten can speed up linear-layer computations in VGG up to 40×, and overall speed up by 8.64× on VGG11.

Summary
========
This paper proposes a framework for privacy-preserving training of neural networks, by leveraging trusted execution environments and untrusted GPU accelerators.
The system builds heavily on the prior Slalom system, and uses standard MPC techniques (three non-colluding servers, multiplication triplets) to extend Slalom's inference-only protocol to privacy-preserving training.
This is a valuable and hard to reach goal. Unfortunately, the paper's evaluation fails to deliver on its strong promises, by ignoring the high network communication between the non-colluding servers.
Specifically, all experiments were conducted with three servers co-located in a public cloud's LAN. In this setting, it is hard to argue that non-collusion is a valid security assumption as the cloud provider controls all servers (alternatively, if the cloud provider is trusted, then there is no need for any trusted execution or cryptography). If the same experiments were conducted on a WAN, the communication costs would alleviate any savings in computation time.
For these reasons, I lean strongly towards rejection of this paper.
Detailed comments
=================
Extending the ideas in Slalom to support privacy-preserving training is a good research question, and Tramer and Boneh had discussed some of the challenges and limitations towards this in their original paper.
Getting rid of the pre-processing stage for blinding factors by leveraging non-colluding servers is a well-known trick from the MPC literature, but it does not seem easily applicable here.
The problem is that the servers need to communicate an amount of data proportional to the size of each internal layer of the network, for each forward and backward pass. If the servers communicate over a standard WAN, the communication time will be much too high to be competitive with the CaffeScone baseline.
In a LAN, as in this paper's experiments, the network latency is low enough for the network costs to be dominated by computation. But this begs the question of whether servers running in a same LAN (e.g., hosted by a single cloud provider) can really be considered non-colluding. In the considered setup, the cloud provider (Google in this case), could just observe the communication between all servers, thereby breaking privacy.
Another security issue with proposed scheme is the lack of computation integrity. This corresponds to the so-called "honest-but-curious" threat model which often appears in the MPC literature, and this should be acknowledged and motivated.
On the experimental side, the considered baseline, CaffeScone, seems pretty weak. In particular, any optimizations that the authors implement for Goten (e.g., better paging) should also be added to their baseline for a fair comparison.
The numbers in Figure 3 show that the baseline could be optimized a lot further.  A gap between hardware/simulation modes of ~6x seems indicative of sub-optimal paging. Even the single-core, simulation mode throughput numbers seem low for CIFAR10.
The experimental setup is quite confusing. Running the baseline and Goten in different environments (e.g., different CPUs) and then re-normalizing throughputs is somewhat convoluted and prone to mistakes. Why not run all experiments on the same setup?
Similarly, converting between results in SGX's hardware and simulation modes is also not very indicative. The authors note (p. 8) that in SGX's simulation mode "code compilation is almost the same as hardware mode except that the program is not protected by SGX, which is fine for our purpose since the DNN training and prediction algorithms are publicly known". This is fundamentally incorrect!
SGX's simulation mode provides absolutely no security guarantees. It simply compiles the code using the SGX libraries and ensures that the enclaved code performs no untrusted operations, but it does not provide any hardware protections whatsoever. In particular, code running in simulation mode will not be affected by the overhead of SGX's paging, as the memory is never encrypted.
As a result, performance results in simulation mode are usually not indicative of performance in hardware mode. Trying to convert runtimes from simulation mode to hardware mode by comparing times of specific layers is also prone to many approximation errors.
Finally, I had some trouble understanding the way in which Goten quantization works. Section 3.3. mentions that values are treated as floats, but then mentions the use of 53 bits of precision. Did you mean double-precision floats here? But then, aren't modern GPU optimized mainly for single-precision float operations? Section 3.3. also says that the quantization ensures that there are nearly no overflows. What happens when an overflow occurs? I guess that because of the randomized blinding, a single overflow would result in a completely random output. How do you deal with this during training?
Minor
=====
- Typo in abstract: Slaom -> Slalom
- I don't understand the purpose of footnote 3 in Appendix B.2. First, the bibliographic entry for (Volos et al. 2018) explicitly says that the paper was published in OSDI 2018, a top-tier peer-reviewed conference. Regardless, claiming a date for a first unpublished draft of your paper is a little unusual and somewhat meaningless. I'm sure Volos et al. had a draft of their paper ready in late 2017 or even earlier if they submitted to OSDI in XXX 2018. If you want to timestamp your paper, post in to arXiv or elsewhere online.