Human and animals continuously acquire, adapt as well as transfer knowledge throughout their lifespan. The ability to learn continuously is crucial for the effective functioning of agents interacting with the real world and processing continuous streams of information. Continuous learning has been a long-standing challenge for neural networks as the repeated acquisition of information from nonuniform data distributions generally lead to catastrophic forgetting or interference. This work proposes a modular architecture capable of continuous acquisition of tasks while averting catastrophic forgetting. Specifically, our contributions are: (i) Efficient Architecture: a modular architecture emulating the visual cortex that can learn meaningful representations with limited labelled examples, (ii) Knowledge Retention: retention of learned knowledge via limited replay of past experiences, (iii) Forward Transfer: efficient and relatively faster learning on new tasks, and (iv) Naturally Skewed Distributions: The learning in the above-mentioned claims is performed on non-uniform data distributions which better represent the natural statistics of our ongoing experience. Several experiments that substantiate the above-mentioned claims are demonstrated on the CIFAR-100 dataset.

I think there might be some interesting ideas in the work, but I think the authors somehow did not manage to position themselves well within the *recent* works on the topic or even with respect to what continual learning (CL) is understood to be in these recent works.
E.g. CL is a generic learning problem, and most algorithms are generic (with a few caveats in terms of what information is available) in the sense that they can be applied regardless of task (be it RL, be it sequence modelling etc.). This work seems limited to image classification. The SHDL wavelets pre-processing, if I understood it, is specific for images and probably even there under some assumption (e.g natural images).
The autoencoder (middle bit) is trained on all tasks the CL needs to face, if I understood the work correctly (phase 0 + phase 1). This potentially makes the CL problem much simpler because you are limiting yourself to the top layer only when dealing with CL, not the rest. Not to mention that I don't understand the motivation of the autoencoder. I think ample results show that unsupervised learning fails in many instances to provide the right features and underperforms compared to learning discriminative features by just backproping from the cross entropy (discriminative loss) all the way down. The only instance I know of for doing this is in low data regime where there is no alternative.
I think the modularity used needs to be better introduced. Why the autoencoder, why the first layer of wavelets? Is it for the benefit for CL? I can understand the wavelets, since they are not learnt. But the autoencoder? The autoencoder being trained on all data feels like a cheat.
I think the citation of the perceptron a bit strange. Do you really use the original perceptrion from 58? Why? We have much better tools now !?
I think the different metrics introduced are interesting and useful. Though you should somehow find common ground to existing works as well to ensure a point of comparison. In the results section I almost got lost. What is the final performance on Cifar. How does this compare to a model that is not trained in a CL regime? What loss do you get from the proposed parametrizaton?
In the comparison with EWC and iCarl, there the whole model was dealing with the CL problem, right? (all intermediary layers). I'm actually surprised iCarl is not doing better (I expect it can do better than EWC). Maybe provide a few more information of hyperparam used for this comparison.
Overall I think the paper is not ready for being published. Not without addressing these points:
 * role of modularity (if not CL -- then why? ; is the modular structure original or part of the previous works cited, e.g. where the wavelets are introduced and so forth)
 * better integration with recent literature; provide answers and settings that allow apple to apple comparison so one can easily understand where this approach falls; if the method is not meant for this "traditional settings and metrics" please still provide them, and then motivate why this regime is not interesting and explain better the regime the method is meant for
 * as it stands the work is light on the low level details; Hyper-params and other details are not carefully provided (maybe consider adding an appendix with all of these). I have doubts that the work is reproducible without these details.


In multi-participant postings, as in online chat conversations, several conversations or topic threads may take place concurrently. This leads to difficulties for readers reviewing the postings in not only following discussions but also in quickly identifying their essence. A two-step process, disentanglement of interleaved posts followed by summarization of each thread, addresses the issue, but disentanglement errors are propagated to the summarization step, degrading the overall performance. To address this, we propose an end-to-end trainable encoderdecoder network for summarizing interleaved posts. The interleaved posts are encoded hierarchically, i.e., word-to-word (words in a post) followed by post-to-post (posts in a channel). The decoder also generates summaries hierarchically, threadto-thread (generate thread representations) followed by word-to-word (i.e., generate summary words). Additionally, we propose a hierarchical attention mechanism for interleaved text. Overall, our end-to-end trainable hierarchical framework enhances performance over a sequence to sequence framework by 8-10% on multiple synthetic interleaved texts datasets.

<Strengths>
+ This paper addresses the problem of summarizing interleaved posts, which involves two subtasks of identifying the clusters of threads and summarizing each thread into a sentence.
+ This is an understudied area of research in summarization research, only few previous works have been done; for example, (Shang et al. 2018). The proposed approach is end-to-end  trainable, whereas (Shang et al. 2018)’s method tackles the problem in two separate stages.
<Weakness>
1. The technical novelty may need to be further justified.
- The proposed model (in Fig.2) consists of the encoder, the decode and hierarchical attention module.
- The encoder and the decoders are standard ones based on (Nallapati et al. 2017) and (Li et al. 2015), respectively. It is hard to find any technical novelty here.
- This paper argues that the proposed hierarchical attention is novel with no clear ground. It simply mentions the inspiration from (See et al. 2017), but does not explicitly what aspects are novel as Hierarchical attention has been popularly used in general seq2seq models for many NLP applications.
- Table 1 is rather cursory and incomplete as it compares with only three different models.
2. Experiments have much room for improvement.
- The two datasets are simply mixture of text from existing datasets and rather artificial and far from actual interleaved posts, as shown in Table 2.
- In page 6, three different corpora are introduced with varying interleave parameters. However, the parameters a,b,m, and n are rather small (<= 5), so it is hard to say the configurations are challenging.
- The SOTA model (Shang et al. 2018) is only compared on the PubMed Easy Corpus in Table 3. Is there any special reason to exclude it from the comparison with more challenging corpora?
- In most results of experiments, the paper focus on showing that the proposed hierarchical attention model is better than basic seq2seq, which may not be strong results to show the effectiveness of the proposed model.
<Minor comments>
- This draft does not seem to follow the ICLR format.
- Fig.2-4 are hard to see in details. They should be enlarged and re-organized.
- References should be added to Table 1.
- This paper exceeds the limit of 8 pages. I cannot find any good strong ground to have more additional pages than the commended length.
<Conclusion>
My rate is borderline with slightly leaning toward reject, mainly due to issues of technical novelty and evaluation. I will decide my rate finally, based on the authors’ response to the concerns above.


Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.

This paper presents a stochastic model based on Glow for conditional video generation. The major novelty of this work is to introduce the flow-based models to video modeling and learn the video dynamics via the dependencies of the latent variables. The general idea is reasonable and the proposed model is technically correct, but I have the following concerns mainly about the originality and the experiments.
**Above all, most of the text in Section 3 is very similar (or exactly the same) to the text of the Glow paper (the background section).**
— Significance and originality —
a.1) In Section 1, the authors discussed some possible application scenarios of video prediction models, e.g. learning from unlabeled data and being used for downstream tasks. However, all models mentioned here, including [Mathieu et al. 2016] and [Finn et al. 2016], are deterministic models. Thus, in what way can the stochastic model proposed in this paper be used in real applications?
a.2) VideoFlow can be viewed as an extension of the Glow model. There are two problems. First, the originality is limited. I don’t think modeling the temporal dependencies of the latent variables with a convolutional network is a significant contribution to the conditional flow-based methods. Second, this paper is not self-contained. After reading Section 4.2, I have to check the previous literature to find the objective function, the network details, or the training procedure.
— Experiments —
b.1) Throughout the experiments, the VideoFlow model is mainly compared with two stochastic video prediction models that were probably proposed by the same research group. If it is possible, the authors might include other stochastic models such as the SVG-LP [Denton & Fergus 2018], and at least one deterministic model such as the E3D-LSTM [Wang et al. 2019] as well.
[Wang et al. 2019] E3D-LSTM: A Model for Video Prediction and Beyond.
b.2) The evaluation metric bits-per-pixel was not directly optimized by the previous video generation/prediction models. Thus, the comparisons in Table 2 might be unfair.
b.3) Since training the Glow model requires a huge computational cost, how is the training efficiency of the VideoFlow model compared with other stochastic video generation models?
— Other —
c.1) In Section 4, it is not clear what the temporal border effect means?
AFTER REBUTTAL:
Though the overall novelty is still not fully convincing, this paper may shed some insights into video generation by introducing flow-based models to this topic. I have increased my score from 3 to 6.


The Graph Convolutional Network (GCN) and its variants are powerful models for graph representation learning and have recently achieved great success on many graph-based applications. However, most of them target on shallow models (e.g. 2 layers) on relatively small graphs. Very recently, although many acceleration methods have been developed for GCNs training, it still remains a severe challenge how to scale GCN-like models to larger graphs and deeper layers due to the over-expansion of neighborhoods across layers. In this paper, to address the above challenge, we propose a novel layer-wise sampling strategy, which samples the nodes layer by layer conditionally based on the factors of the bi-directional diffusion between layers. In this way, we potentially restrict the time complexity linear to the number of layers, and construct a mini-batch of nodes with high local bi-directional influence (correlation). Further, we apply the self-attention mechanism to flexibly learn suitable weights for the sampled nodes, which allows the model to be able to incorporate both the first-order and higher-order proximities during a single layer propagation process without extra recursive propagation or skip connection. Extensive experiments on three large benchmark graphs demonstrate the effectiveness and efficiency of the proposed model.

This paper aims at improving the computational efficiency of GCNs to effectively capture information from the larger multi-hop neighborhood. Conventionally, GCNs use information from all the neighbors up to a certain depth; in which case, with consideration of each further hop, the neighborhood size increases exponentially. To avoid the exponentially increasing memory and computational footprints of GCNs as a result of an exponential neighborhood expansion, this paper proposes a (hop) layer-wise sampling procedure that reduces the complexity to a linear factor. The sampling of nodes at a layer, ‘l’ is based on the transmission probabilities of the nodes at layer ‘l’ and their immediate neighbors sampled earlier in layer ‘l+1’ from both directions of diffusion. The proposed model is based on Graph ATtention Network (GAT) which is adopted here to aggregates neighborhood information only over the nodes sampled with their bi-diffusion sampler.
Strengths of the paper: The paper intuitively suggests that some of the popular sampling-based scaling approaches for GCN may not be powerful enough as they don’t consider bi-directional influences.
Weaknesses of this paper:
- Novelty: The idea is incremental. The paper is similar to the layer-wise sampling model, AS-GCN where instead of the base GCN model this paper uses GAT coupled with its proposed bi-diffusion sampler.
- Experimental results:
    - (a) Inconsistent baseline results: The performance of baselines reported here on standard train/test/val splits are significantly lower than the ones reported in the original papers. For ex: with FastGCN the original papers report 0.88 and 0.937 on PPI and Reddit which is ~0.03 more than what is reported here. With the case of AS-GCN, the original performance scores are superior to the proposed model in the paper, however here they are reported ~0.04 scores lower. Since the codes for all these baselines are available, it is only fair to use the original implementation; if not, it is important to replicate the original results before using a different implementation.
    - (b) Variance and statistical significance results are missing
    - (c) Cluster-GCN though discussed, an experimental comparison with it is missing. Reported results from Cluster-GCN paper on Reddit and PPI suggests a superior performance over BLS-GAT.
    - (d) BLS-GCN missing. This would be a fair comparison to FastGCN and AS-GCN.
    - (e) Experimental comparison with Jumping Neural network (Xu et al) is missing to understand how the proposed solution improves over existing solutions for over-smoothing. It would be helpful to even couple it with Fast-GCN/AS-GCN sampler to better understand the benefits of this paper.
- Writing:
    - The paper is not well written. Though there are only minor grammatical mistakes, multiple sections of the paper are not clear and are hard to read because of complex sentences and long paragraphs.
    - Some of the terminologies used are not clearly described and are not explained prior to the usage. Some of them are neighbor-explosion, over-expansion, the width of neighborhood expansion, local correlations, etc, In some places, over-expansion is used to refer only neighbourhood explosion or only over-smoothing and both. It will be comprehensive if it is grounded.
    - Numerous claims/ideas put forth in this paper are abstract and intuitive. The intuitions should be backed with proper support. Some of the major concerns are:-
        - (a) proof/arguments to show that layer-wise sampling may lead to sparse mini-batches and how does that in-turn impact over-smoothing
        - (b) how does the proposed model avoid over-smoothing?
        - (c) why sub-graph methods are not effective ? .. etc
    - It is true that using a fixed neighborhood weightage function as with GCNs may not be optimal. However, the discussion made on GCN and its lack of an appropriate normalization/ neighbourhood weightage function is incorrect. GCNs aggregate information from further neighborhoods according to the respective higher-order diffusion laplacian matrix entries. You can see that by simply removing the nonlinearity+weights and recursively expanding the GCN equation.
- Other comments:
    - Provide the complexity of the proposed model (GCN + sampler) and compare it with other sampling approaches.
    - The connectivity structure + signal on the nodes of the graphs is the data that is being convolved and they are not the filters. The weights being learned are the filters.
    - In Eqn: 2, I believe you are providing an equation for GS-GCN. In which case the fraction should be N(v)+1/ (N_s(V)+1) to match the original model/implementation for GraphSAGE (GS) paper.
    - I think the summation in the denominator for AS-GCN following Eqn: 3 should run over V instead of V_l.
    - It will be helpful to run the model on directed datasets to see improved benefits of bi-diffusion sampling.
    - Need more discussion about AS-GCN, Cluster-GCN and Jumping Neural networks (Xu 2018b)


Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

*EDIT: Score increased after discussion with authors clarified many concerns raised below*
Summary:
This paper presents an algorithmic approach toward learning Bayes Optimal policies under the uncertainty of the environment. Leveraging meta-learning, the proposed variBAD approximate inference procedure is capable of adapting within the first episode at, what the authors term, meta-test time.
Comments:
It is my estimation that this paper is well positioned to further current state-of-the-art adaptive RL frameworks or methodologies, whether they are meta-learned, transferred or directly inferred through probabilistic mechanisms. The primary contribution of this paper is in how variBAD learns the variational inference procedure. As noted in the related work section, many contemporary policy learning approaches via variational inference are limited by their construction, selection of prior distributions, etc. The advantage of the proposed methodology is that it is capable of efficiently inferring the current environment and adapting the policy learning procedure accordingly. The experiments successfully compare with relevant baselines and prior approaches. The discussion is well framed in highlighting the benefits and limitations of the proposed methodology in relief to prior approaches. One possible weakness in the experimentation, given how closely RL^2 matches the performance of variBAD, is that the specific contributions of each architectural choice or optimization protocol are unclear. There are a few areas in the paper where the authors suggest that ablating their model in specific ways would recover the core approaches present in RL^2. It would be instructive to see how/if performance degrades or converges toward that of RL^2 as the variBAD methodology is ablated.
The paper is well grounded in the literature, albeit skewed perhaps a bit too far toward recent meta-learning results. This is understandable given the focus of this paper, however there are other approaches that might deserve a mention as they similarly parameterize variation over possible MDPs with some latent variables. Namely, I have in mind a few lines of research such as Contextual MDPs (Hallak, et al, 2015; Jiang, et al, 2017; Dann, et al, 2018; etc.), Successor Features (Barretto, et al, 2016,2017,2019; Lehnert, et al, 2017; etc.)  and HiP-MDPs (Doshi-Velez and Konidaris, 2016; Killian, et al, 2017). In particular use of the HiP-MDP framework, Yao, et al (2018) also use the inferred latent variable used to identify the task to condition the policy. While it's always easy to dig into the rabbit holes of related research and distract from the overall objective of a paper, I thought that there was sufficient overlap with these other lines of research that the authors may find interesting. I do not claim that any one of these additional sources of prior work have been overlooked to the detriment of the current paper, they are offered as merely a suggestion to broaden the author's anchoring in the literature.
Now, some more specific questions about the paper and proposed approach. Further clarity along any of these questions would greatly improve the presentation of the paper as well as further convince me of the paper's suitability for publication.
1) What is the advantage of decoding the entire trajectory? It is well understood that this is advantageous in training as that data is available and allows for better inference of the variational parameters. However, under test conditions where the framework may be operating in environments that lie outside the distribution of MDPs it was trained on, I can imagine that errors in trajectory prediction may compound and throw off the entire inference procedure. The experimental set-up did not allay these concerns as there was no mention for holding out-of-distribution tasks/environments aside and the variation in environments is pretty narrow.
2) How is the proposed trajectory decoding more stable than model predictive control? Is stability a large consideration for variBAD when exploring? How much can one trust the exploratory actions under variBAD?
3) The visualization and careful explanation in Section 5.1 of how variBAD executes inference and learning was greatly appreciated. However, are these intuitions valid when extending beyond discrete state and action spaces? Can one make the same claims about the overall approach or procedure in the MuJoCo domains? It was mildly disappointing that a similar explanatory effort was not made in more complex environments. Even an acknowledgement of this being unreasonable would help round out the discussion in Section 5.2.
4) It is not clear what the connection is between the horizon H and the number of rollouts used for evaluation/inference/training. I spent a bit more time than necessary going over and over these items in the paper to where I think that I may understand but I'm still not 100% confident about what is impacted by the number of rollouts used.


Exploration bonuses derived from the novelty of observations in an environment have become a popular approach to motivate exploration for reinforcement learning (RL) agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. In this paper, we introduce the concept of optical flow estimation from the field of computer vision to the RL domain and utilize the errors from optical flow estimation to evaluate the novelty of new observations. We introduce a flow-based intrinsic curiosity module (FICM) capable of learning the motion features and understanding the observations in a more comprehensive and efficient fashion. We evaluate our method and compare it with a number of baselines on several benchmark environments, including Atari games, Super Mario Bros., and ViZDoom. Our results show that the proposed method is superior to the baselines in certain environments, especially for those featuring sophisticated moving patterns or with high-dimensional observation spaces.

The paper proposes a novel way to formulate intrinsic reward based on optical flow prediction error. The prediction is done with Flownet-v2 architecture and the training is formulated as self-supervision (instead of the ground-truth-based supervised learning in the original Flownet-v2 paper). The flow predictor takes two frames, predicts forward and backward flows, then warps the first/second frame respectively and compares the warped result with real frame. The comparison error serves as the intrinsic reward signal. The results are demonstrated on 7 environments: SuperMario + 5 Atari games + ViZDoom. On those environments, the proposed method performs better or on-par with ICM and RND baselines.
I am leaning towards rejecting this paper. Two key factors motivate this decision.
First, the motivation for this work is not fully clear: why would the error in flow prediction be a good driving force for curiosity? Optical flow has certain weaknesses, e.g. might not work well for textureless regions because it's hard to find a match. Why would those weaknesses drive the agent to new locations?
Second, the choice of tasks where the largest improvement is shown (i.e. 5 Atari games) seems not well-motivated and rather crafted for the proposed method. Those 5 Atari games are not established hard exploration games.
Detailed arguments for the decision above:
[major concerns]
* Analysis is need on how the method deals with known optical flow problems: occlusion, large displacements, matching ambiguities. Those problems don't fully go away with learning and it is unclear how correlated corresponding errors would be with state novelty.
* "Please note that ri is independent of the action taken by the agent, which distinguishes FICM from the intrinsic curiosity module (ICM) proposed in Pathak et al. (2017)" - but would it then be susceptible to spurious curiosity effects when the agent is drawn to motion of unrelated things? Like leaves trembling in the wind. ICM was proposed to eliminate those effects in the first place, but what is this paper's solution to that problem? Furthermore, the experiments on BeamRider show that this concern is not a theoretical one but quite practical.
* "CrazyClimber, Enduro, KungFuMaster, Seaquest, and Skiing" - none of those Atari environments are known to be hard exploration games (which are normally Gravitar, Montezuma Revenge, Pitfall!, PrivateEye, Solaris, Venture according to Bellemare et al "Unifying count-based exploration and intrinsic motivation"). I understand that every game becomes hard-exploration if the rewards are omitted but then there is a question why those particular games. Moreover, if you omit the rewards the question remains how to select hyperparameters of your method. Was the game reward used for selecting hyperparameters? If not, what is the protocol for their selection? This is a very important question and I hope the authors will address this.
* "These games are characterized by moving objects that require the agents to concentrate on and interact with." - this looks like tailoring the task to suit the method.
* Figure 6 - those results are not great compared to the results of Episodic Curiosity: https://arxiv.org/abs/1810.02274 . Maybe this is because of the basic RL solver (A3C vs PPO) but that brings up another question: why are different solvers used for different tasks in this paper? PPO is normally significantly better than A3C, why not use throughout the whole paper?
[minor concerns]
* Figures are very small and the font in them is not readable. Figure 2 is especially difficult to read because the axes titles are tiny.
* "complex or spare reward" -> sparse
* "However, RND does not consider motion features, which are essential in motivating an agent for exploration." - this is unclear, why are those features essential?
* "We demonstrated the proposed methodology and compared it against a number of baselines on Atari games, Super Mario Bros., and ViZDoom." - please state more clearly that only 5 out of 57 Atari games are considered, here and in the abstract.
* "Best extrinsic returns on eight Atari games and Super Mario Bros." - but only 5 games are shown, where are the other 3?
Suggestions on improving the paper:
1) Better motivating the approach in the paper would help. Why using the flow prediction error as a curiosity signal?
2) Better motivating the choice of the environments and conducting experiments on more environments would be important for evaluating the impact of the paper.


Exploration strategy design is one of the challenging problems in reinforcement learning (RL), especially when the environment contains a large state space or sparse rewards. During exploration, the agent tries to discover novel areas or high reward (quality) areas. In most existing methods, the novelty and quality in the neighboring area of the current state are not well utilized to guide the exploration of the agent. To tackle this problem, we propose a novel RL framework, called clustered reinforcement learning (CRL), for efficient exploration in RL. CRL adopts clustering to divide the collected states into several clusters, based on which a bonus reward reflecting both novelty and quality in the neighboring area (cluster) of the current state is given to the agent. Experiments on several continuous control tasks and several Atari-2600 games show that CRL can outperform other state-of-the-art methods to achieve the best performance in most cases.

This paper presents a clear approach to improve the exploration strategy in reinforcement learning, which is named clustered reinforcement learning. The approach tries to push the agent to explore more states with high novelty and quality.  It is done by adding a bonus reward shown in Eq. (3) to the reward function. The author first cluster states into clusters using the k-means algorithm. The bonus reward will return a high value for a state if the corresponding cluster has a high average reward. When the total reward in a cluster is smaller than a certain threshold, the bonus reward will consider the number of states explored. In the experiments, the authors test different models on two MuJoCo tasks and five Atari games. TRPO, TRPO-Hash, VIME are selected as baselines to compare with. Results show that the proposed bonus reward reaches faster convergence and the highest return in both MuJoCo tasks. In those five Atari games, the proposed method achieves the highest or second-highest average returns.
Although the paper is generally easy to follow and the motivations of the equations are clear,  the analysis of the results is missing and thus the paper provides very limited insights on the behavior of the proposed method. As a result, my opinion on this paper leans to a rejection. As ICLR recommends paper length to be 8 pages, the authors can and should use the remaining space to give more details. For example, the authors can show the mean reward and number of states in all clusters to see whether the agent is efficiently exploring different clusters. And as the number of clusters K is an important hyper-parameter, the reader will also be curious about the resultant performance of the method with different K.
Some questions:
1) In Eq. (3), you have two hyper-parameters in the bonus reward, and for MuJoCo and Atari games, you are using different settings for the first coefficient. How do you choose the hyper-parameter settings? Do you perform grid search with another environment or a set of environments to determine the hyper-parameters?
3) In the algorithm, the method has to learn new cluster assignments in each iteration. Does it significantly slow down the training of the agent?
4) In the experiments, the authors compare with other methods on only five Atari games. However, there should be more environments available.  What is the reason for choosing these five games? Unless it is difficult to gather the scores on more environments, I believe the authors shall provide results with more Atari environments.
5) The conclusion (Section 6) is extremely short and it claims that "CRL can outperform other SOTA methods in most cases". However, for the five Atari games, the only game that CRL achieves the best performance among seven methods is Venture, according to Table 1.
Such a claim will confuse readers as it is not in line with the results.
All in all, I believe this paper can be significantly improved if more details and analyses are provided.


In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent variables. Unfortunately, this well-motivated strategy often fails to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation. We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of a bounded total correlation. However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy, RTC-VAE, which rectifies the total correlation penalty. Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g., β-TCVAE and FactorVAE.

The paper proposes a method to address a known problem for unsupervised disentangling methods that penalises total correlation, namely that while the total correlation of the samples from q(z) (denoted TC(z)) are encouraged to be small, the total correlation of the means of q(z|x) (denoted TC(mu)), used as the disentangled representation in practice, is not necessarily small and can increase with regularisation strength.
In the introduction, I think that the statement “they concluded pessimistically that it is fundamentally impossible to learn a disentangled representation in an unsupervised setting” is a wrong interpretation of Locatello et al. They show that optimising marginal likelihood in a generative model (such as a VAE) cannot achieve disentangling without any inductive biases in the model. But there inductive biases in the models used by disentangling methods, along with the loss (that is a variant of the ELBO and not the marginal likelihood), that allow disentangling in practice. There are also theoretical works such as [1] that explain this behaviour.
The theoretical contribution of the paper is Theorem 1, that claims to show the existence of distributions with arbitrarily large TC(mu) but with small, bounded TC(z). Indeed the proof shows that TC(z) is bounded by C, but looking at Appendix A it seems as though C is a function of c1,c2 and R that is used to define p(z|mu), and is lower bounded by (2pi)^{-D/2} (a constant, which confusingly, is also denoted by C in the appendix). It appears necessary to have another line that mentions how small C can be chosen to be via choice of c1,c2,R. Also it seems as though the proof can be largely simplified by having sigma’_j(mu)=c_1 if |mu|<R and sigma’_j(mu)=c_4/|mu| if |mu|>R, removing free parameters l,c_2,c_3.
The methodological contribution of the paper is to propose an extra regularisation term that penalises the variances of q(z|x). While this does introduce another hyperparameter to tune, it has the advantage of being simple to implement and having an intuitive explanation of how it can address the problem; if q(z|x) is encouraged to have smaller variance, the distribution of z will be encouraged to be closer to the distribution of mu, hence helping to address the disparity between TC(z) TC(mu).
I like the simplicity of the idea, however the analysis is lacking in rigour. First of all, when comparing the different methods of estimating TC(z), it’s not clear what the mathematical difference of MSS_0 and MSS_1 is. This should be explicitly stated so that one can understand the results in Figure 1. Regarding the following analysis, it’s not clear why the off diagonal elements of the cube (i.e. q(z^(i)_k|n^(j)) when i != j ) should be very small compared to the diagonal elements. For example, it could be the case that z^(i) and z^(j) are close to one another, in which case the (i,j)th entries will be non-negligible compared to the diagonal. Hence the analysis is difficult to accept. Also the claim that MSS and MWS prefer to shut down latent dims should be verified by experiments. Further, it’s unclear why this is undesirable from a disentangling point of view. Of course we don’t want to use fewer number of latent dimensions than the number of ground truth factors, but also we don’t want to use more latent dimensions. Also the at the bottom of page 5 is a Gaussian with a correlated covariance matrix, and it’s claimed that its TC can be arbitrarily large, but surely this is fixed? Finally the authors list lots of reasons why not to use MSS, but there is no discussion about the density-ratio trick method for estimating TC. For example, it is known that this method suffers underestimates of TC (c.f. Kim & Mnih) - it has its own issues, that can be arguably more severe than MSS. This analysis needs a lot more explanation and rigour.
The experimental results are very weak and sparse, that is nowhere near enough to give a convincing case for the newly proposed method. The method has only been trained on dsprites and 3d shapes, with three choices of beta and a single value of eta, and only estimates of TC(mu) and TC(z) are reported, with no evaluation of disentanglement performance. There is some evidence in the paper that the regularisation makes both TC(mu) and TC(z) close to 0 with eta=10, but it’s unclear how this affects disentangling performance, and whether smaller values of eta can give a sweetspot. The experiments should cover a larger range of datasets, with evaluation on how different disentanglement metrics, TC(mu) and TC(z) change for different values of beta and eta, along with a comparison with other disentangling methods, especially DIP-VAE-1, that directly penalises correlation in mu (for an open source library that facilitates this, see e.g. github.com/google-research/disentanglement_lib). Even the authors acknowledge that “the scale of our experiments is limited”, and it is clear that the paper is not yet ready for publication.
Overall, the proposed idea is simple and easy to implement, which is the main advantage of the paper, but it is evident that the analysis and evaluation lacks rigour, hence the paper will need to undergo significant revision to be in a publishable state.
[1] Rolinek, M., Zietlow, D. and Martius, G., Variational Autoencoders Pursue PCA Directions (by Accident). CVPR 2019.
Minor typos/comments:
Eqn(4): the ||.||_1 should be replaced by trace, since the term inside ||.||_1 is a covariance matrix (although it’s diagonal). If A is a matrix, ||A||_1 is the maximum absolute column sum, which is different to what is meant by the paper, the trace (sum of diagonals).
p3: Can MWS also be stated in the paper (or at least in the appendix) to make it self-contained?
p4: followed <- follow, n_{m+1} <- n_{M+1}
p6-7: Section 6.1 should be in a related work section, and not under the Experiment section. The point about modularity is a fair but known issue, closely related to the issue of axis alignment/unidentifiability (see e.g. Rolinek et al)


A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models. Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves. However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings. We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.

*Contributions*
This paper deals with the theoretical study of the gradient dynamics in deep neural networks. More precisely, this paper define a notion of incremental learning for a particular learning dynamics and study how the depth of the network influence it. Then, the authors show two cases where it applies: matrix sensing, quadratic neural networks and provide intuitions on how it could also apply to linear convolutional networks.
This work leverage the framework and the proofs of Gidel et al. 2019 and Saxe et al. 2014 [1] to study the impact of depth in that sequential learning (that is the novelty of this work).
I really like the idea of studying the impact of the depth in the training dynamics. And the results (Thm2 and 3) are really interesting (but a bit hard to interpret in my opinion, see my questions)
Also, this work should make clearer that the results stated in Thm1 were already substantially presented in Saxe et al. 2013 ( eq. 17 and eq. 12 in [1])
Note that this paper is borderline regarding anonymity since it contains an acknowledgement section revealing the fundings of the authors (that could give enough information to identify the authors of this paper).

*Decision*
Weak accept: The key results in this work is Theorem 2 and it’s extension to discrete case Theorem 3. they seems really interesting: when $N > 2$ in order to observe sequential learning, the dependence in the eigengap for the initialization goes from polynomial ($N=2$) to polynomial.
Showing that result is interesting (even in this limited setting) because it theoretically shows (at least for these simple classes of problem) that deeper network perform a notion of incremental learning of components with non-prohibitively small initialization.
However, these results are very hard to read. They could be interpreted and simplified in that purpose. I think it would greatly improve the quality of this work.
I develop these points in the *questions* section of my review.

*Questions*
- The definition 1 is hard to interpret. For instance why do you need $t_i$ and $t_j$ since the functions $\sigma_i$ are increasing ? (note: the fact that these function are increasing is never mentioned in the paper but is key to talk about “incremental learning” and for your Definition 1 to make sense, since otherwise $\sigma_i$ could be “forgotten” without violating Definition 1) thus for any $t \in [t_i,t_j] $ we have $\sigma_i(t) \geq f \sigma_i^*$. Using only one time would make the definition easier to understand.
- The result presented in theorem 2, (and 3) are hard to interpret because of the many parameters that distract the reader to the main point. The dependence in s and f would be interesting if we would like to compute these bound in practice but I think that the interest of this work is in the distinction exponential versus polynomial (in $r$). Thus even though I think that a version with s and f is worth being in the appendix, a version with $s = f = ½$ would make the result statement and the discussion way clearer. (other question why restrict yourself to $s \in (0,½)$ and $f \in (¾,1)$?)
- In the theorem 3 who is $c$ ? who is $\sigma_1$ (the largest eigenvalue?) ?
- In theorem 3, I am very surprised that there is no notion of eigengap that restrict the size of $\eta$. for instance let us consider the three eigenvalues $\{2,2-\epsilon,1\}$ with epsilon very small. I think that this condition is implicitly appearing in A and B. Actually for a fixed $c$, if we do $\sigma_j^* \to \sigma_1^* = \sigma_i^*$ then we got $A = 1/B $ thus one of them is smaller than 1.
- You restrict yourself to a uniform initialization. Could you extend your results (and definitions) to non-uniform initialization (particularly initialization where
- Very small initialization is a big issue in practice because it induces very small gradient at the beginning of the training

- Figure 2, why is the time to learn those components increasing ? (i guess it is because of the $w^(2-2/N)$ that get smaller as $N$ increases. Isn’t it an issue in practice ? What is the sensitivity to the step size? in the discrete size (i.e. can we increase the step size to compensate the slower learning)?

*Minor remark*
- Saxe et al. 2013 as been accepted to ICLR in 2014 (would be better to cite the ICLR proceedings) see [1]

[1] Saxe et al. 2014 in ICLR url: https://openreview.net/forum?id=_wzZwKpTDF_9C
=== After rebuttal ===
I have read the authors' response.
I think the authors could have discussed more the interpretation of Theorem 3. in the revision.
But, I really like the main takeaway which is that there is a huge discrepancy between 2 and 3 layers in terms of dynamics.
I maintain my weak accept


Previous ternarizations such as the trained ternary quantization (TTQ), which quantized weights to three values (e.g., {−W n , 0, +W p }), achieved the small model size and efficient inference process. However, the extreme limit on the number of quantization steps causes some degradation in accuracy. To solve this problem, we propose a hybrid weight representation (HWR) method which produces a network consisting of two types of weights, i.e., ternary weights (TW) and sparse-large weights (SLW). The TW is similar to the TTQ's and requires three states to be stored in memory with 2 bits. We utilize the one remaining state to indicate the SLW which is referred to as very rare and greater than TW. In HWR, we represent TW with values while SLW with indices of values by encoding SLW. As a result, the networks can preserve their model size with improving their accuracy compared to ternary weights. To sparsify non-ternary weights in quantization, we also introduce a centralized quantization (CQ) process with a weighted ridge (WR) regularizer. They aim to reduce the entropy of weight distributions by centralizing weights toward ternary values. Our comprehensive experiments show the efficiency of HWR in terms of the trade-off between model size and accuracy.

This paper is about quantization, and how to represent values as the finite number of states in a low bit width, using discretization. Particularly, they propose an approach to tackle the problems associated with previous ternarization which quantize weights to three values. Their approach is a hybrid weight representation method, which uses a network to output two weight types: ternary weight and sparse-large weights. For the ternary weight, they need 3 states to be stored with 2 bits. The one remaining state is used to indicate the sparse-large weight. They also propose an approach to centralize the weights towards ternary values. Their experiments show that their approach outperforms other compressed modeling approaches, and show an increase in AlexNet performance on the CIFAR-100 while increasing model size by 1.13%.
- Overall, this is an interesting paper, offering a novel solution to tackle the degradation in accuracy occuring in ternary quantization techniques because of the number of quantization steps.  Their method seems technically sound, however I am not familiar with this area, so I would trust more the opinion of other reviewers - subject experts in the matter.
- Their results on AlexNet and ResNet do show an improvement in terms of model accuracy, with only a slight increase in model size. They have also provided extensive experiments studying the benefits of their quantization method, the tradeoff among accuracy and model size.
- I find the abbreviations, being used too often, to be confusing
- The paper is targeted for a very focused audience, and does not give enough background for readers not familiar with "ternarizations".  Even the abstract could benefit from a motivation/ problem statement sentence, as well as less abbreviations being used.
I would vote for acceptance of this paper, although it does seem a too-targeted paper on a specific audience. I would urge the authors to revise writing to make it broader accessible.


We present Neural Phrase-to-Phrase Machine Translation (NP 2 MT), a phrasebased translation model that uses a novel phrase-attention mechanism to discover relevant input (source) segments to generate output (target) phrases. We propose an efficient dynamic programming algorithm to marginalize over all possible segments at training time and use a greedy algorithm or beam search for decoding. We also show how to incorporate a memory module derived from an external phrase dictionary to NP 2 MT to improve decoding. Experiment results demonstrate that NP 2 MT outperforms the best neural phrase-based translation model (Huang  et al., 2018)  both in terms of model performance and speed, and is comparable to a state-of-the-art Transformer-based machine translation system (Vaswani et al., 2017).

This paper presents a phrase-based encoder-decoder model for machine translation. The encoder considers all possible phrases (i.e. word sequences) up to a certain length and compute phrase representations using bidirectional LSTMs from contextual word embeddings computed with another bidirectional LSTM layer. The decoder also considers possible segmentations and computes contextual representations for the previously generated segments. Each word in the current segment is generated by a Transformer model by attending to all phrases in the source sentence. The authors present a dynamic programming method for considering all possible segmentations in decoding. They also present a method for incorporating a phrase-to-phrase dictionary built by Moses into the decoding process.
I like the idea of phrase-to-phrase translation and the relatively simple architecture proposed in the paper. At the moment, however, I am not quite sure how practical their approach is. One reason is the experimental setting. Both of the datasets used in the experiments are quite small and it is not clear how the proposed model performs when several millions of sentence pairs are available for training.
Another reason is that the computational cost of the proposed model is not really clear. The authors state that it is much more efficient than NPMT but it is not clear how it compares to the standard Transformer approach. It seems to me that the computational cost of their model is highly dependent on the value of P (maximum length of phrases).
At first, I thought the decoder was implemented with LSTMs, but I realized that it was actually implemented with a Transformer by reading the appendix. I think this should be explained in the main body of the paper. I am also wondering how the authors’ model compares to a standard seq-to-seq model whose decoder is implemented with a Transformer.
The equation in section 2.2 seems to suggest that the model prefers segmentations with small numbers of segments. I am wondering if there is any negative effect on the translation quality.
Here are some minor comments:
p.2 valid of -> valid
p.4 lookup -> look up?
p.4 forr -> for
p.4 indict -> indicate?
p.5 Table 1 -> Table 1


Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions. While architectural advances have led to improved accuracy, building robust models remains challenging, involving major changes in training procedure and datasets. Prior work has argued that there is an inherent trade-off between robustness and accuracy, as exemplified by standard data augmentation techniques such as Cutout, which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy. We introduce Patch Gaussian, a simple augmentation scheme that adds noise to randomly selected patches in an input image. Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNet Common Corruptions benchmarks while also maintaining accuracy on clean data. We find that this augmentation leads to reduced sensitivity to high frequency noise (similar to Gaussian) while retaining the ability to take advantage of relevant high frequency information in the image (similar to Cutout). We show it can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment. Finally, we find that the idea of restricting perturbations to patches can also be useful in the context of adversarial learning, yielding models without the loss in accuracy that is found with unconstrained adversarial training.

This paper proposes a data augmentation method that interpolates between two existing methods (Cutout and Gaussian), for training robust models towards Gaussian and naturally occurring corruptions. The method is shown to improve robustness without sacrificing accuracy on clean data.
Pros:
The proposed method, despite being simple, seems to empirically work well in terms of the mCE criterion evaluated in the experiments. This does support the authors’ claim that current methods haven’t reached the robustness/accuracy tradeoff boundary yet.
Cons:
I’m a bit concerned about the significance of the work though. The method is a straight-forward combination of existing methods, so methodologically the novelty is kind of limited. Hence, I’m expecting more insights from the analysis of the results, to gain more understanding of why it works so well. However, the presentation of the experiments just seems to aim for the best numbers one can get (I’m not certain how significant the numbers are to this field though). A few examples/pictures of success cases (when the method works) and failure cases (when the method doesn’t work), may help readers (I’m not an expert) to better understand the approach and get more intuitions? The frequency analysis seems quite intuitive. It’s obvious that Gaussian filter blocks high-frequency components, and Cutout keeps some original parts of the image which allow high-freq details to be captured. But, considering CIFAR image size is only 32x32, a patch of size 25 is quite large, how much is the method different from plain whole image Gaussian then?


We consider the task of automated theorem proving, a key AI task. Deep learning has shown promise for training theorem provers, but there are limited humanwritten theorems and proofs available for supervised learning. To address this limitation, we propose to learn a neural generator that automatically synthesizes theorems and proofs for the purpose of training a theorem prover. Experiments on real-world tasks demonstrate that synthetic data from our approach significantly improves the theorem prover and advances the state of the art of automated theorem proving in Metamath.

This paper focuses on the problem of developing deep learning systems that can prove theorems in a mathematical formalism -- in this case, MetaMath. This has been a rapidly growing topic in the past few years, as evidenced by the numerous cited works. What sets this work apart from others is its focus on the instrumental task of generating data to train a prover, rather than directly training the prover on human theorems (via reinforcement learning) or human proofs (via imitation learning).
The paper proposer two approaches to generating theorems imitation learning (IL) and reinforcement learning (RL). The IL approach trains a neural policy to imitate the same steps taken in human proofs. The RL approach first trains a language model on human theorems (not proofs), and uses the likelihood under the model as a reward function for an RL agent which must take forward proof steps.
Both approaches result in a policy that can be used to take proof steps, with the goal of producing new theorems which are similar to the human ones. Since the proof steps are known for the generated theorems, a prover agent (which operates in backwards mode, working from the goal back to the hypotheses) can be trained to imitate the steps taken in the synthetic proofs (along with the human ones, if any are present).
At test time, the learned prover imitation policy is then used to guide an MCTS agent, as described in the Holophrasm paper. It is compared against the original Holophrasm algorithm, rerun on modern hardware.
This is to my knowledge a novel approach in the neural theorem proving domain, and in my opinion one that offers a potentially significant advantage over the existing fixed-dataset appraoches.
The main result of the paper is that an extra 35/2720 (1.2%) of the test theorems are proven, a 6% improvement over the Holophrasm baseline of 539. It is difficult to judge how relevant of an improvement this is, and there is no analysis of the difficulty of the MetaMath problem set. In addition, due to the 10-1-1 train-validation-test split, the neural agents are likely shown relatively similar problems during training as at test time, including potentially stronger versions of the same theorems. There is also no comparison against non-neural approaches, such as Z3, Vampire, or similar theorem provers.
To accept this paper, I would like to see stronger evidence that the introduced method produces significant improvements in prover ability. For example, the same method could be applied to datasets such as HOList, Mizar, and CoqGym which have received more attention recently than MetaMath.
Some additional questions and comments:
1. How big does the theorem graph G get? Since the relevance policy is over all nodes of the graph, this could lead to a very large neural network that would be difficult to fit into memory. Certainly not all 1M synthetic theorems could be generated in one graph.
2. The paper claims that all theorems from set.mm are used as background theorems in algorithm 1, including the test ones -- this potentially sounds like training on the test set, or even worse, having access to the test theorems as "proven background knowledge" at test time.
3. Please include some more details about the training of the Holophrasm baseline. Does it simply do RL on the human theorems, or does it also do IL on human proofs?


We consider the problem of generating configurations that satisfy physical constraints for optimal material nano-pattern design, where multiple (and often conflicting) properties need to be simultaneously satisfied. Consider, for example, the trade-off between thermal resistance, electrical conductivity, and mechanical stability needed to design a nano-porous template with optimal thermoelectric efficiency. To that end, we leverage the posterior regularization framework and show that this constraint satisfaction problem can be formulated as sampling from a Gibbs distribution. The main challenges come from the black-box nature of those physical constraints, since they are obtained via solving highly non-linear PDEs. To overcome those difficulties, we introduce Surrogate-based Constrained Langevin dynamics for black-box sampling. We explore two surrogate approaches. The first approach exploits zero-order approximation of gradients in the Langevin Sampling and we refer to it as Zero-Order Langevin. In practice, this approach can be prohibitive since we still need to often query the expensive PDE solvers. The second approach approximates the gradients in the Langevin dynamics with deep neural networks, allowing us an efficient sampling strategy using the surrogate model. We prove the convergence of those two approaches when the target distribution is log-concave and smooth. We show the effectiveness of both approaches in designing optimal nano-porous material configurations, where the goal is to produce nano-pattern templates with low thermal conductivity and reasonable mechanical stability.

The paper considers the problem of sampling points from a constrained set in R^d where the constraints can only be accessed in a zero order fashion. They consider the specific situation where the constraints are a solution of a complicated PDE solver and hence the derivatives or specific functional forms of the constraint cannot be obtained. They repose the problem as sampling from a Gibbs distribution whose potential contains constraints as penalties in a Lagrangian fashion. They now wish to sample from the Gibbs distribution using Langevin diffusion.  The Langevin process requires a derivative of the gradient. The setting does not allow for that and therefore the authors propose two approaches -
1. Constructing the gradient from zeroth order entries of a gaussian smoothed potential (much like works of Nesterov et al on zero order optimization).
2. Using a parameteric function class (like an RKHS or a neural network) to learn a function which well approximates the gradient of the constraints as well given zeroth order constraint evaluations.
For the latter, the authors propose two approaches. Hermite learning which directly approximates the error on the gradient as well as the function evaluation. However this involves a separate estimate of the gradient (via a zero order approach such as Gaussian smoothing). An alternative which seems more sample efficient is to do direct learning from zero order samples by penalizing first-order Taylor approximation between given points.
The authors provide a theoretical analysis of the all the approaches and sub-approaches given above and further provide experiments to evaluate this on a problem from material design. I am not at all familiar with the domain of the experiments to comment on competing approaches. As far as I can see the authors also compare only their own proposed algorithms which are many. I will focus on the theoretical analysis which seems to form the bulk of the paper anyway.
The theoretical analysis seems quite rigorous as it begins by first providing a basic guarantee for constrained langevin sampling when gradients are computed with error. The non error gradient part of this analysis has been established before and the authors mention the references appropriately. I have a couple of questions regarding the precise statements of the theorem that i will ask towards the end of the review. Overall its hard to comment on the tightness of the analysis as the non-error versions are also unclear of the tightness of the bounds. Nevertheless the bounds achieved do not look much worse than the non-error counterparts and are easy to implement. The rest of the bounds focus on achieving low error in approximation of the gradients in various settings. Overall the theory in this part seems very loose in terms of bounds as exponential factors in dimensions start to appear and in that regard seems quite preliminary but its hard to comment on whether its natural or can be improved.
Overall the paper is a rigorous treatment of multiple components that would go into the problem of sampling zero order constrained sets and merges many ideas which can all be useful. Nevertheless the paper is a little lacking of novelty in the sense that it brings together many existing ideas and provides an analysis of the effect of bringing them together but none of the theory significantly improves over the existing theory.
Some questions I have regarding the theorem statements
Regarding theorem 1 SPLMC convergence  (and corollaries of the theorem) -  I find it surprising that there is no lower bound assumption on eta in terms of K - only an upper bound. This seems wrong particularly as the theorem as stated then allows eta to be set extremely small while K is finite, in which case there should be no convergence theorems at all. The condition on eta should be a theta(f (K)) for some f type of condition like in the second part of the theorem. I would suggest the authors to look into the theorem - or claify why this is the case.
I am confused by the presentation of the Shi et al results as there is no penalty appearing for the approximation error due to an RKHS, only a finite sample penalty. Does the result assume that phi belongs to the function class of the RKHS in question? Probably yes and in that case that should be specified.


Extending the capabilities of robotics to real-world complex, unstructured environments requires the capability of developing better perception systems while maintaining low sample complexity. When dealing with high-dimensional state spaces, current methods are either model-free, or model-based with reconstruction based objectives. The sample inefficiency of the former constitutes a major barrier for applying them to the real-world. While the latter present low sample complexity, they learn latent spaces that need to reconstruct every single detail of the scene. Real-world environments are unstructured and cluttered with objects. Capturing all the variability on the latent representation harms its applicability to downstream tasks. In this work, we present mutual information maximization for robust plannable representations (MIRO), an information theoretic representational learning objective for model-based reinforcement learning. Our objective optimizes for a latent space that maximizes the mutual information with future observations and emphasizes the relevant aspects of the dynamics, which allows to capture all the information needed for planning. We show that our approach learns a latent representation that in cluttered scenes focuses on the task relevant features, ignoring the irrelevant aspects. At the same time, state-of-the-art methods with reconstruction objectives are unable to learn in such environments.

########### Post-rebuttal summary ############
The proposed method relies on the fact that the distractors have highly unpredictable movements such that a mutual information objective between frames of a sequence learns to ignore them. The experimental evaluations are performed with distractors that randomly change positions / directions, which fits the requirements of the method but not necessarily the behavior of distractors in real environments. To properly evaluate this, experiments in environments with more realistic distractors (who's dynamics are not chosen by the authors) are necessary. Therefore, I do not recommend acceptance of this submission.
########################################
## Paper Summary
This paper combines a mutual information maximisation objective a la CPC with objectives for dynamics and reward prediction to learn a representation for downstream planning / skill learning that is more robust to visual distractors than comparable representations learned with reconstruction-based objectives a la VAE. In particular, the MI objective maximises the mutual information between the representation of the current state and future observations. The authors show improved robustness of their representation to visual distractors over a baseline with pixel-reconstruction-based representation learning (PlaNet). As a result they are able to achieve better performance when model-predictive control is used on top fo the learned representation to perform control on simulated DM Control Suite tasks with added simple visual distractors.
## Strengths
- the paper addresses a relevant problem with an intuitive approach
- the paper is well written and easy to read
- the analytic toy experiments at the end of Sec 4.2 and in 6.2 help understand the properties of the learned representation
- the proposed method, when applied to the model-based algorithm PlaNet, shows improved robustness in settings with artificially introduced visual distractors in simulated DM Control Suite tasks
## Weaknesses
(1) unclear whether artificial distractors are indicative of behavior with real distractors: the fact that distractors do not follow coherent motions but instead randomly change position between two consecutive frames makes it hard to estimate how this would perform on more natural distractors. As there is no MI between the distractor's position in one image and any other image, a CPC-style representation learning objective will naturally be encouraged to ignore them. However, if they move with more natural, coherent motion this might not be the case. I would suggest to add an experiment with more natural distractor motion (see below).
(2) it is unclear how much of the invariance to distractors is coming from the MI objective and how much simply from the fact that the representation is learned with jointly predicting the reward. In order to justify that the MI objective is helpful for learning the representation I would suggest to run an ablation experiment that trains using *only* the reward prediction objective and compare performance. (see also suggested experiment below)
(3) parts of the model formulation require clarification: when reading Sec 4.2 that describes the model some parts were unclear to me (see concrete questions below).
(4) the amount of detail provided in the paper is insufficient for reproducing the results: the paper lacks detailed information about the used architectures, hyperparameters and versions of the baselines employed (e.g. PlaNet with stochastic/deterministic prediction?), code is not provided.
(5) lacks reference to recent work on CPC-style objectives for RL (see suggested references below)
## Questions
(A) why is the latent state variable s_t observed in the model depicted in Fig 3 (i.e. part of the input data)? Shouldn't this variable be latent? How can the KL constraint on it be computed if it is not observed in the input data?
(B) should the first term on the right hand side of equation (2) have \hat{s}_t instead of \hat{s}_{t+h}? Otherwise, how is the I_{NCE} computation conditioned on the current state s_t?
(C) what is the scale of the x-axis in Fig 5, i.e. does it show the number of environment rollouts / steps or the number of model re-training iterations? if the latter is the case, how does that translate to the number of environment interactions?
## Suggestions to improve the paper
(for 1) add an experiment where the distractor has more natural dynamics so that there is MI between the distractor positions in consecutive frames (e.g. ball bouncing in the image frame in the background instead of randomly jumping to new positions)
(for 2) add an experiment with a reward-prediction only baseline, i.e. only action-conditioned reward prediction so that task-irrelevant parts are ignored by default (i.e. also no reconstruction objective, but also no MI objective) -> show how PlaNet performance compares to the so far reported numbers when using this representation for planning
(for 4) add details about the architecture, hyperparameters and training schedule, for both the method and all comparisons to the appendix
(for 5) add references to related works that use CPC-style MI objectives for representation learning in the context of RL/skill learning:
	- [1] Nachum et al., ICLR 2019 -- applies CPC-style objective to hierarchical RL setting
	- [2] Anand et al., NeurIPS 2019 -- investigates MI objectives for representation learning on a wide range of Atari games (don't apply to RL)
	- [3] Gregor et al., NeurIPS 2019 -- while the main proposed model is generative they compare to a contrastive version that uses CPC to learn predictive representations (don't use it for RL)
	- [4] Guo et al., Arxiv 2018 -- similar investigation to [3] of CPC-style objective for representation learning in RL environments (don't use it for RL)
	- it should also be mentioned that the original CPC paper already showed that adding CPC-style auxiliary loss to RL improves performance (even though they did not compare to other model-based methods)
- add qualitative rollouts for predictions from the PlaNet predictive network both with and without distractor to the appendix
## Minor Edit Suggestions
- "Learning latent dynamics from pixels", Hafner et al. is cited twice in the reference section
- it might help to add the reward prediction module to Fig 3 or mention in the caption that it is omitted, it is only described later in the text and was confusing for me on first sight
[Novelty]: okay
[technical novelty]: minor
[Experimental Design]: okay
[potential impact]: high
#######################
[overall recommendation]: weakReject - I am inclined to accept this paper but am not fully convinced that the random distractors provide a good intuition about how the proposed method would behave with more natural distractors. If the authors are able to report positive results on the two requested experiments I am willing to raise my score.
[Confidence]: High
[1] Near-Optimal Representation Learning for Hierarchical Reinforcement Learning, Nachum et al., ICLR 2019
[2] Unsupervised State Representation Learning in Atari, Anand et al., NeurIPS 2019
[3] Shaping Belief States with Generative Environment Models for RL, Gregor et al., NeurIPS 2019
[4] Neural Predictive Belief Representations, Guo et al., Arxiv 2018
### Rebuttal Comment (copied here so that it's visible to the authors) ###
I appreciate the effort the reviewers put into the rebuttal! My main concern was that the mutual information objective only encourages the model to ignore the distractors because they change positions randomly between frames, i.e. there is no mutual information between distractor positions in consecutive frames. On the other hand, if the distractor's motion was perfectly deterministic there would be infinite mutual information between distractor positions in consecutive frames and therefore the representation might exclusively model the distractor. I.e. the more predictable the distractor, the worse the proposed method will perform.
I asked the authors to perform an experiment with a more predictable distractor that bounces in the image frame to test this hypothesis. The authors instead chose a distractor that moves predictably for a few steps before randomly changing movement directions. While this sounds similar it can actually make a big difference depending on how frames are sampled for the CPC objective, i.e. if the required pair of frames is sampled across a random direction change there is again no mutual information between the distractor positions in both frames.
In real scenes the behavior of distractors likely lies somewhere in between these extremes: they will likely not be fully deterministic but certainly not have frequent moments of purely random direction changes. Therefore, to properly evaluate the merit of the proposed approach, experiments on more realistic / previously published environments are needed where the distractor dynamics are given and cannot be altered to better fit the proposed method.
Due to this fundamental concern about the method I cannot recommend acceptance of the submission.


Deep networks face challenges of ensuring their robustness against inputs that cannot be effectively represented by information learned from training data. We attribute this vulnerability to the limitations inherent to activation-based representation. To complement the learned information from activation-based representation, we propose utilizing a gradient-based representation that explicitly focuses on missing information. In addition, we propose a directional constraint on the gradients as an objective during training to improve the characterization of missing information. To validate the effectiveness of the proposed approach, we compare the anomaly detection performance of gradient-based and activationbased representations. We show that the gradient-based representation outperforms the activation-based representation by 0.093 in CIFAR-10 and 0.361 in CURE-TSR datasets in terms of AUROC averaged over all classes. Also, we propose an anomaly detection algorithm that uses the gradient-based representation, denoted as GradCon, and validate its performance on three benchmarking datasets. The proposed method outperforms the majority of the state-of-the-art algorithms in CIFAR-10, MNIST, and fMNIST datasets with an average AUROC of 0.664, 0.973, and 0.934, respectively.

[Summary]
The paper proposed to use gradient-based presentation in deep neural networks to capture missing information unavailable in the activation-based network representation. It is claimed that the missing information that could not be encoded during learning from training set can be revealed by taking the gradient of an example with respect to the model parameters. Based on this idea, a new learning algorithm is proposed to combine both the conventional loss for activation-based presentation and gradient-based regularization. As an illustration, the method is evaluated on anomaly detection benchmarks with gradient-based representation as part of the anomaly inference.
[Comments]
I’m not sure if I fully understand the claimed contribution and how it is justified either theoretically and empirically. It seems to me that the paper claims the the current activation-based network does not fully encode information from training set during learning, and taking the gradient over a novel example (either training or test) with respect to (w.r.t,) model parameters can encode the missing information. I don't quite follow a few things, and had a difficult time in interpreting the novelty here.
- What is the missing information (a model should learn but fails to capture from training set) exactly? The example in the introduction uses digit 0 for training and digit 6 for testing and claims the vertical edge in 6 is missing. But isn’t that expected if the model is only trained with digit 0s only? How would one expect the model to learn the vertical edge if there is none in the training data?
- More critically, the idea of taking the gradient of a model w.r.t.its parameters over an example to encode geometric relationship of the example in the data manifold is by no means novel at the conceptual level. There have been at least a bunch of similar classical methods derived from the perspective of information geometry. For instance, the Fisher kernel and vector  have been well studied for more than a decade, with both theoretical treatment (e.g., “Exploiting Generative Models in Discriminative Classifiers” in NIPS 1998) and application across multiple areas (e.g., “Fisher Kernels on Visual Vocabularies for Image Categorization” in CVPR 2007). I don’t see any novelty here by doing this to a new model (deep neural networks), and it is frustrating to see classical methods are being “invented” over and over again by decorating them with fashionable wrappers without any reference to the origin in the literature.
- Besides, to use gradients to capture missing information, the paper proposes to append a regularization term by enforcing the gradient at particular learning iteration to be close to the mean of those of previous iterations (via cosine similarity) as in (3). This seems not quite different from many existing popular inertia-based strategies like gradients with momentum, Adam, etc. It needs to be elucidated more how the proposed optimizer compares to these benchmark methods.
In terms of implementation, I also found some details missing. For instance, the objective function of (4) requires evaluation of second order derivatives (since the second term in the cosSIM involves dL/dPhi(x, Phi)). How to efficiently compute this term is not clear to me yet.   It is also mentioned that only decoder of the AE is used for constraint gradient, The reason for this is also not explained.
For evaluation, only anomaly detection examples are provided. It would be great if more general tasks like image classification can be studied. After all, the proposed method is claimed to be a fairly general framework (the introduction actually uses image classification as the example), and evaluation on the most popular benchmarks provides the most convincing justification. Even on the anomaly detection tasks, the proposed approach does not seem to have solid advantages over baselines. I’m not sure how significant are the differences of 0.007 between GradCon and OCGAN (table 2 and 3), and those less than 1% in table 4.
Overall the paper does not seem to have a clear and well-defined motivation, contribution is also vague and trivial compared to literature, plus very convincing empirical justification. Thus I do not think it is ready for publication at ICLR.


This work addresses continual learning for non-stationary data, using Bayesian neural networks and memory-based online variational Bayes. We represent the posterior approximation of the network weights by a diagonal Gaussian distribution and a complementary memory of raw data. This raw data corresponds to likelihood terms that cannot be well approximated by the Gaussian. We introduce a novel method for sequentially updating both components of the posterior approximation. Furthermore, we propose Bayesian forgetting and a Gaussian diffusion process for adapting to non-stationary data. The experimental results show that our update method improves on existing approaches for streaming data. Additionally, the adaptation methods lead to better predictive performance for non-stationary data.

Contributions:
The paper extends variational continual learning with memory. In this setting the posterior distribution of the model parameters is approximated using a mean-field Gaussian approximation and a small set of datapoints is kept in a memory to combat catastrophic forgetting.
- The paper proposes a new rule for updating the memory and the Gaussian approximation after examining each batch of points. For the memory update, instead of the previously used k-means or random sampling, it examines the factor 'r(d|w)' that each datapoint contributes and selects the most significant ones.
- A second contribution is the use of two adaptation methods in the online learning setting in case there is a distribution shift in the data: Bayesian forgetting and the Ornstein-Uhlenbeck process.
(1) The first contribution, the new memory update rule and the Gaussian update, is novel to my knowledge. The idea is to calculate the Gaussian factor that each datapoint contributes to the posterior (the update corresponding to each datapoint in the message-passing interpretation) and the select the points that contribute the most. Effectively, it calculates the change in the ELBO if each candidate datapoint was moved to the memory and selects the set that minimizes the ELBO of the remaining points. This  is a sensible heuristic because we want to keep datapoints in the memory that contribute the most to the posterior approximation. If a datapoint contributes nothing to the posterior then it shouldn't be kept in memory.
The idea clever and it seems to work well in practice. Questions:
- The Gaussian that each datapoint contributes is an approximation in the sense that if we take that datapoint and its contribution away, the remaining approximate posterior might be suboptimal. This has the consequence that S_tk (eq. 8) is only an approximation to what the optimal ELBO would be without the datapoints selected for the memory. Did the authors conduct experiments or have thoughts on how well S_tk approximates the ELBO of the variational approximation of the remaining points?
-Regarding the experiments, there is a comparison to k-centre and random selection, both of which are proposed in VCL. I am very surprised to see in the experiments that both of these are outperformed by 'no-memory'. For example on 'energy', the baselines at t50 are outperformed by the non memory model at t0. Can this all be contributed to the baselines training their variational approximation without the memory at t0?
- It is unclear whether the gains can be contributed to the new Gaussian update rule or the new memory update rule. To see the contribution of each, there should be an experiment where the proposed Gaussian update rule is used along with k-centre for the memory update. The fact the the no-memory model performed to close to GRS suggests that k-centre with the new Gaussian update rule would be even closer to it.
(2) The second contribution of the paper is the use of Bayesian forgetting and OU to use to deal with the data distribution shift. It shows how these two approaches can be used along with the proposed Gaussian and memory updates. The paper is already very long, so there likely isn't any space for fleshing out this section, but it would be nice to have the experimental results included in the main paper, because most of the experiments are left to the appendix. Perhaps it would be a good idea to focus only on Bayesian forgetting (or the OU process) and try to shorten the section a bit.
In terms of writing and clarity, I have no complaints. The paper is well written and easy to understand.
Minor:
- 3.2 'If the likelihood term p(dtk|w) is well approximated by r(w;dtk)' -  I find this sentence a bit confusing. What does it mean for p(dtk|w) to be well approximated by  r(w;dtk)?
- 3.3 'In order to reduce the variance of the Monte-Carlo approximation'. What is the MC approximation made here?
- The method never actually uses the assumption that the datapoints are sampled i.i.d.. The algorithm should still work if the datapoints are not examined in a random order, e.g. consider seeing all the images with label '0' and then all the images with label '1' etc.. This would likely degrade the performance significantly but the method should still work.
Overall assessment:
Pros: I like the ideas in the paper and they are presented well.
Cons: The paper is a bit too long. The experiments could more thoroughly investigate the source of the gains.


The significance of computers, handheld devices and educational applications are not deniable in the modern world of today. The interest of applying AI and Machine Learning algorithms in this technologies is growing. Texts are entered to these devices using word processing programs as well as other techniques. Word prediction is one of the techniques and assistive technology for people with disabilities. The main purpose of word prediction is to facilitate data entry and help dyslexic people in reducing writing errors. Dyslexia is a learning problem some kids and few adults have. Dyslexia makes it tough to read and spell. For developing countries such as Ethiopia this kind of problems are neglected and the language spoken within the country are under resourced. Therefore this work has a major contribution to those who have this type of disability in their own language. Prediction can also help education by facilitating data entry. Building prediction model with better speed, grammatically correct suggestions and less search space is the main focus of the research. Amharic is used by a large number of populations, however no significant work is done on the topic of word sequence prediction. In this study, Amharic word sequence prediction model is developed with statistical methods using Hidden Markov Model by incorporating detailed parts of speech tag, some morphological features and user profiling or adaptation. Evaluation of the model is performed using developed prototype and keystroke savings (KSS) as a metrics. According to our experiment, prediction results using a bi-gram with morphological features and detailed Parts of Speech tag model has higher KSS and performed better compared those without Parts of Speech tag. Therefore, statistical approach with detailed POS, morphological features like gender, number, and person with suggested root or stem words using voice, tense, aspect, affixes statistical information and grammatical agreement rules of the language has quite good potential on word sequence Prediction for Amharic language.

I would not like to sound offensive, but this paper is clearly below the standards of the conference, and outside any academic orthodoxy for the matter:
- It is only 3 pages long including references, and does not even follow the conference template.
- It has only 3 sections ("introduction", "methodology" and "conclusions") and 2 subsections ("background of study" and "limitation"), and all of them are only one paragraph long.
- The work done is not adequately described, so it is not possible to say much about it, but it seems clear that there is no novelty nor sufficient rigor in it. The problem tackled is defined as "word prediction", which seems to be some form of language modeling. The proposed method combines HMMs with n-grams and morphological and POS features, which are all well established and should be considered more of a (nowadays outdated) baseline. There is no proper evaluation: the experimental settings are not described, and only one number is reported, with nothing to compare to.
To the authors: Please do not feel discouraged by my review. Your motivation (helping people with dyslexia in Ethiopia) is certainly laudable, but neither the work carried out nor its presentation meets the standards of our research community. I would suggest that you check some of the accepted papers in last year's conference to get a sense of what is expected. I assume that you are new to the field. Don't feel the need to publish your own research from the first day, and take your time to become familiar with the field and study the basic concepts. It takes time, but we all had to go through it :)


Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records. Existing techniques for training differentially private (DP) models give rigorous privacy guarantees, but applying these techniques to neural networks can severely degrade model performance. This performance reduction is an obstacle to deploying private models in the real world. In this work, we improve the performance of DP models by fine-tuning them through active learning on public data. We introduce two new techniques -DIVERSEPUBLIC and NEARPRIVATE -for doing this fine-tuning in a privacy-aware way. For the MNIST and SVHN datasets, these techniques improve state-of-the-art accuracy for DP models while retaining privacy guarantees.

---SUMMARY---
This paper gives two methods for improving an existing differentially private model. Both methods assume a source of unlabeled data that requires no privacy guarantees. The two methods make different uses of this public data to augment the original private model. Both preserve (a degree of) privacy for the original model.
The first method is DiversePublic. DiversePublic only post-processes the original private model M -- in particular, it does not touch the original data -- and so its privacy is immediate. At a high level, DiversePublic works by picking the "best" unlabelled data points for augmenting M. First, it applies M to the unlabeled data and then applies PCA to the result. Next, it selects highly "uncertain" points, projects them onto the top k principal components, and clusters the results. Finally, it selects samples from each cluster and uses them to augment M. This augmenting process, called FineTune, is left as a black box.
The second method is NearPrivate. The given motivation for NearPrivate is that the public dataset may have a different distribution than the original private training data. To address this, NearPrivate uses DP-PCA (a pre-existing technique for differentially private PCA) on the private dataset and projects both the private and public data onto the top k principal components. Then it compares uncertain points in the public and private dataset projections and only augments M (again using the black box FineTune) using uncertain public examples with many nearby uncertain private samples. Intuitively, this should select for points that are not too different than the original private training data.
The paper then evaluates these algorithms with experiments on MNIST and SVHN under reasonable privacy parameters (I think -- see below). These experiments compare the performance of DiversePublic and NearPrivate, where both train M using vanilla DP-SGD [Abadi et al. 2016]. DiversePublic and NearPrivate trade performances on MNIST and SVHN, but NearPrivate does (as might be expected) better when the public dataset is polluted.
In summary, the paper suggests DiversePublic and NearPrivate as useful ways to augment a given differentially private model M using public data, and their experiments suggest this is reasonable.
---DECISION---
Reject.
This is an empirical paper. It proposes some algorithms and justifies these algorithms through experiments. However, the paper makes a confusing omission: it does not mention PATE [1, 2].
[1] "Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data". Papernot et al. 2017.
[2] "Scalable Learning with PATE". Papernot et al. 2018.
I am more of a pure differential privacy researcher and am therefore less familiar with this line of mostly empirical work. But as far as I can tell PATE works in the same setting as that of this paper: there is a private dataset, a public unlabelled dataset, and the goal is to train a model with a differential privacy guarantee for the private dataset. As a result, PATE seems directly relevant. In particular, the claim that "there are no published examples of a differentially private SVHN classifier with both reasonable accuracy and non-trivial privacy guarantees" seems wrong, as the PATE papers include just that.
Moreover, PATE appears at least competitive with the algorithms here. For example, the comparison given as Table 1 in [2] suggests that PATE can get 98-99% accuracy on MNIST for privacy parameters at least as low as those used here with only a few hundred new labelings. In contrast, if I understand the experiments here, they are using thousands of new labelled examples to get the same accuracy and privacy. Similarly, on SVHN the experiments here get to 84% accuracy with 10,000 new labelled examples, whereas the same Table 1 puts PATE at 90% with almost identical privacy parameters and thousands fewer new labelled examples.
That comparison is my main concern. Perhaps I am missing a reason why PATE is not comparable. If it is, it should certainly appear in the experiments. If PATE is indeed comparable, then the main improvement contributed by this paper is the post-processing aspect: PATE is a way to train a private model from scratch, but these methods work on an existing model. Even then, the question of how exactly the existing model is modified (the FineTune method) is unclear.
More broadly, there are several parts of this paper that could be much clearer. I am sympathetic to paper length limitations, but I am certain many or all of these issues can be addressed in 8 pages.
1. DiversePublic description: What does it mean to "obtain the 'embeddings'...E_{public}"? How is k picked?
2. How does the private point-public point assignment in NearPrivate work? Is it just random? How many points do we pick?
3. In general, how is FineTune meant to work? I understand it is meant to be an abstract black box in the algorithm descriptions, but the experiment description doesn't explain it either. How does FineTune work in the experiments?
4. What does 'headroom' mean in Section 4.2?
5. The way privacy parameters are specified at various points in the experiment section makes things hard to read. A concise table summarizing privacy parameters, accuracy, and additional labels would help.
But overall, I am most interested in the PATE comparison.
---EDIT AFTER AUTHOR RESPONSE---
I missed the data-dependent aspect of PATE's privacy guarantee. That is a point in favor of this paper. As such, I've increased my score from reject to weak reject.
I actually think a revised version of this paper could reasonably appear in ICLR or similar venues. However, I think the necessary revisions are too large/numerous to accept the paper in its current form, even with promises of revisions. Here are a few revisions that I think would make the paper a much better submission to future conferences:
1. Add discussion of PATE. The data-dependence of their privacy guarantees is well-taken. But it is not clear that data-dependent privacy guarantees are as "trivial" as the paper currently claims with "there are no published examples of a differentially private SVHN classifier with both reasonable accuracy and non-trivial privacy guarantees" suggests. (In fact, I find it confusing that one of the PATE authors agrees with this claim -- they think their own work is trivial?). I agree that data-independent privacy guarantees are preferable, but this should certainly be discussed. As the authors themselves note, the original PATE paper was well-received, so if this paper wants to claim its privacy guarantees are somehow meaningless or even just unsatisfactory, that claim needs to be defended.
2. Elaborate on the fine-tuning process. Perhaps "fine tuning" is indeed "super standard in the object recognition literature", but as all three reviews here indicate, the presentation of fine tuning is unclear in this paper. The author responses also seem to simultaneously claim that fine tuning is  both "super standard" and "[not] obvious".
3. To make room for the text above, it's not clear that the material about data pollution or even presenting both methods is necessary. I would prefer to see a thorough explanation of the active learning/post-processing paradigm through one algorithm. The authors seem to want to claim that active learning is a useful private approach because it gets data-independent privacy guarantees and performance similar to algorithms with data-dependent privacy guarantees. A paper that focuses on and thoroughly defends that claim actually sounds pretty good!