Model-based reinforcement learning has been empirically demonstrated as a successful strategy to improve sample efficiency. Particularly, Dyna architecture, as an elegant model-based architecture integrating learning and planning, provides huge flexibility of using a model. One of the most important components in Dyna is called search-control, which refers to the process of generating state or stateaction pairs from which we query the model to acquire simulated experiences. Search-control is critical to improve learning efficiency. In this work, we propose a simple and novel search-control strategy by searching high frequency region on value function. Our main intuition is built on Shannon sampling theorem from signal processing, which indicates that a high frequency signal requires more samples to reconstruct. We empirically show that a high frequency function is more difficult to approximate. This suggests a search-control strategy: we should use states in high frequency region of the value function to query the model to acquire more samples. We develop a simple strategy to locally measure the frequency of a function by gradient norm, and provide theoretical justification for this approach. We then apply our strategy to search-control in Dyna, and conduct experiments to show its property and effectiveness on benchmark domains.

Disclaimer:
This is an emergency review that I got assigned on Nov 5th, so I unfortunately had only a few hours to assess the paper.
Summary:
This paper proposes a new mechanism for “search control”, that is, choosing the planning-start-states from which to roll out the model, in the context of Dyna-style updates. The key idea is to favour updates to states where the the value functions changes a lot, because those need more information to become accurate.
The topic is very interesting, and not studied enough, the proposed method looks novel, and the experimental results look promising. However, the clarity of the paper is not great, the theoretical result (Theorem 1) seems to be incorrect, the narrative from Claude Shannon to Joseph Fourier to gradient norms to practical choices is somewhat confusing, the empirical results are not really conclusive, and despite a large appendix there are missing implementation details. I think this paper is currently one or two major revisions away from the acceptance threshold.
Comments:
1. Theorem 1, equation (3): consider a constant function f(x) = K. Then its derivative is zero, so the left-hand side is zero, yet the right-hand side is the product of two strictly positive factors (each of them proportional to K^2), so how can the equation hold?
2. Sine experiment: How can you reasonably do *linear* regression onto a sine function? What are your input features?
3. Figure 2 is odd: panels (b) and (c) look very similar in how they bias the sampling toward the left side, yet the performance difference (a) is very stark, how come? Also, how can there be such a high contrast in (b)/(c) if 60% of all samples are chosen uniformly, as stated in the text?
4. The paper has numerous grammatical mistakes, to the extent it becomes difficult to read. I know this can happen in deadline mode, but please revise the draft thoroughly (special call-out to the dozens of missing definite/indefinite articles and plural forms). Also, use “\citep” where appropriate.
5. The objective g as the sum of a gradient norm and a Hessian norm seems odd, as these terms have completely different scales, so usually one of them will dominate, can you explain and motivate this further, and compare empirically to the two terms in isolation?
7. For the prioritizedER baseline, what variant and hyper-parameters are you using?
8. Please describe “out of boundary” mentioned in Algorithm 1.
9. Section 5.2 states that the “variance of the evaluation curve” is smaller, indicating robustness, yet Figure 3(a) appears to have high instability in the (early) learning curve?
10. Section 5.2 states that Figure 5 should show that the bottleneck areas are sampled more densely, but that’s a dubious claim. Please quantify the claim or drop it.
11. Figure 4(b) has oddly wide error-bars for DQN-Value, which looks suspiciously like a single failed run. Can you add a plot to the appendix with median/quantiles instead of mean/std statistics?
--------------
Update Nov 17: most of my concerns have been addressed, and I have thus increased my rating.


This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

This paper presents a novel method to extract cross-modal text-visual embeddings on the HowTo 100M corpus. The core idea is to extend previous work on clip-level embeddings (e.g. the max-margin ranking loss proposed for HowTo 100M) to a transformer architecture which takes into account the entire context of a video, which should lead to better learned representations and improved performance in downstream tasks. In addition, the max-margin loss is replaced by noise contrastive estimation.
The paper is well written and explains the main problem well, however I do have a few questions:
- I do not understand the sentence "However, for images and videos, the inputs are real-valued vectors." (Section 3.2) - Transformers are being used for speech recognition or speech translation - the input features are not the problem. The outputs are assumed to be discrete (in the original formulation)
- Why not directly compare your approach to the approach presented in (Miech, 2019c) - it would be interesting to see a direct comparison, but as far as I can tell, there is no overlap in tasks?
- What is the influence of adding punctuation to the ASR output, how good is it, and how good is the underlying ASR? Why did you not use the original text annotations provided by HowTo 100M, but run the audio through Google ASR (again?) It would be good to know how good the ASR is, and if adding in punctuation post-hoc works well, and how this influences your use with a pre-trained BERT model. My guess is that the BERT model will be happy as long as it sees a "." at the end?
- Also, would it be possible to compare the results of your work with some of the work in (Miech, 2019c) - it almost seems that your work avoids comparing your results to this previous work.


Improving the sample efficiency in reinforcement learning has been a longstanding research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires O(1/ 3/2 ) 1 episodes to find anapproximate stationary point of the nonconcave performance function J(θ) (i.e., θ such that ∇J(θ) 2 2 ≤ ). This sample complexity improves the existing result O(1/ 5/3 ) for stochastic variance reduced policy gradient algorithms by a factor of O(1/ 1/6 ). In addition, we also propose a variant of SRVR-PG with parameter exploration, which explores the initial policy parameter from a prior probability distribution. We conduct numerical experiments on classic control problems in reinforcement learning to validate the performance of our proposed algorithms.

Summary:
The paper proposed a policy gradient method called SRVR-PG, which based on stochastic recursive gradient estimator. It shows that the complexity is better than that of SVRPG. Some experiments on standard environments are provided to show the efficiency of the algorithm over GPOMDP and SVRPG.
Comments:
1) I think you may ignore the highly-related work: Yang and Zhang, "Policy Optimization with Stochastic Mirror Descent", June 2019. Since both papers are highly-related, I would suggest the author(s) have some discussions to differentiate two papers.
2) Could you please provide the reasons why you are only choosing two methods (GPOMDP and SVRPG) to compare? There are many policy gradient algorithms such as A3C, A2C, ... which have not mentioned or discussed in this paper. Are they completely different here? Is there no way to compare the performance among them with SRVR-PG?
3) I am not sure if you are using the right word "novel" to describe your method. Basically, you adopt the existing estimator based on SARAH/SPIDER in optimization algorithms into RL problems. I do not think the word "novel" is proper here since it is not something total new. Notice that, the complexity result achieved in this paper is also matched the one for SARAH/SPIDER/SpiderBoost in nonconvex optimization.
4) There are some discussion on choosing a batch-size in the experimental part. However, I do not see the discussion on the learning rate. The choice of parameters for GPOMDP and SVRPG may also need to be discussed.
5) In Papini et al., 2018, for the experiment part, they use a snapshot policy to sample in the inner loop, and use early stopping inner loop. Moreover, they also check variance to recover the backup policy when it is blowup. Do you apply any trick to your experiments? I wonder if your numerical experiments are totally followed on your theory.
Minor comments:
- Redundancy ")" in \eta*S*m in Theorem 4


Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promising in delivering high-quality results but challenging to meet stringent latency requirements because of the memory-bound execution pattern of RNNs. We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference. Leveraging the error-resilient feature of nonlinear activation functions used in RNNs, we propose to use a lightweight little module that approximates the original RNN layer, which is referred to as the big module, to compute activations of the insensitive region that are more errorresilient. The expensive memory access and computation of the big module can be reduced as the results are only used in the sensitive region. Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact on model quality.

In this paper, the authors design a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference.
It cannot only mitigate the memory-bound problem to speedup RNN inference but also leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation. They also conduct several experiments to evaluate their approaches.
Strength:
(1)	Well written in general.
(2)	Contributions clearly stated and justified
A couple of minor questions:
(1)	The organization in Section 3 can to be improved. It is better if the author can give a brief overview of their method first and then go into details.
(2)	Some of the technical details necessary for understanding the soundness of the techniques are either missing or are poorly explained. For example, in Section 3
a.	the authors did not mention how to construct the HH module
b.	the authors did not provide detailed information of how to conduct dimension reduction since this will affect the performance
c.	many mathematical notations and equations need to be revised to increase the readability. For instance, there is no information in the paper that explain why the authors design functions in such certain way (such as equation (2) and (3))
d.	the authors did not provide enough detailed information about how to select the quantization methods since there are lots of approaches such as static (uniform) or dynamic quantization, where different methods may have different impacts on the final performance
e.	the authors mentioned that they have tried both sigmoid and tanh activation function to find the sensitive region. However, they do not provide enough reason to do so, how about using other non-linear activation functions
(3)	The organization in Section 4 can to be improved. It is better if the author can introduce the motivation of each experiment.
(4)	Parameters of the evaluation are unclear or missing. For example,
a.	what is the data size, what is the dropout, learning rate, how many time stamps for the RNN modules
b.	why the authors only use single-layer LSTM and why they select 750 and 1500 hidden units in the experiments
(5)	While the authors have applied their models on other existing method, they do not provide good discussion of results and such model seems old (released in 2016). It would be great if this approach can also be applied on other newly released models
(6)	Some tables need to be reorganized. For instance, for table 6, there needs some space between the title and the table.
(7)	While the paper has good coverage of the prior work, I do suggest the authors can also cite or discuss some newly designed models (in 2019).


Real-world question answering systems often retrieve potentially relevant documents to a given question through a keyword search, followed by a machine reading comprehension (MRC) step to find the exact answer from them. In this process, it is essential to properly determine whether an answer to the question exists in a given document. This task often becomes complicated when the question involves multiple different conditions or requirements which are to be met in the answer. For example, in a question "What was the projection of sea level increases in the fourth assessment report?", the answer should properly satisfy several conditions, such as "increases" (but not decreases) and "fourth" (but not third). To address this, we propose a neural question requirement inspection model called NeurQuRI that extracts a list of conditions from the question, each of which should be satisfied by the candidate answer generated by an MRC model. To check whether each condition is met, we propose a novel, attention-based loss function. We evaluate our approach on SQuAD 2.0, NewsQA and MS MARCO datasets by integrating the proposed module with various MRC models, demonstrating the consistent performance improvements across a wide range of existing methods.

Paper Summary:
This paper proposes a neural question requirement inspection models called NeurQuRI.  It is different from existing answer verifiers in that NeurQuRI pinpoints where the mismatch occurs between the question and the candidate answer in unanswerable cases. Experiments with SQuAD 2.0 show the effectiveness of NeurQuRI.
Strengths:
—NeurQuRI improved the accuracy of three popular reading comprehension models, which are BERT, DocQA and QANet, on SQuAD 2.0.
—NeurQuRI requires 33M additional model parameters to check answerability of the candidate answer.  It is quite less than those of the previous answer verifier (150M) proposed by Hu et al. (2019).
—NeurQuRI can be easily extended to other question answering models and tasks.
Weaknesses:
—The authors do not experimentally compare their model to the answer verifier proposed by Hu et al. (2019). The base model of Hu et al. (2019), RMR, is different from the base models used in this paper, BERT, DocQA, and QANet.
—The authors use only one dataset (SQuAD 2.0) to evaluate their method.  NewsQA, MS MARCO, CoQA and QuAC also contain unanswerable questions.
Comments/Suggestions:
—I think that coverage mechanisms used in NMT (Tu et al., 2016) and summarization (See et al., 2017) should be included in the section of related work.
—The idea similar to NeurQuRI is used in a multi-hop QA model, proposed by Nishida et al. (2019), to find the evidence sentences that cover important information with respect to the question sentence.
—It is worthwhile to present the results of ablation tests with respect to the modified answerability label (described in Section 2.2).
Review Summary:
The paper is well motivated.  However, I think the authors need to compare NeurQuRI with the answer verifier proposed by Hu et al. (2019).  Also, I think this paper can benefit a lot with a more comprehensive analysis with other datasets such as CoQA.
References:
Hu et al.: Read + Verify: Machine Reading Comprehension with Unanswerable Questions. AAAI 2019: 6529-6537
Tu et al.: Modeling Coverage for Neural Machine Translation. ACL (1) 2016
See et al.: Get To The Point: Summarization with Pointer-Generator Networks. ACL (1) 2017: 1073-1083
Nishida et al.: Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction. ACL (1) 2019: 2335-2345
EDIT Nov. 20, 2019:
I appreciate the authors' revision.
The revision has satisfied my concerns, and I decided to increase the score of the paper (weak reject -> weak accept).


Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., L 2 regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvements on the task performance, and the improvement is typically more significant when the task is more difficult. We also compare with the widely used entropy regularization and find L 2 regularization is generally better. Our findings are further confirmed to be robust against the choice of training hyperparameters. We also study the effects of regularizing different components and find that only regularizing the policy network is typically the best option. We hope our study provides guidance for future practices in regularizing policy optimization algorithms.

This paper investigates the use of conventional regularizers for neural networks in the reinforcement learning setting. Contrary to the standard practice of foregoing regularizers in deep RL, the paper finds that their addition can improve the performance of policy gradient algorithms on a standard suite of continuous control tasks. Various regularizers are tried, including l2/l1 regularization, entropy regularization and dropout in a combination with a few standard deep RL algorithms such as TRPO, PPO and SAC. Other experiments also verify the impact of these regularizers on the sensitivity of other hyperparameters and whether regularization should be applied to the value or policy networks.
Overall, I find this paper to be a solid empirical study of regularization in deep reinforcement learning. The experiments are thorough, with various aspects being examined in more detail. I find several of the findings interesting, such as the importance of regularizing solely the policy network and that batch norm/dropout are effective for off-policy methods but not on-policy ones. There were certain points which warranted some further clarification.
I would be willing to increase my score based on the authors' response to the following points:
1) In section 6, the last two sentences ("For A2C, TRPO, and PPO ... so further regularization is unnecessary.") are unclear to me.
	- "rewards are already normalized using running mean filters." I thought that rewards are also normalized for SAC, so I'm not sure how this could explain the difference between the on-policy algorithms and the off-policy ones.
	- "mitigates the overestimation bias...further regularization is unncessary." Could you clarify the connection between regularization and overestimation bias? Related to this point, in section 2 of the paper, it is written that "L2 regularization is applied to the critic Q network because it tends to have overestimation bias (Fujimoto et al., 2018)" but I was not able to find such an explanation in the cited paper though I may have missed it.
2) In section 7, in the paragraph on BN/Dropout, could you clarify the point starting from "1) For both BN and dropout,..."? In particular, which discrepancy between the sampling policy and the optimization policy is being referred to here?
3) Did you consider trying weight decay ("Fixing Weight Decay Regularization in Adam", Loschilov et al. 2018) as a regularizer? Given the success of L2 regularization, it could be possible that weight decay is even more effective.
4) For the hyperparameter sensitivity plots, where one hyperparameter is varied at a time, why are the step sizes for the policy and value networks not included in these experiments? They are usually a critical hyperparameter.
Minor comments and typos:
- On p.5, when defining "hurting", perhaps it could be better to choose a looser definition such as "\mu_r < \mu_b" or "\mu_r - \sigma_r < \mu_b - \sigma_b". This way, there could be a larger distinction between the most effective methods. Currently, both l2 and entropy regularization achieve 0.0% and the next best two regularizers are also under 10%.
- In abstract: "regularizing the policy network is typically enough." Rephrase perhaps? The experiments seem to show that applying a regularizer to only the policy network is better than on both.
- In abstract: "large improvement" -> "large improvements"
- p.2, par. 2: "those regularizations" -> "those regularizers"
- p.3, Weight Clipping: "This greatly stablizes" -> "This greatly stabilizes". This sentence could be rephrased since "This" seems to refer to only weight clipping, but is not the only change in WGANs.
- p.3, Dropout: "regularization technique" -> "regularization techniques"
- p.4, par. 1: "due to more stochasticity" -> "due to increased stochasticity"
- p.4, 2nd to last par.: "during policy update" -> "during policy updates"
- p.5, 2nd to last par.: "sometimes help" -> "sometimes helps", "easier ones baseline is" -> "easier ones the baseline is"
- p.8, 2nd to last par.: "it naturally accepts" -> "they naturally accept", "been shown effective" -> "been shown to be effective"
- p.8, last sentence: "policy network without the value network." -> "policy network but not the value network."


Deep neural networks have gained tremendous success in a broad range of machine learning tasks due to its remarkable capability to learn semantic-rich features from high-dimensional data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption into unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications into critical domains where obtaining massive labelled data is prohibitively expensive. To enable downstream unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space. Random mapping is a theoretical proven approach to obtain approximately preserved distances. To well predict these random distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space. Experimental results on 19 real-world datasets show our learned representations substantially outperform state-of-the-art competing methods in both anomaly detection and clustering tasks.

###### Overall Recommendation
I vote for the “Weak Accept” decision for this paper.
### Summary
This paper introduces a novel model termed Random Distance Prediction model, it can predict data distances in randomly projected space. The distance in projected space is used as the supervisory signal to learn representations without any manually labeled data, avoiding the concentration and inefficiency problem when dealing with high-dimensional tabular data. Their main contribution is extending the random distance in projected spaces to approximate the original distance information of the hight-dimensional tabular data effectively. Overall, the idea in this paper is interesting and effective, the experiment results in two typical unsupervised tasks (anomaly detection and clustering) also look very promising. However, the writing sometimes has unclear descriptions, given these clarifications in an author's response, I would be willing to increase the score.
### Strengths
1. The illustration of the authors' idea is clear and concise.
2. The theoretical analysis of the proposed method is solid and systematic, the validity of the subparts have been proven previously.
3. The experiment part is well organized. The RDP model are compared with several state-of-the-art unsupervised learning methods in 19 real-world datasets of various domains. The experimental setup is solid with realistic considerations, the results are very convincing and promising.
4. This paper provides sufficient detail for reproducing the results.
### Weaknesses
Lack of systematic description of the authors’ major contribution. The proposed model looks more like a combination of previous conclusions, which makes readers feel the core parts of this paper build heavily on previous work.
### Questions
1. What is the relation between “random distance prediction loss” and “task-dependent auxiliary loss”?
2. Are there any solutions and references about how to choose the task-dependent loss L_aux?
3. Why you shade the second part of the loss function in Figure 1;
4. How long is it for training the proposed model and getting the experiment results? Does the RDP model still outperform the other algorithms?
### Suggestions to improve the paper
1.  It would be better to reorganize Section 1 and Section 2, please describe the contribution in a more systematic way.
2. Add details for the architecture of the model, please give more descriptions about Figure 1.
3. It might also help to add an algorithm comparison box for the test time for the proposed method.
### Minor Edit Suggestions
1. It would be better to give more descriptions about Figure 1; the lower right part in Figure 1 is not explained in the caption; the shadow part in Figure 1 is not precise.
2. Figure 1 was bad organized, please make the legend readable size.
3. I don't think there exist the proofs of Eqns. (2)-(4) in the reference paper (Vempala, 1998), which was written in Page 4. The number should be revised.


Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time. We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time. Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.

This paper propose a video instance embedding loss for jointly tackling the instance tracking and depth estimation from self-supervised learning.
Pros:
1: I think the method is moving towards the right direction that 3d geometry and 2d instance representation should be considered jointly under the scenario of video learning.
2: The video instance embedding loss is also making sense as an extension of image instance embedding.
Cons:
1: I think the major argument I have is this method is lack of technical novelty, since it is straight forward to adopt the loss of  Brabandere et.al 2017 to video cases for including pixels in the same group under ground truth tracking, and the self-supervised loss is exactly the same as previous methods. The fusion between depth and segments are relatively weak since it just ask the embedding to also decode depth, is there any further analysis of visual effect of explaining where the depth helps segments?
2: In the experiments, the baseline for comparison over MOTS is fairly old, and I think it makes sense to include the number of MOTS paper, which is currently hard to align with that shown in the paper.  In Tab.2, the author only highlight the improved motion metric, while in per-frame AP the results are actually lower than the baselines. It also needs to be well explained.
3: The paper claims `"it generates temporal consistent segmentation " (which is not guaranteed, maybe just statistically better but not exact).
Overall, in my opinion I suggest it to be a workshop paper, but the contribution is somehow not significant for a major publication.


The information bottleneck method of Tishby et al. (2000)  provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superfluous. In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label in unknown. This enables us to identify superfluous information as that which is not shared by both views. A theoretical analysis leads to the definition of a new multi-view model which produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.

This is a good multiview representation learning paper with new insights. The authors propose to learn variables z_1 and z_2, which are consistent, contain view-invariant information but discard as much view-specific information as possible.
The paper relies on mutual information estimation and is reconstruction-free. It is mentioned in some previous works (e.g. Aaron van den Oord et al. 2018), that reconstruction loss can introduce bias that has a negative effect on the learned representation.
Comparing to existing multiview representation learning approaches that try to maximize the mutual information between learned representation and the view(s), this paper clearly defines superfluous information that we should try to throw away and figure out how to obtain sufficiency learned representation for output. The authors also draw clear connections between a few existing (multiview) representation learning methods to their proposed approaches.
The experimental results on the right side of Figure 3, deliver a very interesting conclusion. In low-resource case, robust feature (obtained by using the larger beta, discarding more superfluous information) is crucial for achieving good performance. While when the amount of labeled data samples is enough, vice-versa.
Here are my major concerns:
1.	In the paper, the authors said the original formulation of IB is only applicable to supervised learning. That is true, but the variational information bottleneck paper [Alexander A. Alem et al. 2017] already showed the connection of unsupervised VIB to VAE in the appendix.
2.	I would not consider the data augmentation used to extend single-view data to “pseudo-multiview” as a contribution. This has been done before (e.g. in the multiview MNIST experiment part of the paper "On Deep Multi-View Representation Learning").
3.	Which MV-InfoMax do you really compare to? You listed a few of them: (Ji et al., 2019; Henaff et al., ´ 2019; Tian et al., 2019; Bachman et al., 2019) in the related work section.
4.	I think the authors should also make a more careful claim on their results in MIR-Flickr.
I’d rather not saying MIB generally outperforms MV-InfoMax on MIR-Flickr, as MIB does not (clearly) outperform MV-InfoMax when enough labeled data is available for training downstream recognizers. But MIB does clearly outperform MV-InfoMax when scaling down the percentage of labeled samples used.
5.	Regarding baselines/experiments
a.	In Figure 4, it seems that VAE (with beta=4) outperforms MV-InfoMax. Why the ``"pseudo-second view" does not help Mv-Infomax in this scenario? Why VAE is clearly better than Infomax?
b.	In Figure 3, you might also tune beta for VCCA and its variants, like what you did for VAE/VIB in a single view.
6.	Do you think your approach can be extended to more than two views easily?
For me, it seems the extension is not trivial, as it requires o(n^2) terms in your loss for n views.
But this is minor.


The character-based model, such as BERT, has achieved remarkable success in Chinese named entity recognition (NER). However, such model would likely miss the overall information of the entity words. In this paper, we propose to combine priori entity information with BERT. Instead of relying on additional lexicons or pre-trained word embeddings, our model has generated entity classification embeddings directly on the pre-trained BERT, having the merit of increasing model practicability and avoiding OOV problem. Experiments show that our model has achieved state-of-the-art results on 3 Chinese NER datasets.

Summary:
      This paper discussed an approach to do named entity resolution (NER, the paper focuses only on Chinese NER but I think it could generalize to other languages as well). The idea is based on smart integration and extension of multiple existing building blocks: 1) BERT pre-trained model 2) a previous work to get document embedding by doing weighted average of word embedding (https://openreview.net/pdf?id=SyK00v5xx) and 3) Scaled dot-product attention mechanism applied directly to multi-label classification. The "Introduction", "Related work", and "Experiment Settings" sections are well written and covers many details and decent references. Especially, the "experiments" section is described in a great amount of details, which should be very helpful for reproducibility.

Contributions:
      * The author found an interesting application of the original algorithm (https://openreview.net/pdf?id=SyK00v5xx) to represent the entity class embedding based on averaging "BERT" embeddings of all the component words. This could be implemented as a pre-processing step against any training dataset to derive "pre-learned" entity class embedding.
      * Instead of the common approach of connecting the BERT sequence outputs directly to CRF layer, the author added an intermediate layer to calculate the classification attention between a sentence (sequence of token embedding) and any entity class (based on the above pre-learned entity embedding). This result plus the original sentence embedding are concatenated.  The concatenation is further fed into a few additional layers to produce the final inputs into CRF layer.
Weakness:
     * The paper lacks novelty. As pointed above, I did not see that the contribution from the paper is sufficiently original. It is a good application of various existing methods though.
I also have a few suggestions/questions below:
* The ERNIE paper (https://arxiv.org/abs/1907.12412v1) is mentioned in the related work. Since ERNIE can potentially learn a good vocab for Chinese, did you ever compare your approach vs ERNIE+CRF?
* There is one paper that I know which is pretty relevant to what you are doing here, which is probably worth a reference. https://arxiv.org/abs/1805.04174.  In that paper, the idea is to co-learn a class embedding and perform text classification. Their class attention is performed through dot-production attention though.
* The Table index seems wrong in your paper. (I think Table 2 is not mentioned in your paper, but all tables (3-6) is offset by 1).
* There are some minor typos or places that need some clarifications.
   - in the abstract: "character-based" model. This is a little confusing. Because BERT is a word-piece based model. word-piece could across multiple characters for English. IIUC, You probably want to say "Chinese-character" instead of character.
   - in "Introduction", "providing greater weight to characters identical to each entity class", you might want to revise this sentence to clarify its meaning further.
   - In section 3.2, you might want to give some explanation to some notations (the first time you refer to it). For example, what is $L$, what is $m$ and $n$.  What is $S$?  Also why the denominator of Emb(Word) is not $n-m+1$?
   - The last paragraph in section 3.3 needs more clarification as well. How do you merge the three tensors after attention stage? (a concatenation ?) . The last sentence mentioned "residential", I guess instead you want to say "residual".  You might also want to clarify where the "3 layers" of residual appear in your network.
  - In your experiment, (if I did not miss), did you freeze the BERT parameters and entity embeddings when finetuning your NER model?
  - in Table 2 and Table 3, why the 13-layer BERT + CRF performs significantly worse on Recall (Table 2) and significantly better on Recall (Table 3)?


Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialogue modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images, and human-written dialogue utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.

### Summary

This paper provides a framework to augment dialogue generation with external data sources using K-Nearest Neighbors in the embedding space. The idea seems simple and intuitive, and the results show improvements over prior work in dialogue generation and retrieval.

### Strengths
- The paper shows quantitative improvement over some prior works on dialogue agents (however this needs to be correctly validated).
- The reviewer appreciates the human study and conversation example provided in the paper to qualitatively evaluate their model.
- The paper provides ablation studies of the various training tricks for each dataset they have proposed.

### Weaknesses
- There are certain changes in the training pipeline that makes the comparison with prior work difficult (Tables 1 and 2), and find the real contribution of data-augmentation caused by KNN-based information fetching (KIF). e.g., In Wizard of Wikipedia experiment, most recent dialogue utterance and turn number are used as salient features. Similarly, the paper utilizes a "personality" feature in ImageChat dataset. It seems the results taken from prior papers have different training settings. Could the authors verify that the extra assumptions made for their model are equivalently applied to other generative baselines? If not, the reviewer recommends to provide some experiments with same conditions.
- Could the authors provide the embedding dimensions and other training details, to assess the contribution of different components of the input's embeddings.

#### Minor:
- Is there a sound reason behind using concatenation of input representation and fetched representations? This design choice makes the architecture inflexible to the number of data sources. Another way to combine embeddings - e.g. addition or inner product, can also be tried to see if they provide performance improvements.
- It is mentioned that attention based modules scale poorly with large sized datasets. If the authors conducted a quantitative evaluation to test this, it would be valuable to add it to the paper.
- The title and abstract should be limited to generative dialogue modeling, instead of transformers, since the contributions proposed are more suited for this particular application and are quite engineered. Hence, it is not correct to make this claim for transformers in general.

### Score
Weak Reject (Leaning towards Accept if appropriate experiements can be provided)


Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks.

This paper proposes a new algorithm which brings closer SGD and Adam while also incorporating new corrections to improve behavior of the optimizer in contexts where there is very small or very large eigen values.
Decision
I vote for weak rejection because the core modification proposed to Adam is minor and is mostly supported by intuition and preliminary experiments.
Justification
There is many modifications proposed, but most are secondary corrections for stability, such as the warm-up schedule with the redefinition of beta_2. These modifications could as well be incorporated in Adam without the core modification that is the smoothing presented in section 3. These additional modifications also make it difficult to measure the importance of the core contribution. Without getting rid of them, an ablation study on toy problems (even synthetic data) would be necessary for a better understanding.
In section 3.1, the temporal definition of beta_2t is integrating a warm-up. While the reason for doing so is supported in introduction of section 3, the effect of this modification should be weighted against no warm-up, and also compared with its effect on Adam.
There is an error in algorithm 1. The last element of the last line (Perform the update) should be \alpha (1 - \eta) m_t/d_t. The code in Appendix corroborates this correction. Minor related note, the use of d_t to define the denominator of what d_i represents in section 3 is very confusing. I would suggest to use the ratio notation of d_i from the equations in the algorithm for coherence.
If we get pass the warm-up scheduling, by massaging the equation we get that the algorithm is different from Adam on 2 points, 1) the bias are not corrected and 2) the denominator sqrt(v) + epsilon is replaced by sqrt(v) + sqrt(mean(v) + epsilon^2). I have difficulty convincing myself that smoothing by the average is solving the issues raised in the paper and there is no experiments to study its effect directly.
The experiments are on 3 datasets, but only the computer vision ones are run on multiple architectures. Caption of figure 1 explains that each model is trained 3 times, but the source of variation between each run is not described. Are the models initialized differently? In any case, there is an overlap for 3 of the 5 models between SGD and SoftAdam which makes the comparison rather unconvincing. There is no standard deviation for Adam, and none on Penn Treebank dataset and IWSLT. For a better comparison, all hyper-parameters of the algorithms should be optimized for each run. I understand that SoftAdam is meant to be close to both SGD and Adam, but using the same hyper-parameters may induces misleading results by favoring some (model, optimizer) combination nevertheless.
Minor comments
In section 2, second paragraph, the term 'mini-batch' should be used instead of 'batch'.
In section 2, last sentence, the betas should have no t.
In section 2.1, fourth equation (unnumbered), the eigen vector xi_i is presented as a vector and then used as a scalar. Notation should be uniformed.
In Section 2.1 around equation (2), the use of i and j is incoherent.
In Section 3:
- Overall, this understanding *of* has
- we consider the *an* update


Self-attention-based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called Explicit Sparse Transformer. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model.

CONTRIBUTIONS:
C1. Sparse Transformer: A modification of the Transformer, limiting attention to the top-k locations. (That is a complete statement of the proposed model.)
C2. Experiments showing that, quantitatively, the Sparse Transformer out-performs the standard Transformer on translation, language modeling, and image captioning.
C3. Experiments showing that, qualitatively, in translation, when generating a target word, the Sparse Transformer better focusses attention on the aligned source word
RATING: Reject
REASONS FOR RATING (SUMMARY). The innovativeness seems low given the several previous proposals for sparse attention, the results are not dramatic enough to compensate for the lack of originality, and the comparison to other models is wanting.
REVIEW
Strengths: The paper is clearly written. The question of whether the Transformer’s attention is too diffuse is of interest. The proposal is admirably simple. The quantitative metrics include comparison against many alternative models.
Weaknesses: A primary area of deficiency concerns the relation of the proposed model to other proposals for sparse attention: the authors cite 5 of them (and 2 more are cited in the comment by Cui). The paper should clearly identify the differences between the proposed model and earlier models: it does not discuss this at all. The deficiencies in these previous models should be clearly stated and demonstrated:  they are only described as “either restricted range of attention or training difficulty” (Sec 6). A rationale for why the proposal can be expected to remedy these deficiencies should be stated clearly: it is not stated at all. Experimental demonstration that the proposed innovation actually remedies the identified deficiencies should be provided, but is not.
A proposal to use a top-k filter immediately raises the question of the value of k. This is not discussed at all. In particular, no empirical results are given concerning the sensitivity of the reported successes to choosing the correct value for k. We are only told that “k is usually a small number such as 5 or 10” (Sec 3). The experimental details in the appendix do not even state the value of k used in the models reported.
It is an interesting discovery that in the translation task, attention at the top layer of the standard Transformer is strongly focused on the end of the input. This is described as an “obvious problem” (Sec 7). But it can’t obviously be a problem because the performance of the standard Transformer is only very slightly lower than that of the Sparse Transformer: if anything is obvious, it is that processing in the standard Transformer packs a lot of information into its final encoding of the end of the input string, which functions rather like an encoding of the entire sentence.
Presumably, the experimental results reported are those from a single model, since we are not told otherwise. There should be multiple tests of the models with different random initializations, with the means and variances of measures reported. It is possible, however, that limitations of computational resources made that infeasible, although the Appendix seems to indicate that no hyperparameter tuning was done, which greatly reduces computational cost.
COMMENTS FOR IMPROVEMENT, NOT RELEVANT TO RATING DECISION
Although the tiny sample of visualized attention weights provided is useful, a large-scale quantitative assessment of a main claim concerning translation might well be possible: that attention is in fact concentrated on the aligned word might be testable using an aligned bilingual corpus or perhaps an existing forced aligner could be used.
Much space could be saved: it is not necessary to review the standard Transformer, and the modification proposed is so simple that it can be precisely stated in one sentence (see C1 above): the entire page taken up by Sec. 3 is unnecessary, as it adds only implementation details.
Errors that took more than a moment to mentally correct, all on p. 12:
The definition of the BPC should be E[log P(x(t+1) | h(t))]: all parentheses are missing
“regrad” should be “regard”
“derivative” should be “differentiable” in the final sentence


This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piecewise linear non-linearity activations. We use tropical geometry, a new development in the area of algebraic geometry, to provide a characterization of the decision boundaries of a simple neural network of the form (Affine, ReLU, Affine). Specifically, we show that the decision boundaries are a subset of a tropical hypersurface, which is intimately related to a polytope formed by the convex hull of two zonotopes. The generators of the zonotopes are precise functions of the neural network parameters. We utilize this geometric characterization to shed lights on new perspectives of three tasks. In doing so, we propose a new tropical perspective for the lottery ticket hypothesis, where we see the effect of different initializations on the tropical geometric representation of the decision boundaries. Also, we leverage this characterization as a new set of tropical regularizers, which deal directly with the decision boundaries of a network. We investigate the use of these regularizers in neural network pruning (removing network parameters that do not contribute to the tropical geometric representation of the decision boundaries) and in generating adversarial input attacks (with input perturbations explicitly perturbing the decision boundaries geometry to change the network prediction of the input).

1. Summary of the paper
This paper describes the decision boundaries of a certain class of
neural networks (piecewise linear, non-linear activation functions)
through the lens of tropical geometry.
An earlier result by Zhang et al. (2018) is extended to multi-class
classification problems (technically, only the result for a binary
classification is given in the main text, though).
Similar to this earlier work, the network is shown to be represented as
a tropical rational function. The dual subdivision of this function is
shown to be represented as the convex hull of two zonotopes.
This characterisation is used to explain different phenomena of neural
network training, viz. the 'lottery ticket hypothesis', network
pruning, and adversarial attacks.
2. Summary of the review
This is a highly interesting paper with a very relevant subject. I think
that the perspective of tropical geometry leads to valuable insights. My
background is *not* in tropical geometry, so this paper required several
passes to fully grasp.
I like the novel insights that this paper creates; it is very interesting
to observe known phenomena via tropical geometry. I appreciate the
thorough description of all concepts in this paper. This is to some
extent both 'boon and bane': on the one hand, the paper contains a lot
of information and concepts that need to be understood; on the other
hand, the experiments go *wide* but not *deep*. I suggest to accept the
paper, but to fully endorse it, I would recommend to work on the
following issues:
- Clarity & exposition: In some places, the paper could build intuition
  for non-experts (such as myself) better. This is closely tied to the
  second point.
- Focus: I would maybe pick *one* or *two* of the experimental areas and
  use the remaining space to explain all concepts in more detail, build
  some intuition, and provide a more in-depth setup. Nothing has to
  removed of course; it could still be put in the appendix.
This paper has the potential to be a very strong insightful contribution
to to our community; the authors are to be commended!
3. Detailed comments (clarity)
The paper describes its concepts well and has a high information
density. At times, there is the risk that readers are provided with too
much information in the main text, leaving the necessary intuition
somewhat lacking (unless they are already experts in the subject matter,
in which case a lot of the information can be skipped).
I realise that writing a paper based on methods that are not yet
well-established is no small feat; the authors are to be commended for
that!
Here are some suggestions from someone with a background in differential
topology:
- The introduction and contributions are somewhat repetitive; I would
  suggest merging the 'Contributions.' paragraph with the one preceding
  it
- Even though it *should* be a well-known definition, I would briefly
  explain that the semiring lacks an additive inverse
- Add an explanation of the tropical quotient to Definition 1; I find
  the current phrasing of Definition 3 to be confusing at first glance
- What are *upper faces*? Faces with a specific coordinate fixed?
- A definition of $\pi$ is required. It is my understanding that $\pi$
  is a projection function that 'drops' the last coordinate. Is this
  correct? If so, it should be briefly mentioned on p. 3; else, the
  discussion about the bias-free case on p. 4 cannot be understood.
- Theorem 2 lacks an 'intuitive' formulation; the results are stated in
  a terse mathematical fashion, but it would be helpful (also in light
  of the subsequent discussion) to briefly comment on their *meaning*.
  For example, the first result could be restated as 'the decision
  boundary is a subset of a tropical hypersurface of the polynomial
  $R(x)$'.
  The paragraph 'Theorem 2 bridges the gap...' could maybe also be moved
  to _precede_ the theorem statement.
- The paper refers to $\mathcal{T}(R(x))$ as a super-set, but in my
  understanding, it is a *level set* because Definition 4 uses an
  equality, not an inequality. Am I misunderstanding this? One potential
  misinterpretation of my part could be that superset refers to the fact
  that $\mathcal{B} \subseteq \mathcal{T}(R(x))$; so not a superset
  in the sense of level set analysis, but rather a superset in terms of
  set theory. If this is the case, maybe rephrase the sentence above to
  something like 'boundaries $\mathcal{B}$ through their superset
  $\mathcal{T}(R(x))$ according to the first statement of Theorem 2'.
- What does the colour map in Figure 2 depict? The number of iterations?
  Moreover, I find the polytope though to understand at first glance;
  how is it related to the decision boundaries that are shown in the
  leftmost figure?
- I would suggest to place Figure 1 after stating Theorem 2, since it is
  only referenced later on. Furthermore, the red structures are somewhat
  confusing. According to Theorem 2, the decision boundary is a subset
  of the hypersurface, right? What is the relation of the red structures
  in the convex hull visualisation? The caption states that they are
  normals, but as far as I can tell, this has not been formalised
  anywhere in the paper (it is used later on, though).
- To what extent is the existence of the functions described by Theorem 2
  unique? On p. 5, in Section 4, the paper alludes to *not* using the
  functional form of the network directly because it does not seem to be
  unique. I would like this to be explained a in more details, as
  I found the justification of why the dual subdivision is used quite
  hard to follow.
- In Section 4, how many experiments of the sort were performed? I find
  this a highly instructive view so I would love to see more experiments
  of this sort. Do these claims hold over multiple repetitions and for
  (slightly) larger architectures as well?
  Please also see my comments on Figure 2 above.
- The claim that orientations are preserved should be formalised.
  I immediately understand the intuition behind this concept, but
  if possible, I would like a quantification of this. Might it be
  possible to *measure* changes in orientation with respect to an
  original polytope? If so, it should be possible to provide more
  experiments about these effects and summarise them accordingly.
  Maybe it would also be interesting to investigate whether other
  initialisations can be compared in terms of their orientations?
- In Section 5, I would give a brief link to the appendix for the
  definition of a Minkowski sum.
- Section 5 has (in contrast to the other sections) a lot of details
  containing the experimental setup, but it is missing a description of
  the competitor methods. Adding to what I wrote above, I feel that the
  paper should rather pick *one* area in which experiments are
  performed; the pruning (together with the lottery ticket hypothesis
  explanation, which could be seen as a motivating example) strikes me
  as a good candidate for this.
  I really like this concept of tropical pruning, by the way---it is an
  elegant, principled description!
- The plots in Figure 4 should summarise multiple pruning runs, if
  possible. Why not show a standard deviation as well? Given the
  stochasticity of training, I would think this highly necessary.
- In Section 6, I find the comment on normals generating a superset to
  the decision boundaries hard to understand.
-  The perspective of perturbing the network such that the decision
   boundaries change is really interesting, but I am missing
   a 'take-away message' or a discussion of the insights. Currently,
   this section seems more like a feasibility study: it appears to be
   possible to use the tropical description to find new parameters that
   misclassify a given input. I would propose a discussion of the
   implications of these findings.
- Concerning future extensions of this work, are there some promising
  results or directions for CNNs or GCNs? If so, it would strengthen
  the conclusion if they were mentioned.
4. Minor style issues
The paper is well-written. I found some minor style issues:
- 'piece-wise' --> 'piecewise' (occur multiple times)
- 'recently demonstrated' --> 'demonstrated'
- 'Thereafter, tropical hypersurfaces divide' --> 'Tropical hypersurfaces divide'
- 'If set $B$ --> 'Letting B'
- I am not sure if I would call adversarial attacks a 'nuisance'; maybe rather a 'problem'?
- Use '\operatorname' or '\mathrm' to typeset the loss in Eq. 5
5. Update after rebuttal
The authors addressed all my important comments; their efforts in rewriting and revising the paper in such a short time period are to be commended. I am very happy to raise my score.


Deep neural networks (DNNs) can be huge in size, requiring a considerable amount of energy and computational resources to operate, which limits their applications in numerous scenarios. It is thus of interest to compress DNNs while maintaining their performance levels. We here propose a probabilistic importance inference approach for pruning DNNs. Specifically, we test the significance of the relevance of a connection in a DNN to the DNN's outputs using a nonparemetric scoring test and keep only those significant ones. Experimental results show that the proposed approach achieves better lossless compression rates than existing techniques.

This paper introduces a nonparametric score test to estimate the importance of network connections on the final output of Deep Neural Networks. They derive this by modeling the log-transformed joint density of each connection and final output in a tensor product reproducing kernel Hilbert space. They finally derive an asymptotic distribution of the proposed test statistics which only depends on the eigenvalues of the kernel. This importance test is applied to ranking the importance of each connection in Multilayer Perceptron Networks, and Convolutional Networks and sparsifying the networks by removing the least important connections. The method is applied post-training to fully trained networks but evaluated by retraining the sparsified network from scratch. The method is demonstrated in experiments on compressing three networks.
I believe this paper is a borderline accept. It provides a more statistically principled method to examine the importance of connections in a Neural Network and rank them compared to existing compression methods. Due to this ranking, there is a clear method to easily adjust the target compression rate. They are able to achieve high lossless compression rates. However, the benefits shown in the empirical results could be more convincing. It lacks baselines on more complex networks and could benefit from more empirical analysis of the theoretical benefits and properties of this approach.
Pros:
The method is able to maintain accuracy while achieving high compression rates on MNIST and CIFAR10. It does better than the baselines compared to the small networks. They show a capability for increasing generalizability by decreasing error rates.
It provides a well derived statistically principled method to examine the importance of connections and rank them.
Indicates an additional use to visualize the importance of features.
Cons:
The empirical results could be clearer. It lacks baselines for larger models on Cifar10. Could you compare it with the published results of other algorithms? How does it do on larger networks like Imagenet?
Computational efficiency is mentioned but could be examined in greater detail.  How long does it take to run and how is that affected by model size?
The experimental setting is somewhat unclear. The baseline Louizos et al. (2017)  was designed to optimize group sparsity/speed, but the experimental results here only examine the compression rate. Was the baseline run to optimize speed or sparsity?
It would be interesting to examine the correlation of importance score with actual impact on network performance. This might be done with a comparison with random pruning or pruning higher importance connections. It might be useful to examine the performance of the networks after pruning nodes of differing importance without full retraining or just fine-tuning. It is unclear how important full retraining is in this method.
It would be interesting to visualize the importance of features at different depths in the deep convolutional networks.
Minor suggestions: In the introduction, you mention l0 and l1 norm methods, but cite Han et al. (2015) which compares l1 and l2 norm and found l2 norm to be better overall.
typos:
In Abstract: nonparemtric scoring test  -> nonparametric scoring test
In 4.4: sample averarge -> sample average
In 4.5: ASYMPTOTICALLY DISTRIBUTION  ->   ASYMPTOTIC DISTRIBUTION


A simple method for obtaining uncertainty estimates for Neural Network classifiers (e.g. for out-of-distribution detection) is to use an ensemble of independently trained networks and average the softmax outputs. While this method works, its results are still very far from human performance on standard data sets. We investigate how this method works and observe three fundamental limitations: "Unreasonable" extrapolation, "unreasonable" agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out-of-distribution inputs, but do not contribute to the classification. To mitigate these problems we suggest "large" initializations in the first layers and changing the activation function to sin(x) in the last hidden layer. We show that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).

I have read the reviews and the comments. Overall I am still positive about the paper and I have confirmed the rating.
======================
This paper proposes a method for the uncertainty estimates for Neural Network classifiers, specifically out-of-distribution detection. Previous methods use an ensemble of independently trained networks and average the softmax outputs. The authors investigate this method (ensembles of ReLU networks) and observe three fundamental limitations:
“Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classification (CONSTANT FUNCTIONS ON THE TRAINING MANIFOLD).
To mitigate these problems the authors proposed the following:
- Changing the activation function of the last hidden layer to the sin(x) function, and they claimed that this is going to guard against overgeneralization.
- Use larger than usual initialization to increase the chances of obtaining more diverse networks for an ensemble.
- They claimed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets.
The paper addresses an important problem, out of distribution detection, by proposing a Fourier network which is somewhere between a ReLU network (small initialization) and a nearest neighbor classifier (large initialization). The authors claimed that this leads to an out-of-distribution detection which is better than either of them.
The paper is well written and easy to follow. The authors did an interesting and precise investigation in how to force the confidence score to decay like a Gauss function by proposing to use the Fourier transform of such a Gauss function. By doing so they get the advantage of ReLU (ability to generalize) and prevent the network to become arbitrarily certain of its classification for all points. However, the authors claimed that when they switch the activation function to sin(x) the increase of |x| will usually stop at the first maximum or minimum of sin which is (around \pi/2). However, the authors did not explain how they get this result (i.e the value \pi/2). It would be interesting if the authors could show results for the case greater than or less than (\pi/2) to show the difference.
In addition, Figure 3 shows that the ensemble of ReLU networks is overconfident in most of the area, whereas the ensemble of Fourier networks is only confident close to the input, and in the discussion of constant function of their training manifold the authors discuss some example.
I would like to ask the authors:
Did the Fourier networks learn the input distribution?
How are defined: usual initialization, small initialization and large initialization?
The experiment section is adequate. However, it would strengthen the paper if the authors compared against other approaches such as:
- Predictive uncertainty estimation via prior networks, NeurIPS 2018.
- Generative probabilistic novelty detection with adversarial autoencoders, NeurIPS 2018.


Molecular graph generation is a fundamental problem for drug discovery and has been attracting growing attention. The problem is challenging since it requires not only generating chemically valid molecular structures but also optimizing their chemical properties in the meantime. Inspired by the recent progress in deep generative models, in this paper we propose a flow-based autoregressive model for graph generation called GraphAF. GraphAF combines the advantages of both autoregressive and flow-based approaches and enjoys: (1) high model flexibility for data density estimation; (2) efficient parallel computation for training; (3) an iterative sampling process, which allows leveraging chemical domain knowledge for valency checking. Experimental results show that GraphAF is able to generate 68% chemically valid molecules even without chemical knowledge rules and 100% valid molecules with chemical rules. The training process of GraphAF is two times faster than the existing state-of-the-art approach GCPN. After fine-tuning the model for goal-directed property optimization with reinforcement learning, GraphAF achieves state-of-the-art performance on both chemical property optimization and constrained property optimization.

This paper proposes a generative model architecture for molecular graph generation based on autoregressive flows. The main contribution of this paper is to combine existing techniques (auto-regressive BFS-ordered generation of graphs, normalizing flows, dequantization by Gaussian noise, fine-tuning based on reinforcement learning for molecular property optimization, and validity constrained sampling). Most of these techniques are well-established either for data generation with normalizing flows or for molecular graph generation and the novelty lies in the combination of these building blocks into a framework. Training can be carried out in parallel over the sequential generation process, as no hidden states with sequential dependency are assumed (unlike a regular RNN). Experimental validation is carried out on a standard ZINC molecule generation benchmark (graphs with up to 48 nodes) and the reported metrics are competitive with recent related work.
Overall, the paper is very well written, nicely structured and addresses an important problem. The framework in its entirety is novel, but the building blocks of the proposed framework are established in prior work and the idea of using normalizing flows for graph generation has been proposed in earlier work (see [1] and [2]). Nonetheless, I find the paper relevant for an ICLR audience and the quality of execution and presentation of the paper is good.
I have two major (technical) concerns with the flow-based formulation used in the paper with regards to order-invariance and the utilized de-quantization scheme.
* Order-invariance: The paper states that the “exact density of each molecule can be efficiently computed by the change-of-variables formula”. This seems to be incorrect, as the exact density is a product over all order-specific densities for all possible permutations in which the molecular graph can be represented. The change-of-variables formula does not provide an efficient way to circumvent this order-invariance issue, at least not in the way it is presented in the paper. Even when using BFS-ordered representations, the subspace of possible permutations is still typically too large to allow for efficient evaluation of the exact density. I suspect that the authors assume a canonical ordering of the graph representations, which is a strong assumption, but does not seem to be mentioned in the paper. How is the canonical ordering chosen? How is local structural symmetry broken in a consistent manner?
* De-quantization: The de-quantization scheme used in this paper seems to be ill-suited for categorical variables. What motivates the use of adding Gaussian noise to categorical (one-hot encoded) variables, other than that it seems to work OK in the reported experiments? Adding Gaussian noise in this way can move these variables outside of the probability simplex — is this a valid technique in the framework of normalizing flows? Adding Gaussian noise makes sense if the data represents quantized continuous data, e.g. bit-quantized image data, but I have concerns about the validity of using this method for categorical data (both edge type and node features are categorical in this application). Other comparable generative models for graph-structured data use a relaxed discrete distribution (concrete / Gumbel softmax), e.g. in MolGAN [De Cao & Kipf (2018)], to address this issue — would this also be applicable here?
I think that these two issues will have to be addressed before this paper can be considered for publication, and I recommend a weak reject at this point.
[1] Madhawa et al., GraphNVP: An invertible flow model for generating molecular graphs. (2019)
[2] Liu et al., Graph Normalizing Flows. (2019) — not cited
UPDATE:
My two main technical concerns have been addressed in the rebuttal and I think that the revised version of the paper can be accepted to ICLR (my comment w.r.t. novelty still holds and hence I recommend 'weak accept').


In this paper, we present an approach to learn recomposable motor primitives across large-scale and diverse manipulation demonstrations. Current approaches to decomposing demonstrations into primitives often assume manually defined primitives and bypass the difficulty of discovering these primitives. On the other hand, approaches in primitive discovery put restrictive assumptions on the complexity of a primitive, which limit applicability to narrow tasks. Our approach attempts to circumvent these challenges by jointly learning both the underlying motor primitives and recomposing these primitives to form the original demonstration. Through constraints on both the parsimony of primitive decomposition and the simplicity of a given primitive, we are able to learn a diverse set of motor primitives, as well as a coherent latent representation for these primitives. We demonstrate, both qualitatively and quantitatively, that our learned primitives capture semantically meaningful aspects of a demonstration. This allows us to compose these primitives in a hierarchical reinforcement learning setup to efficiently solve robotic manipulation tasks like reaching and pushing.

Paper Summary:
The paper proposes a method for learning a set of primitives for robotic movements from a dataset of demonstrations, showing a diverse set of tasks, in an unsupervised fashion. The central underlying idea is that robotic tasks can be solved by combining fundamental building blocks, the so-called "motor programs", in the right way. The described algorithm takes a demonstration and uses a transformer network to embed the trajectory into a sequence of latent variables. Then each individual latent is transformed to a 10 step trajectory for the joint space of the robot via an LSTM network. Finally the individual trajectories are concatenated and the reconstruction is compared to the original demonstration through dynamic time warping. In this structure the latent variables represent a query to a specific learned primitive, which can be accessed using the LSTM.
In the experimental section, the paper gives mostly qualitative insight into the learned representation. The paper visualizes different movements and their corresponding latent variables projected onto two dimensions. Further, it is shown that the segmentation of demonstrations roughly compares to how a human expert would manually segment the given tasks. Finally, the paper shows that a hierarchical RL algorithm trained in the learned latent space outperforms one which works directly in the low-level control space of the robot in the sense that it learns to solve the given task much faster.
Evaluation:
The problem of discovering primitives is approached by the authors in a novel and interesting way, however in my opinion the paper should be rejected because:
    (a) the experimental section is not convincing enough to support the claim that the method captures the shared motions across different skills. Especially, the paper misses to adequately show how these motions can be recombined and used to solve robotic tasks.
    (b) the paper is imprecise and missing important details in both the description of the method and the experimental verification
    (c) the paper misses important related work, which tackles the same problem.
The two main claims in the paper are:
1. The presented method learns a latent space which represents common shared motions among diverse tasks encountered in robotics
2. Robotic tasks can be solved by recombining primitives from the aformentioned space
Although it is impossible to verify the first claim based on the paper alone, the provided webpage, which shows an animated version of Figure 3 nicely visualizes the learned latent space and shows that the representation is somewhat smooth with similar movements clustered together.
a1) Figure 5 is supposed to show that the method manages to segment given demonstrations in a meaningful way, but even after zooming into the pdf, it is impossible to see what is actually going on. A visualization in a video would be preferable.
a2) 4.1.1 and Figure 4 show that individual primitives can be executed on a real robot, however, the paper fails to show the execution of a combination of primitives.
a3) Given that the method seems to loose the connection of movements to time it would be interesting to see whether a combination actually results in smooth, natural movement of the robot.
a4) The main quantitative assessment of the usefulness of the learned motor program network is given by the RL experiments in section 4.3. However, the baseline method seems to output one single velocity control action per evaluation of the policy (?), whereas the presented method essentially outputs an action sequence of 50 actions per policy evaluation. State-of-the-art methods commonly use frame-skipping and repeat the same action for multiple timesteps, because it makes the resulting optimization problem easier and speeds up the learning. See for example (Mnih 2013) (Mnih 2015) (Lillycrap 2015) (Hafner 2018). It would be interesting to compare against a baseline which also incorporates some form of frame-skipping and validate the speedup is not simply due to chunking of action sequences.
a5) Finally, the method outperforms the plain PPO baseline when it comes to speeding up the learning process, but the solutions found do not look like natural robot movements. You can clearly spot different primitives and transitions between individual segments look unnatural and jerky. How does the baseline solution compare in this regard?
a6) Given that the presented movements look unnatural and the fact that the paper only shows the execution of individual primitives in the rest of the paper, I simply cannot support the second claim.
Details I am missing from the paper:
b1) How is the "continuation probability" computed with the transformer network?
b2) Are the biases in section 3.2 necessary to make the method work at all?
b3) In section 4.2, how does the sequence alignment work?
b4) In section 4.2, what is the training set for the labelling task?
b5) What are the exact task parameters given to the policy in the RL task?
b6) What do you mean with, "these motor programs are executed without environment feedback"? Does the policy determine the complete sequence of programs in one step?
c1) Finally, to my knowledge the problem of learning meaningful primitives and showing that they can be combined in a different way to solve novel, unseen tasks has already been investigated in (Lioutikov 2017). Although this paper approaches the problem very differently and the dimensionality of the primitives is lower, I still consider this paper very much related to what is presented by the authors. I would suggest adding it to the related work. The paper shows that previous methods for discovering primitives from a set of different tasks exist.
In my opinion the paper in its current is probably not yet ready for publication. However, I strongly encourage the authors to address the above mentioned problems.
References:
Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013).
Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529.
Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).
Hafner, Danijar, et al. "Learning latent dynamics for planning from pixels." arXiv preprint arXiv:1811.04551 (2018).
Lioutikov, Rudolf, et al. "Learning movement primitive libraries through probabilistic segmentation." The International Journal of Robotics Research 36.8 (2017): 879-894.


Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter p, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

% post author response %
Thanks for your detailed response.
R1. Note that in almost all classical optimization routines, the learning rate has a (very intuitive) scaling on the problem parameters - for e.g. in gradient descent, the learning rate looks like 1/smoothness. This is mirrored in the definition of Newton methods - where, in the direction of hessian^{-1} grad, one uses a scale free line search to estimate an appropriate stepsize between 0 and 1 (to emphasize, this value has *no* dependence on problem scaling). While this begins to fall apart with the case of adaptive gradient methods, how can we even hope to justify the potentially arbitrary power of, say, 1/4 or 1/8 used by Padam? This is the reason behind my comment that the algorithm is unnatural. By using such a power of the smoothness of the problem, the other component of the learning rate (alpha_t) is no longer a scale free quantity. It has to depend on other problem dependent parameters for the overall learning rate to be scaled appropriately based on the problem characteristics.
R2. While the paper performs grid search (sec C.3 in the paper) for the partially adaptive parameter, the lambda value for YOGI is set as one suggested in their paper. I can accept the claim of authors if I see more experiments tuning the lambda parameter for YOGI as well.  Otherwise, I dont quite see why one specific parameter value for lambda works for every problem.
R3/R5. My point is that the original paper for adaptive methods (adam/adagrad) never mentions learning rate decay. This seems to have been added in subsequently just to boost the performance. What I do not understand is what is the specific advantage of adaptive methods over SGD if every component used by SGD (including learning rate decay) is used even by adaptive methods. Even from a theoretical bound perspective, to the best of my knowledge, there is no clear indication that the partially adaptive momentum methods actually improve over vanilla sgd (+momentum) in settings that do not involve dense parameters/gradients.
R4. Again, the learning rate decay has a specific use in the papers I referenced in my review. Somehow, the bounds presented in the paper do not reflect the use of a step decay schedule on the learning rates, so, I see that the theorem (aside from the assumptions mentioned) is detached from the practical results even in this respect.
%%
This paper considers generalization issues experienced by adaptive gradient methods compared to well-tuned SGD + momentum, a topic that is of interest in the development of optimization methods for deep learning. The paper is well-written and elaborates on (i) issues faced by adaptive gradient methods in contrast to standard SGD + momentum, (ii) presents experimental results on training standard conv-net based architectures on image classification benchmarks and on training LSTMs on PTB and (iii) presenting theoretical analysis relating convergence of the method to a first-order critical point for smooth stochastic non-convex optimization.
I have questions about certain aspects of the paper, which I will elaborate below:
— If one takes a step back to understand the origins of diagonal adaptation methods (introduced by Adagrad), this was motivated by the infeasibility of using the inverse square root of a full pre-conditioning adaptation matrix. If we think of such full matrix adaptation methods, is this paper implying the use of other matrix powers (other than a square root) as used by adagrad (or other adaptation approaches)? This appears very unnatural to me.
-- If the main issue is that it is not possible to typically use a large learning rate at the start with ADAM or other preconditioning methods and that leads to using other powers of the diagonal adaptation matrix, a more natural fix would be to use a conservative (trust region inspired) approach to reduce aggressive steps at the start. What I mean is as follows: Suppose H_t is a preconditioning matrix, g_t is the gradient (or the discounted sum of gradients with/without bias correction). The current approach is to make H_t diagonal and use H_t ^{-1/2}g_t as the update matrix. One option to prevent aggressive initial steps is to use (H_t + \lambda)^{-1/2} g_t, either with having lambda fixed through the optimization or making it a function of iterations.
— I am not aware that papers which employ adaptive methods also tend to use some form of learning rate decay (on the alpha_t ’s) - at least, by looking at the original papers of Adam/adagrad, I do not see the combination of learning rate decay and adaptive methods. In a sense, if one has an `"adaptive" optimization method, it’d be unnatural to have to use some form of step decay of the learning rates (alpha_t's) in conjunction with these methods. One would typically just use SGD+momentum with some form of such a step decay of the learning rates. This to me is a serious shortcoming - it appears to make the use of adaptive methods almost irrelevant because the only hyper-parameter that it gets rid of is the dependence on the initial learning rate (because, for SGD, we anyway have the other hyper-parameters like momentum, stepdecay factor, when to decay learning rate etc.)
— With regards to theory (and connections to experiments): Despite the fact that the paper employs a step decay schedule on the learning rates (alpha_t), their theorem statement (or any corollary) doesn’t actually employ this specific step size decay scheme (on the alpha_t ’s) and attempts to understand what are the advantages of the step decay schedule on the convergence statements provided.  The step decay schedule has featured in several recent efforts in the stochastic optimization community (both with convex (https://arxiv.org/pdf/1607.01027.pdf, https://arxiv.org/pdf/1904.12838.pdf)/non-convex (e.g. https://arxiv.org/pdf/1907.09547.pdf) objectives), where, the results indicate non-trivial advantages of using these step-decay schemes on alpha_t’s (though, these are with non-adaptive optimization methods).
— Along these lines (of the previous point), it is important to note what the performance of the optimization method is when alpha_t’s are fixed to a specific value (without being decayed over the course of optimization) - since this relates to the most standard definition (and advantage) associated with using adaptive gradient methods. My understanding is that this result continues to be fairly sub-optimal compared to using SGD+momentum with a step decay schedule.
Other minor comments:
— I do not understand the use of the term “second order” momentum for calling variables that have a running average of squared gradients. This term is misleading in what it represents.


Convolutional neural networks (CNNs) excel in image recognition and generation. Among many efforts to explain their effectiveness, experiments show that CNNs carry strong inductive biases that capture natural image priors. Do deep networks also have inductive biases for audio signals? In this paper, we empirically show that current network architectures for audio processing do not show strong evidence in capturing such priors. We propose Harmonic Convolution, an operation that helps deep networks distill priors in audio signals by explicitly utilizing the harmonic structure within. This is done by engineering the kernels to be supported by sets of harmonic series, instead of local neighborhoods for convolutional kernels. We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks. With Harmonic Convolution, they also achieve better generalization performance for sound source separation.

The paper considers the effectiveness of standard convolutional blocks for modelling learning tasks with audio signals. The effectiveness of a neural network architecture is assessed by evaluating its ability to map a random vector to a signal corrupted with an additive noise. Figure 1 illustrates this process with a network taking a single standard normal vector as input and having a single target output consisting of some signal corrupted with additive noise.
The paper is not well written and it is rather difficult to follow. It is also not well structured with a number of relevant concepts properly described only sections after they appear for the first time.
The first issue I had with the paper was the notion of audio prior. It was only after reading the whole paper that I have realized what this means. Having said this, it is unclear why the employed notion would work in general. I see why it could work when the distribution of the input vector and additive noise are correlated. This has not been clarified nor discussed and I believe it merits a couple of sentences.
In the introduction, the paper states "... unlike CNNs for image modelling, the design of deep neural networks for auditory signals has not yer converged". First, it is not clear what it means for the architecture to converge. If we assume that it refers to standard convolutions with a couple of widely accepted filter size and max pooling, the I would say that in speech recognition the structures that work are quite similar for mel-frequency coefficients or fbank features as inputs (which are again convolutional feature extraction layers).
Shortly after this, there is a question on justification of network designs. I disagree with a potential implication that this is well understood for image processing. For some insights relevant to speech, the work by Mallat ("Group invariant scattering", 2012) might be useful.
Figure 1 and the paragraph just below its caption are not clear. It is not explained what is the input/output of the network and this is of great importance for the understanding of the illustration in Figure 1.
The introduction does not explicitly define the notion of audio prior and the whole paper is about this. In my opinion, it is wrong to assume that a reader has seen the paper by Lempitsky et al (2018).
Section 2.1, the optimization objective as formulated implies that z and x_0 are completely independent. I do not see how any meaningful conclusion can be derived by fitting a map between independent input and output vectors. Some assumption is required for the proper notion of "audio prior" (if not, then a discussion arguing for the opposite).
Section 3, opening paragraph concludes that standard CNNs are not the best blocks to model learning tasks with audio signals. For this implication, one needs the exact structure of CNN network and more details with regard to the experiment itself. In particular, there are deep CNNs (with mel-frequency coefficients as inputs) that work very well in speech recognition (e.g., on noisy datasets such as aurora4). This illustration does not say anything about the influence of the depth and number of convolutional blocks on a learning task. The language should be more moderate here and, in general, some additional work is required on the motivation of harmonic convolutions.
In my understanding, harmonic convolutions are a special case of deformable convolutions (Dai et al., 2017). In essence, standard convolution is applied over time and deformable over the frequency axis of a spectrogram. The main contribution seems to be in that the work provides a structure to the offsets in Dai et al. (Section 2.1, 2017). If I am correct, then this should be discussed in details and the harmonic convolution needs to be placed in the context of prior work. It might help by starting with a review of that work and then introducing the imposed structure on the offset vectors. I am having problems understanding the illustration in Figure 3.
In the experiments, the work is evaluated on signal de-noising (audio restoration) and sound separation.
The first task is carried out under the assumption that the signal has been corrupted with Gaussian noise and shows advantages of the approach over baselines which include standard convolutional networks. It would be interesting here to see how the depth of a convolutional network affects the performance. Also, as the approach is (in my understanding) a special case of deformable convolutions it would be insightful to show an experiment with that baseline. While additive noise is difficult on its own, many signals are corrupted by channel noise. It would be interesting to add an experiment with different types of channel noise and which network design is more likely to de-convolve the noise from the signal.
The second experiment deals with separation of sounds of different musical instruments and the results again show advantages of harmonic convolutions over the baselines.


We introduce and study the problem of Online Continual Compression, where one attempts to learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. This problem is highly relevant for downstream online continual learning tasks, as well as standard learning methods under resource constrained data collection. To address this we propose a new architecture which Stacks Quantization Modules (SQM), consisting of a series of discrete autoencoders, each equipped with their own memory. Every added module is trained to reconstruct the latent space of the previous module using fewer bits, allowing the learned representation to become more compact as training progresses. This modularity has several advantages: 1) moderate compressions are quickly available early in training, which is crucial for remembering the early tasks, 2) as more data needs to be stored, earlier data becomes more compressed, freeing memory, 3) unlike previous methods, our approach does not require pretraining, even on challenging datasets. We show several potential applications of this method. We first replace the episodic memory used in Experience Replay with SQM, leading to significant gains on standard continual learning benchmarks using a fixed memory budget. We then apply our method to online compression of larger images like those from Imagenet, and show that it is also effective with other modalities, such as LiDAR data.

This work contributes to introducing a problem called Online Continual Compression. This problem requires to avoid catastrophic forgetting and learn in an online way. Generative methods should be one of the popular ways to do continual learning. This work’s model can be categorized into this clue since it also aims to save samples from old tasks by learning a generative model. In this way, the generator plays a similar role Experience Replay (ER) (here is called Generative Replay). The main core of this work should be the stacked quantization modules (SQM) which can be regarded as a hierarchical variant of the VQ-VAE model. In their SQM, hidden encodings z_q^i will be encoded and its input is z_q^{i-1} which is from previous layer.
This works covers related works very well. However, there are some questions I am really concerned:
1)	About the studied problem “Online Continual Compression”, what’s the difference between “online” and “continual”? In continual learning, tasks will be learned sequentially, right? If so, continual learning should run in an online learning way.
2)	The motivation of the hierarchy in this work is unclear. What I mean is that the hierarchical model should be expected to capture higher-level semantic features. But in this work, the index outputs z_q^{i-1} is encoded by its subsequent layer. It seems a bit weird since the z_q^{i-1} is not an image and its elements are index values. So what is the higher-level semantic information? By the way, it seems that there is an error in the model figure 1. The last MSE from Block 1 should be connected to the block before decoder 1 in Block 1, rather than the reconstructed one from decoder 1. Therefore, I strongly suggest authors give more insights and clarify the motivation of hierarchy. Writings in the METHODOLOGY part is unclear. More details about the SQM model should be described in a mathematical way.
3)	Another question about the details of generative replay. How do you do the replay? Details about this can’t be found in this work? In Alg.1, what is the \theta? Is the \theta_{ae} at line 14 of Alg.1 wrong? It should be \theta_{gen}, right?
4)	You use the data-stream technique reservoir sampling to add and update the memory buffer (alg. 4). Will it lead to some information loss? Can we just update memory without reservoir sampling? Please give more insights about this.
5)	How to find the distortion threshold d_th in Alg.2?
6)  the part of ablation studies is good. But I suggest authors should consider a baseline with the same proposed framework but using a single-layer VQVAE with the same memory capacity as the hierarchical models.