Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in the broad comparison, we introduce the deep ensemble equivalent (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of very few independently trained networks in terms of the test log-likelihood.

The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.
It seems that the paper provides the analysis missing in the current literature. Whereas as mentioned Yukun Ding in a public comment there is a related work on identifying issues with popular uncertainty metrics, the mentioned paper is missing the through comparison of the methods for estimating uncertainty.
Such kind of thorough analysis (especially performed on scale on large datasets) and comparison is of obvious interest to the community as well as objective comparison of the current state-of-the-art.
The paper is clearly written and easy to follow.
Based on this, I believe this is a strong technical paper and it should be accepted. However, the analysis in the paper is not overwhelmingly exhaustive. Some of the arguments on that are listed below.
Below is the list of comments/thoughts for potential improvement of the paper:
1.	“In this case, a model is expected to provide correct probability estimates:” – may be not the best choice of words, because for out of domain uncertainty estimation we still expect a model to provide correct probability estimates
2.	The first paragraph on page 3 seems to better fit in Section 2, for example, on the very beginning of Section 2.
3.	“Comparison of the log-likelihood should only be performed at the optimal temperature.” and others alike – personally, I do not support this kind of formatting for a scientific paper
4.	“can produce an arbitrary ranking of different methods. <…> Empirically,” – in the current form it seems that the first statement is somehow theoretically justified and then additionally it is confirmed empirically in this paper. I believe that the authors use empirical observation itself as the justification of the first statement, if that the case it should be reworded here. For example, “can produce an arbitrary ranking of different methods as we show below/ as we show empirically. We demonstrate that the overall …” If my belief is incorrect and there are other grounds that justify the first statement that it is required a reference after this statement.
5.	Italic and non-italic LL usage is unclear
6.	Having “Brier score” emphasised as a paragraph, it seems that there should be a paragraph log-likelihood as well
7.	“In that case, both objects and targets of induced binary classification problems remain fixed for all models” – do the authors consider in this case all out-of-domain objects as having a positive class and all in-domain objects as having a negative class? Because the models are still going to make individual misclassification mistakes
8.	Figure 2 – legend occupies too much space of the plot occluding almost a third part of the plot. Maybe taking the legend out of the plot to the right and squeezing the plot to make a room for the legend would be a better solution
9.	In eq. (4) and (5) subscript DE is not defined
10.	“SSE and cSGLD outperform all other techniques except deep ensembles” –cSGLD was not applied on ImageNet, therefore this statement is a bit misleading
11.	Colour of SWAG in Figure 3 is not very clear. Only excluding other colours I can determine which line is SWAG. Similar to cSGLD, is seems that SWAG was not applied on ImageNet. Why is that if that is the case? And it should be clearly stated at least in experimental setup in Supplementary
For colours in general, lines in legends are very thin and it is difficult to assess their colour. I appreciate the authors compare a lot of methods and therefore have to use a lot of colours, but it is quite difficult to assess them even on screen not to mention if the paper is printed out. Could the authors please use thicker lines in legends at least?
12.	“Being more “local” methods” – without any context in the main paper this referral to “local” methods is unclear. Also it is good to add a reference to Appendix review of the considered methods in the main text.
13.	Missing details of what kind of augmentation is used in Section 4.3. Is it the same as training augmentation specified in Supplementary? It would require a reference to Supplementary
14.	“(Figure 1, Table REF)” – missing number for Table
15.	“Our experiments demonstrate that ensembles may be severely miscalibrated by default while still providing superior predictive performance after calibration.” – unclear which experiments demonstrate this and superior in comparison to what
16.	The issues of uncalibrated log-likelihood and TACE are clearly shown in the paper, whereas the issues with misclassification detection are only verbally discussed. An illustrative example, at least a toy thought example, could really improve the paper here
17.	The chosen main performance metric is not very convincingly motivated. It is clear why it is based on calibrated log-likelihood, but it is not very convincing why one cannot just used calibrated log-likelihood as a performance metric, why one should base the metric on deep ensembles instead. Also from the long-term perspective, if the community comes up with methods clearly outperforming deep ensembles, the metric would need to be based on one of these new methods
18.	There is an indirect uncertainty metric that is not mentioned in the paper – uncertainty used in active learning (see, e.g., Hernández-Lobato and Adams, 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks)
19.	Figures 4 and 5 are too small
20.	“the original PyTorch implementations of SWA” – SWA is not considered in the paper
21.	“hidden inside an optimizer … The actual underlying optimization problem” – it seems that the ICLR audience should be familiar with “actual optimization problems” rather than using blindly the optimizer. It is always good to explicitly write down an equation that is used in a paper, but this wording seems a bit off for ICLR
22.	“\hat{p}(y^∗_i = j | x_i, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x_i” – it should be \hat{p}(y_i = j | x_i, w), y^*_i is observed
23.
a.	Why was dropout applied only for limited number of architectures and not applied on ImageNet at all?
b.	Why wasn’t cSGLD applied on ImageNet
24.	“On CIFAR-10/100 parameters from the original paper are reused” – it is better to repeat the reference here
Minor:
1.	Font size in eq. (10) should be the same as the rest of the paper
2.	“Or models achived top-1 error of”: “Or” - ?, “achived” -> achieved
3.	“for a 45 epoch form a per-trained model”: “form” -> “from”?


Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In this paper, we propose a novel policy optimization framework using an uncertainty-aware objective function to handle those issues. In this framework, the agent simultaneously learns an uncertaintyaware dynamics model and optimizes the policy according to these learned models. Under this framework, the objective function can represented end-to-end as a single computational graph, which allows seamless policy gradient computation via backpropagation through the models. In addition to being theoretically sound, our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines.

# Introduction
The motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.”
There has been work on learning a parametric policy from MPC. Therefore, you can extract a parametric policy from the optimization that MPC performs. The statement “Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy” is not true. The MPC will transfer better to other tasks that have the same dynamics, since it is not task specific. Given the learned dynamics model and the reward function you can act optimally in any new task as long as the learned dynamics are valid.
# Related work
My main concern with the related work section is that there a lot of literature on risk sensitive and optimism in the face of uncertainty (which is a subset of your method when c>0) in control, bandits, and some on *reinforcement learning* that has been neglected.
# Uncertainty-Aware Model-Based Policy Optimization
As said before, risk-sensitive in reinforcement learning has been done before and there’s even more work on control and bandits. For instance in [1] (page 5, paragraph (b)) has the same equation and they discuss the effect of the constant being negative or positive. More recent work has also used similar formulations [2].
This section mostly contains previous work, e.g., bootstrap rollout, policy gradient, using a deterministic policy ([3, 4, 5]). One thing that it’s still not clear from reading the paper, are you backpropagating the through the dynamics model, are you using a policy gradient method (REINFORCE, TRPO, PPO,… )?
# Algorithm Summary
One of the novelties introduced is the fact that the data used in each model comes from sampling a Poisson variable. However, this is not ablated in the results sections. Is it necessary? [6] Claims that there’s no need to use different data for the learned models.
# Experiment
The experiment section lacks from more complex environments, in this case the most complex is half-cheetah. Furthermore, given that 3 of the tasks are short horizon tasks you should probably also compare against model-based methods that build on top of policy gradients (e.g., [6]).
It seems that some choices in the algorithm are not ablated: 1) use of poisson, 2) use of deterministic vs stochastic policy, 3) Is there a single risk that works across environments? Which environments are risk prone/adverse? 4) How about having c ~ N(0, 1), effectively modelling V as a gaussian?
-----------------------------------
Overall, the paper is not mature enough to be accepted: there is not enough novelty, and the results lack of novelty, enough delta in performance from prior work, and have high variance.
------------------------------------
Minor/Typos:
First paragraph: “trying model the transition”
What does it mean that the accuracy is not satisfied?
Why the related work on deep model-based reinforcement learning is called Deep Neural Networks?
3.2 third paragraph: “Next we provide a convergence convergence …”
3.3.2, first paragraph: “no matter how uncertain it may know about the world”
Why the axis in the results section mean different things?
[1] Risk-sensitive Reinforcement Learning. Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer.
[2] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch
[3] Continuous control with deep reinforcement learning. Timothy P. Lillicrap et. al.
[4] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine
[5] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee.
[6] Model-Ensemble Trust-Region Policy Optimization. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel.


While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, there are issues that remain to be tackled in order to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation-and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.

Summary: The paper addresses continual learning challenges such as catastrophic forgetting and task-order robustness by introducing a new hybrid algorithm that uses architecture growth as well as parameter regularization where parameters of each layer are decomposed into task-specific and task-private parameters. They also use a simple trick to The authors perform experiments on Split CIFAR100, CIFAR100 Superclass, Omniglot, and a sequence of 3 datasets (SVHN,CIFAR10,CIFAR100). The maximum number of tasks in the experiments is 100 for Omniglot-rotation. The authors show superior performance to EWC (a regularization-based method), P&C (architecture-based method), DEN (architecture-based method), PGN (architecture-based method), RCL (architecture-based method), etc.
Pros:
+ The paper is well-written and has motivated the problem of scalability and forgetting
+ Proposing a new hybrid approach that benefits from the best of both worlds (maximum usage of the capacity with parameter regularization followed by logarithmic architecture growth at arrival of new task using layer-wise parameter decomposition.
Cons that significantly affected my score and resulted in rejecting the paper are as follows:

1- Lack of measuring forgetting:
Authors indicate in the abstract that “a continual learning model should effectively handle catastrophic forgetting” and reiterate on this on other parts of the paper yet there is no table/figure that shows the initial performance of the model on each task so that readers can compare it with the reported accuracy after being done with all tasks. Having a method with high average accuracy does not necessarily mean it has minimum forgetting. You can use forgetting measurements such as BackWard Transfer, introduced in [1] or forgetting ratio defined in [2] for this assessment. A continual learning paper without proper measurement of forgetting is incomplete.
2 - Large-scale experiment is not convincing:
Authors believe scalability has not been addressed well in the literature (page 1&2) and claim it as one of their main contributions and making it crucial to support this claim. However, the experimental setting chosen for this claim is not convincing. Authors have chosen Omniglot-rotation as their longest sequence of tasks with 100 tasks where each task has 12 classes and in each class, there exists 80 images. This will make the total dataset of size 96K images which is still far from being large-scale. While I am aware of the fact that in the current CL literature, the maximum task sequence’s length is only 20 (Split CIFAR100) and I  agree that having an order of magnitude increase in the # of tasks is beneficial,  however, Omniglot is still a toy benchmark and does not serve as a large-scale dataset by only extending it to different random rotations. Moreover, the architecture used for this experiment is LeNet which oversimplifies the problem to address. For incremental learning, I would personally think of ImageNet as a good example and for continual learning of multiple datasets you can consider the existing sequence of 8 tasks benchmarked in [2] and [3] where you can evaluate your method on more realistic images with a total of over 400K images and significant shift in the distributions. As a side note, the key idea behind the proposed method is that this method is able to decompose the parameters into task-specific and task-private whereas in the Omniglot experiment it is not intuitive that what is there between the random rotations that is shared among the task. A more detailed discussion on this would be enlightening.
3 - No standard deviations shown in the results:
Although the results are said to be average over 3 runs, no STD is reported. Given that in the most important experiment of this paper (Omniglot) the difference between Accuracy obtained by PGN and APD is not significant (79.35% vs 81.6%).
In the current CL literature, robustness to the order of the tasks is shown by performing multiple permutations of the tasks and reporting average and STD. It is needed that authors show results for this for a fair comparison.
4 - Lack of regularization-based baselines:
Considering the fact that the proposed method is a hybrid approach, it is reasonable to compare against both architecture-based and regularization-based approaches. However, most of the baselines are chosen from the former category and EWC is the only baseline for the latter category which is relatively old and has been outperformed by large margins in the past couple of years such as SI [4], VCL[5], HAT [2], PackNet [6], MASS [7], and UCB [3].
Less major (only to help, and not necessarily part of my decision assessment):
Please consider explaining connection to prior work (HAT): While the literature review seems comprehensive, authors have missed one important previous work from ICML 2018 [2] called “Overcoming Catastrophic Forgetting with Hard Attention to the Task” or HAT. Both HAT and APD use an attention mechanism to alleviate forgetting. Considering HAT is a very strong baseline, I highly recommend authors provide a comparison with it. It’s an efficient and relatively scalable method that has very small BWT.
I recommend authors provide their method’s ability for zero-shot transfer or so called forward transfer metric to further support their method.
Hyper parameter tuning: It is also worth mentioning how the tuning process was performed. In continual learning we cannot assume that we have access to all tasks' data, hence authors might want to shed some light on this.
Minor point: On page 8, last paragraph, the authors state that a masked-based pruning technique (Piggyback) is immune to forgetting which is not an accurate statement (Note that PGN is indeed zero-forgetting by definition). All masked-based methods lose some of their performance prior to pruning. While it is correct to say that their post-pruning performance is 100% recoverable by saving the mask, forgetting should be measured with respect to their performance prior to pruning because that is their trade-off to give up accuracy in lieu of freeing space for future tasks.
References:
[1] Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in Neural Information Processing Systems. 2017.
[2] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557
[3] Ebrahimi, Sayna, et al. "Uncertainty-guided Continual Learning with Bayesian Neural Networks." arXiv preprint arXiv:1906.02425 (2019).
[4] Zenke, Friedemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[5] Nguyen, Cuong V., et al. "Variational continual learning." arXiv preprint arXiv:1710.10628 (2017).
[6] Mallya, Arun, and Svetlana Lazebnik. "Packnet: Adding multiple tasks to a single network by iterative pruning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[7] Aljundi, Rahaf, et al. "Memory aware synapses: Learning what (not) to forget." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Post-rebuttal response:
Thank you for taking the time to go through comments and providing your responses.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' response:] In Table 2 (paragraph “Continual learning with heterogeneous datasets”), we have experimental results with heterogeneous datasets, where continual learning models are evaluated on a sequence of different datasets (CIFAR-100, CIFAR-10, SVHN). We agree that experiments with massive datasets will be helpful, but we do not have sufficient time to perform all the experiments during the short rebuttal period. Hence, we only compared against HAT in our revision (Please see Table A.7 and Figure A.10) on the sequence of 8 tasks [2][3] you mentioned. We directly followed all experimental settings on the paper and the code of the authors (https://github.com/joansj/hat). APD (82.42 +- 0.5 %) outperms HAT (80.36 +- 1.2 %) in terms of accuracy. Although HAT shows a marginal forgetting (0.14 %) during training, the models is task-order sensitive (AOPD: 7.95%, MOPD: 23.15%) on difficult sequences of tasks as CIFAR10, CIFAR100, and FaceScrub (Please see Figure A.10) while APD consistently shows a reliable performance with lower OPDs (AOPD: 2.09%, MOPD: 4.40%) regardless of the task order. We will add in all baselines and APD variants in the final version of the paper for this 8-dataset experiment, if it gets accepted.
[Reviewer's response:] while I thank the authors in providing this comparison, I would not call this "significant outperforming". According to Table A.7 HAT method achieves  80.36% using the memory needed to store one single network in the memory on which they learn attention masks without using extra memory while APD uses 81% more memory only to achieve 82.42% average accuracy (2.13% increase) which is clearly not a fair comparison to me. Authors should either use the same memory for HAT (using a larger network architecture) or use a smaller memory size for APD and re-evaluate this comparison otherwise it is not conclusive which method is superior. Given the large difference in memory usage I suspect HAT will outperform ADP if given more capacity. While I agree with authors that regularization based approaches can be limited by the number of tasks, in the experimental setting used in this paper, this is not proven to be the case as these methods have not reached their maximum capacity and in fact are still performing strongly well compared to a hybrid approach which is presumably supposed to be better. I appreciate the novelty of the idea of decomposing parameters but it is not clear whether this factorization is actually performed given the high capacity needed to learn these tasks. Therefore, the results are still not convincing to me.
In addition,  as also brought up by R1, "My biggest concern is the choice of baselines. The paper (rightly)  highlights that their work improves over many existing works as they provide a mechanism to "retroactively update task-adaptive parameters" for the previous task. But none of their baselines have this mechanism built-in. So while there is a clear advantage with the proposed approach, the comparison is unfair and the baselines should have considered approaches like GEM [0[ and A-GEM[1] while also provide a kind-of retrospective mechanism to correct the weights corresponding to the subsequent tasks. Without such a comparison, it is difficult to comment on the benefits of the approach." --> regarding the comparison between APD and HAT in the order-robustness, this also seems as a big concern to me. Maybe I am missing it in the long list of comments and replies, but I am not able to find a clear response from authors in providing a fair assessment without their order robustness constraint mechanism (Eq. 2). This is an auxiliary advantage that only APD is benefitting from and makes the comparison difficult.  It should be either given to all methods or none.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' response:] This is a critical misunderstanding as those results are already provided in the paper.  We already clearly report the performance evolution of several tasks during the course of training in Figure 5 (a)-(c). Figure 5 (a)-(c) clearly show that APD does not suffer from catastrophic forgetting and even improves performance on previously trained task during continual learning (Figure 5 (b)), which is an effect of update on the task-shared parameters. Please see Page 7, “Preventing catastrophic forgetting” paragraph for more detailed discussions.
[Reviewer's response:] I disagree with authors' response regarding Figure 5 being a complete forgetting measurement supported by this sentence in “Preventing catastrophic forgetting” paragraph: "APD-Nets do not show any sign of catastrophic forgetting" and showing the performance of only 3 tasks out of 20 in which accuracies are barely readable due to the coarse scale of the figure and more importantly authors do not provide other methods' performance on these tasks. However, I thank authors for providing the comparison with stronger baselines and proper forgetting measurement in A.6 and A.7, I highly recommend swapping Figure 5 with your newly obtained quantitative forgetting measurements shown in A.6 and A.7 in the appendix
 (once fairly compared according to my comment above) as they provide a better support for forgetting avoidance.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' response:] The only hyperparameters required for APD are \lambda_1 and \lambda_2, which controls the capacity of the task-adaptive parameters and model drift respectively.  Since there is a trade-off between efficiency (network capacity) and accuracy, the users only need to tune them according to their priority. The model is not sensitive to hyperparameter configurations unless they are in the correct scale, and the details of the hyperparameter configurations are given in A.1.
[Reviewer's response:] Hyper-parameters can be a lot more than just \lambda_1 and \lambda_2 in your experiments. Batch size, optimizer's learning rate, weight decay, validation set size, etc are the parameters that are usually left out  in the CL settings and are not explained how they were tuned. As I said before, it is worth explaining what this sentence from A.1 means "All hyperparameters are determined from a validation set." so if this validation is composed of the data from all tasks or they followed some procedure like A-GEM paper in which it is assumed data of only 3 tasks is available in the beginning for tuning purposes.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' response:]  What you mentioned about mask-based methods may be true in general, but not in the case of Piggyback. Piggyback is composed of frozen backbone and task-specific masks, such that the backbone network is fixed without any updates during training, and the model only learns the task-specific pruning masks, which are *stored*, such that we can recover the performance on any previous tasks at any future points. Thus Piggyback does not perform actual pruning, and thus there is no loss of accuracy and forgetting on the previous tasks.
[Reviewer's response:] I disagree with your statement about "no loss of accuracy". Restating from Piggyback paper on its page 4: "The key idea behind our method is to learn to selectively mask the fixed weights of a base network, so as to improve performance on a new task. We achieve this by maintaining a set of real-valued weights that are passed through a deterministic thresholding function to obtain binary masks, that are then applied to existing weights. By updating the real-valued weights through backpropagation, we hope to learn binary masks appropriate for the task at hand."
This simply means they learn binary mask per task by using a thresholding function and save this mask. However, if you evaluate the model on a given task **prior to making** you obtain a different performance compared to evaluating after masking (this is what you save) where the former is usually higher or sometimes similar to the former because prior to masking, there exist more parameters while by masking some parameters will be 'freed' to be used for future tasks. This difference is what I am referring to as true forgetting and is zero only if evaluation prior and post masking are exactly the same because by storing the learned masks you can only recover the post masking performance.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' response:] The scalability problem we aim to tackle in our work is the scalability to number of tasks. This is because scalability to the size of the network or the number of data instances is basically the problem with generic machine learning and are not the main problem associated with continual learning. Since continual learning models learns on a sequence of tasks, we were more interested in how the existing (expansion-based) continual learning methods and ours behave on large number of tasks, in terms of catastrophic forgetting and network capacity. However, we agree that the term ‘large-scale continual learning’ may be misleading and have renamed the paragraph to ‘scalability to large number of tasks’ in the revision.
[Reviewer's response:] I disagree with the first sentence that dataset size is not the main problem associated with CL. It is an important factor that should be considered because dataset size and number/diversity of classes can significantly increase forgetting on early tasks as the distribution shift between the tasks will be significant. I understand the intention of authors and their interest in modeling large sequence of tasks, however introducing the method as scalable is misleading and in addition to the text which is corrected now, should be also corrected in the title of the paper.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I accept the response for the remaining questions from authors but intend to keep my score. However, I will be happy to update it based on the authors' response to the very first comment above regarding providing a fair comparison in memory size and order-robustness mechanism.


Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD). In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice. Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost. Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy at = 8/255 in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at = 2/255 in 12 hours, in comparison to past work based on "free" adversarial training which took 10 and 50 hours to reach the same respective thresholds.

This paper revisits Random+FGSM method to train robust models against strong PGD evasion attacks. Coupled together with tricks for accelerating natural training, such as cyclic learning rate, mixed precision, the robust models can be trained faster than previous methods.
+The experimental results are impressive. The trained model is robust (at Madry’s PGD level), and the total training procedure is fast (6 min for CIFAR-10 and 12 hr for ImageNet).
+ The method is simple, and I guess reproducible.
+The paper shows surprising facts of a well-known method.
+The paper is generally well-written and easy to follow.
I do have some concerns of the work
- The paper is empirical and the techniques are combinations of previous methods. Even for the surprising fact that Random+FGSM, it has been discussed in several previous papers, for example,  ICLR 2019 Defensive Quantization: When Efficiency Meets Robustness  https://openreview.net/forum?id=ryetZ20ctX. So the main contribution of the paper is limited to show RFGSM works well when combined with optimization tricks like cyclic learning rate.
-In previous methods claiming random+FGSM can train robust model, their method seems to be slightly different from Alg 3 in page 4 of this paper. The alg in this paper seems to be identical to Madry’s implementation of R-FGSM, which is shown not robust to PGD attacks. See discussions in https://openreview.net/forum?id=rJzIBfZAb and https://openreview.net/forum?id=ryetZ20ctX. I would like the authors to clarify their method to resolve such conflicts and make it clear how R-FGSM can be as robust as PGD as in table 1.
-The first two paragraphs of section 4.1 seem to be inaccurate. One important trick in the “adversarial training for free” paper is to replay each minibatch m times. It is hard to say how much nonzero initialization helps. According to “universal adversarial training” (https://arxiv.org/pdf/1811.11304.pdf). It may help, but cannot compete with Madry’s PGD training when defending against PGD attacks.
-I am not sure if using a larger norm 1.25 * \epsilon is a fair comparison. A baseline of PGD training bounded by 1.25 * \epsilon would help.
-Could the authors combine table 4 and 5 for easy comparison of robust accuracy and training time? Did the authors try the optimization tricks on ImageNet for the baseline free adversarial training method?
================== after rebuttal =================
I change my rating to weak accept. I tend to accept for the following reason
(1) there seems to be no obvious flaw in the authors implementation. I quickly skimmed their code, and looks like a few researchers have tried their code and responded in public discussion. The surprising robustness of RFGSM, though the originality is questionable and the technical difference comparing to previous methods are subtle, seems to hold true.
(2) The authors work hard to address the comments.
I still have some concerns, mainly regarding the fairness of experimental comparison.
(1) As pointed out in public discussion, the success of the proposed RFGSM relies on early stopping. I am wondering if early stopping also helps other methods since it turns out to be some sort of selection procedure.
(2) The authors did not update time in table 1 for CIFAR-10 results, which I consider almost no extra efforts. I am wondering how much more time each method needs from 45% in figure 2 to higher robust accuracy in table 1.
(3) I cannot understand why the proposed method is a particular good fit with cyclic LR and low precision tricks comparing to other methods.


In this paper, we introduce Random Path Generative Adversarial Network (RP-GAN) -an alternative design of GANs that can serve as a tool for generative model analysis. While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network. As we show, this design allows to understand factors of variation, captured by different generator layers, providing their natural interpretability. With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about the roles that different layers play in the image generation process. Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data.

This paper addresses the issue of interpretability of GAN generation through an alternative approach to the introduction of variability. To seed the generation, instead of providing a random input vector (typically sampled from a standard Gaussian distribution), the authors instead modify the generator architecture so as to allow for randomization in the routing: each layer is replaced by a bucket consisting of several blocks, and in forward propagation only through randomly chosen blocks. In this case, the input vector is chosen to be a constant - the only source of randomization
provided to the generator is in the choice of blocks through which to propagate. The explanability derives from the tendency of blocks to associate with a common interpretation after training. Their use of blocks necessitates the introduction of a block diversity loss, to discourage mode collapse. The scheme is referred to as RPGAN, for "Random Path GAN".
The main strengths of the paper:
(1) Their proposed approach is highly flexible. In principle, any underlying GAN architecture can be adapted by assigning each layer to a distinct bucket, and then replicating the layer across the blocks.
(2) Experimental results do show that different block sequences are associated with common image characteristics after training, especially for the initial and final layers.
(3) The use of non-standard ways of introducing stochasticity to GAN generation is an interesting idea in itself.
The main weaknesses:
(1) Although the authors provide experimental examples showing images associated with various paths through the architecture, is not clear how interpretations can be associated with these paths. In the examples presented, there seems to be a tendency for greater interpretability at the initial and final layers, with the explanation given for the intermediate layers being less convincing.
(2) The number of experimental examples is low, yet the authors draw rather firm conclusions (end of Section 4.1) regarding interpretability across layers. I am not sure that their conclusions adequately capture what is going on here, nor am I convinced that they generalize to other situations.
(3) The number of buckets limits the numbers of explanations. Essentially, the method has the same difficulties as in clustering, where specifying too many or too few groups can profoundly influence the nature and quality of the result. Although the authors do discuss an approach by which the number of buckets can be incrementally increased (thereby allowing for variation in the number of explanations generated), the experimental evidence is insufficient.
(4) Presumably, the replication of a layer across the blocks assigned to its bucket would require more training data and/or greater training times. What is the relationship in both time and quality between the original GAN network and its RPGAN versions?
(5) There are many presentational problems with this paper, in grammar, vocabulary and terminology, sentence structure, etc.
Overall, in its current state (not least due to presentational issues) the paper appears to be below the acceptance threshold.


Single Image Super Resolution (SISR) is the task of producing a high resolution (HR) image from a given low-resolution (LR) image. It is a well researched problem with extensive commercial applications like digital camera, video compression, medical imaging, etc. Most recent super resolution works focus on the feature learning architecture, like Chao Dong (2016); Wang et al. (2018b); Ledig et al. (2017). However, these works suffer from the following challenges: (1) The low-resolution (LR) training images are artificially synthesized using HR images with bicubic downsampling, which have much more information than real demosaic-upscaled images. The mismatch between training and realistic mobile data heavily blocks the effect on practical SR problem. (2) These methods cannot effectively handle the blind distortions during super resolution in practical applications. In this work, an end-to-end novel framework, including high-to-low network and low-to-high network, is proposed to solve the above problems with dual Generative Adversarial Networks (GAN). First, the above mismatch problems are well explored with the high-to-low network, where clear high-resolution image and the corresponding realistic low-resolution image pairs can be generated. With high-to-low network, a large-scale General Mobile Super Resolution Dataset, GMSR, is proposed, which can be utilized for training or as a benchmark for super resolution methods. Second, an effective low-to-high network (super resolution network) is proposed in the framework. Benefiting from the GMSR dataset and novel training strategies, the proposed super resolution model can effectively handle detail recovery and denoising at the same time.

In the paper, the authors proposed an end-to-end single image super-resolution framework, which is composed of two parts. First, a high-to-low network is trained to generate realistic HR/LR image pairs for training super-resolution models. Using this network, the authors designed a large-scale General Mobile Super Resolution Dataset, GMSR, which can be utilized for training or as a benchmark for super-resolution methods. Second, a low-to-high network is trained to produce a super-resolution image. Beneﬁting from the GMSR dataset and novel training strategies, the proposed low-to-high network could thus deal with the real distortions and restore ﬁne details.
The paper is easy to follow. The proposed framework is logical and the experiments proved to be effective. However, there are some problems as follows:
1. In the high-to-low network of framework, Gaussian random noise is added to clean high-resolution images to simulate the randomness of the distortions in LR. Why can do this in this way? The rationality of this setting should be given. Are there other kinds of noises be applied? As far as I knew, the in-camera processing pipeline will involve  Poisson-Gaussian noise rather than Gaussian only and other factors such as demosaicing, Gamma correction, and JPEG compression. It is hard to only consider Gaussian noise to stimulate the whole in-camera process.
2. The experimental part is remarkable insufficient to an ICLR paper. There only have a quantitative evaluation of the proposed framework with other super-resolution algorithms, while lacking the qualitative evaluation of the framework itself.
3. There are some formatting issues in the paper,
   1. Page 8, figure 5, here some errors in the label color of the figure.
   2. Page 1, introduction, line 14, "However, In most practical..." -> "However, in most practical...";
   3. Page 2, line 10, "...many problems, such as..., which is hard to avoid" -> "...many problems, such as..., which are hard to avoid";
   4. The last paragraph of the introduction, line 6, "...parameters.Third,..." -> "...parameters. Third,..."
      ...


To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data-of representative samples, of outliers, of misclassifications-is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-provided labels. However, manual data inspection is problematic for privacy-sensitive datasets, such as those representing the behavior of realworld individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models-trained using federated methods and with formal differential privacy guarantees-can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.

Goals
The paper identifies a key challenge in a large class of real world federate learning problems where we also have to ensure user level data privacy. In these settings the modeler can not inspect the raw data samples from the user (due to privacy concerns) and hence all modeling tasks (from data wrangling to hypothesis generation to labeling to model class selection to validation) become far more challenging. The paper proposes that in these circumstances one may use a generative model that learns the data distribution using federated learning methods with provable differentiable privacy guarantees. The generative model can then produce data (unconditional, or conditional on some features or class labels) which can be inspected by the modeler without compromising user privacy.
Experiments
The authors illustrate the approach using existing federated DP RNN learning methods, and using a slightly novel GAN learning algorithm for images (largely similar to other algorithms). They use these methods to provide two examples: 1) learning a language model from text (word sequences) where there is a bug in pre-processing steps (tokenization); 2) learning a GAN for images of handwriting on checks where there is a pre-processing bug that inverts the grayscale of images. These examples illustrate the potential for such methods to possibly be useful to modelers. While one may quibble about some details (see section below) the experimental set up is reasonable to illustrate the need and some of the challenges modelers are likely to face in the real world (
Evaluation & Questions
I'm really torn because I really enjoyed the paper very much overall but I have some strong concerns as well.
Positives: the paper is well motivated and very well written (it is really a pleasure to read, and it is very clear about the details -- especially after they release the code it should be possible to reproduce the results too). The authors shine a spot light on a problem that is very important & widespread (eg while learning from condifential data on cell phones). The proposed solution is fairly simple, intuitive, and quite high level (lets use a generative model that creates phantom data that can be inspected)
Negatives: I am not entirely sold on this being a realistic approach in the long term -- ie that some of the key problems will ever be solvable (I'm quite ok even if they are not solved now in the first paperr). The authors do a very good job of being transparent about several potential issues (see eg last paragraphs of main paper and appendix D). My biggest concerns are below:
* the phantom samples generated from the model need to be very realistic in order to be useful. In other words, we need to have excellent, high fidelity generate models. Even to create proper hypothesis, create proper model classes, assess convergence, or assess whether the generative model is good enough one needs to be able to inspect the raw data -- which can not be done in the first place. This can not be entirely automated eliminating need for human inspection -- and the problem is much worse in generative models (which need to encode more information) than in discriminative models which need to encode less information (bits) almost by definition. Thus one has simply traded the problem of needing to inspect data to model the final algorithm (whcih could be discriminative) and has to deal with the problem of needing data to inspect the intermediate, generative model (which is also learned using federates, DP guaranteeing ways). It is not at all clear what one accomplished by doing this.
* GANs are notoriously hard to train with mode collapse etc.  Setting hyper parameters of any generative model also needs access to original data and impacts the privacy guarantees.
***NOTE added after author response***
The rebuttal has sufficiently address the quibbles I raised below. I'm leaving it here to allow traceability. I'm not fully convinced about the response to the main issue I raised above (ie if the generative model is not very representative, high-fidelity, then one cant know whether a potential bug discerned by inspecting its samples is an artefact of the generative model or whether it is truly a fundamental bug upstream -- and training a high quality generative model also requires one to inspect the raw data in the first place so the problem has simply been swept under the carpet). Nevertheless for a first paper on the topic I think the contributions and intuition provided here are quite valuable so I am ok leaving this for for future work.
* quibble#1: theoretical DP bounds are not very tight. For example, in table 2 they may want to use realistic estimates instead of epsilon even to prove their high level point. I'm not sure I can buy their argument even on this illustrative problem as it stands.
* quibble#2: You may want to at least make an effort to compare against the nearest possible methods in your experimental setup even if they are not a great match to the problem. I'm not intimately familiar with the recent literature but you mention Triascyn & Faltings (2019) so perhaps you could also use that and expand a bit more on the novelty here


Many machine learning tasks involve analysis of set valued inputs, and thus the learned functions are expected to be permutation invariant. Recent works (e.g., Deep Sets) have sought to characterize the neural architectures which result in permutation invariance. These typically correspond to applying the same pointwise function to all set components, followed by sum aggregation. Here we take a different approach to such architectures and focus on recursive architectures such as RNNs, which are not permutation invariant in general, but can implement permutation invariant functions in a very compact manner. We first show that commutativity and associativity of the state transition function result in permutation invariance. Next, we derive a regularizer that minimizes the degree of non-commutativity in the transitions. Finally, we demonstrate that the resulting method outperforms other methods for learning permutation invariant models, due to its use of recursive computation.

The rebuttal did not address my concerns convincingly. There were also simple fixes that the authors could have implemented but they decided not to update the paper. I will keep my original assessment.
--------------
The premise of the work is very interesting: RNNs that are permutation-invariant. Unfortunately, the paper seems rushed and needs a better justification for not having a RNN memory that is associative. It also should cast the contributions in light of other existing work (not cited). The paper says "In this section and the remainder of the paper, we focus on the latter [commutative RNN memory operator], namely introducing a constraint (or equivalently, regularizer) that is commutative", but it never talks about the impact of a RNN memory using a non-associative operator. Being commutative is easy, isn't Equation (2.4) commutative if \Theta = W? Being associative is hard, since non-linear activations are not easily amenable to associativity.
Section 4: "The above example demonstrates that RNNs can in some cases be a natural computational model for permutation invariant functions." => Janossy pooling (Murphy et al., 2019) gives an alternative way to use RNNs, with a way to make their method tractable. Actually, my guess to why the RNNs experiments work well, even without an associative memory, is because the training examples come in multiple permuted forms, which is the data-augmentation version of the pi-SGD optimization described in Janossy pooling.
On page 1, "consider the problem of computing the permutation invariant function f(x_1, . . . , x_n) = max_i x_i", what follows is not a proof of necessity. It is an informal argument that either should be made formal or should be described as informal.
There is a lot of missing related work for sets:
Murphy, Ryan L., Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. "Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs." ICLR 2019.
Wagstaff, Edward, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. "On the limitations of representing functions on sets." ICML 2019.
Lee, Juho, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks." ICML 2019.
Also missing related work for graphs:
Bloem-Reddy, Benjamin, and Yee Whye Teh. "Probabilistic symmetry and invariant neural networks." arXiv:1901.06082 (2019).
Murphy, Ryan L., Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. "Relational Pooling for Graph Representations." ICML 2019.
The paper has an interesting question but needs to build on prior work. As of now, I am unconvinced that not having an associative operator for the RNN memory will lead to a good nearly permutation invariance function (unless there is data augmentation, per Janossy pooling).


Sequence labeling is a fundamental framework for various natural language processing problems including part-of-speech tagging and named entity recognition. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios. In many cases, ground truth labels are costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed or transferred from different domains. A typical example is crowdsourced datasets which have multiple annotations for each sentence which may be noisy or incomplete. Additionally, predictions from multiple source models in transfer learning can be seen as a case of multi-source supervision. In this paper, we propose a novel framework named Consensus Network (CONNET) to conduct training with imperfect annotations from multiple sources. It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism. Finally, it leads to a model reflecting the consensus among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings.

## Updated review
I have read the rebuttals. The new version of the paper is clearer and the new baseline experiments are a good addition.
## Original review
This paper presents an approach to train a neural networks-based model for sequence modelling using labels from different sources. The proposed approach explicitly models the annotator and uses an attention model to select the best aggregation method. The model is used on two scenarios: learning with crowd annotation and cross-domain adaptation. For the first scenario, noisy annotators are simulated with models trained on subsets of the data, the proposed model is compared with related works and is shown to achieve the highest f-1 score. For the second scenario, different domains in three NLP tasks are used and the model is shown to yield the best performance.
I think this paper should be accepted, for the following reasons:
- The approach is novel as far as I can tell, and the approach of learning to aggregate labels is significant, as it could also be applied to tasks where inter-annotators agreement is a problem.
- The experiments are convincing and show the potential of the proposed approach.
- The comparison with related works is thorough.
Detailed comments
- I don't understand the notion of "normalized expertise" in Section 5.4, can the authors briefly describe it in the paper ?
- The paper is not easy to read, for instance the first paragraph of Section 5 contains critical information to understand the experiments, maybe it should be moved the Section 4 and developed more, typically in two subsections "Application to crowd annotation" and "Application to cross-domain" for example.
- Typos:
    - Section 2, 3rd paragraph "for traget corpora" -> "target"
    - Same paragraph: "Yang & Eisenstien (2018) represented" -> "represents" to be consistent
    - Section 4.1: "BiLSTM-CRF" -> "BLSTM-CRF"


This paper demonstrates that a computational neural network model using ion channel-based conductances to transmit information can solve standard computer vision datasets at near state-of-the-art performance. Although not fully biologically accurate, this model incorporates fundamental biophysical principles underlying the control of membrane potential and the processing of information by Ohmic ion channels. The key computational step employs Conductance-Weighted Averaging (CWA) in place of the traditional affine transformation, representing a fundamentally different computational principle. Importantly, CWA based networks are self-normalizing and range-limited. We also demonstrate for the first time that a network with excitatory and inhibitory neurons and nonnegative synapse strengths can successfully solve computer vision problems. Although CWA models do not yet surpass the current state-of-the-art in deep learning, the results are competitive on CIFAR-10. There remain avenues for improving these networks, e.g., by more closely modeling ion channel function and connectivity patterns found in the brain.

This paper proposes a novel neural network architecture inspired by the analysis of a steady-state solution of the Hodgkin-Huxley model. Using a few simplifying assumptions, the authors use conventional backpropagation to train DNN- and  CNN-based models and demonstrate that their accuracies are not much lower than the state-of-the-art results.
The paper is well-written, sufficiently detailed and understandable. Derived self-normalizing Conductance-Weighted Averaging (CWA) mechanism is interesting in itself, especially contrasting CWA results with those obtained for the non-Batch-Normalized networks. It is also inspiring to see that this model can be derived based on a relatively accurate biological neuron model.
My main question is actually related to the potential impact of this study. I am curious about the implications and the ways in which these results can inspire other researchers.
After reading the paper, I got an impression that:
(a) From the point of view of a machine learning practitioner, these results may not be particularly impressive. They do hint at the importance of self-normalization though, which could potentially be interesting to explore further.
(b) From the point of view of a neuroscientist, the proposed model might be too simplistic. It is my understanding, that neural systems (even at "rest") are inherently non-equilibrium (and I assume the presence of simple feedback loops could also dramatically change the stead-state of the system). Is it possible that something similar to this "steady-state inference" mode could actually take place in real biological neural systems?
(c) Presented results appear to be important from the point of view of someone who wants to transfer insights from biology into the field of deep learning. But there might be an extent to what is achievable given a simple goal of optimizing a supervised accuracy of an artificial neural network trained using gradient descent (especially considering limitations imposed by hardware). I am optimistic about the prospect of knowledge transfer between these disciplines, but it is my feeling that the study of temporal dynamics, emergent spatiotemporal encodings, "training" process of a biological neural system, etc. have potentially much more to offer to the field of machine learning. These questions do appear to be incredibly complex though and the steady-state analysis is definitely a prerequisite.


Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel-or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain. During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance. However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples. In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation. To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model. To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task. On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of proposed framework.

This paper proposes a method to efficiently select hard samples during the training of a neural network. This is achieved via a variational auto-encoder (VAE) that encodes the samples into a latent space. The VAE is trained in a preparation stage using the images only and fixed at later stage. During training of a DNN framework, samples are selected in the latent space and then decoded via the Decoder in VAE to generate the input for DNN framework. The advantage of such a framework is that now it is able to calculate the gradient w.r.t. the input samples of DNN. This gradient is used to determine the sampling strategy in the next iteration to select harder samples. Two different sampling methods are explored, including nearest neighbor and interpolation (with annotation tool step). The experiments are conducted on small-scale datasets like MNIST CIFAR-10, and IVUS MSE with satisfactory gain over the baselines. Overall, the paper is very well-written and easy to follow. Although the experiment results are not super exciting mainly because of small-scale datasets and not enough gain in the numbers, some of the analysis in Figure 4 are quite insightful to validate the assumption and motivation of this work. So I propose to accept this work for its novelty. I think this work will benefit future research in this direction.
However, I do have some concerns that I wish the authors could clarify if possible. First, the approach is very similar to online hard negative mining (OHNM) that is purely based on the loss to repeatedly select the samples that generate a larger loss. The major difference is that this work can model the sample distribution and thus select samples based on the gradient w.r.t. the samples in the latent space. This is very novel to me. However, I am wondering if the authors could compare with this sample baseline of OHNM. My concern is that the baselines in this work is too simple and it is not surprise that there is advantage over a simple baseline that is trained without any hard sample mining.
Second, the experiments are all conducted on small-scale and simple datasets like MNIST and CIFAR10. I am concerned how effective this approach could work for large-scale dataset. In the experiment, even for CIFAR10, a vanilla VAE will not work to reconstruct the input. So the authors have used alpha-GAN to help image reconstruction. If that is the case for CIFAR10 with only 10 classes, how could we extend this work to even larger dataset with more complicated background like ImageNet? I would think the preparation step itself is a very challenging task. This is my major concern that will question the effectiveness of the approach in real applications.
Third, a related question to the above one. As the input to the DNN is the reconstructed image from the pre-trained decoder, there will be some information loss during the reconstruction process. This is the major challenge, I think, for large-scale applications. Is that possible to use the original image as the input to DNN while still being able to find hard samples using the latent space and the image space correlation?
Fourth, I really like the visualization of Figure 4 that shows the trajectory of the sampling process that follows the boundaries between classes. The authors also mentioned that some trajectories explore towards outside util there is no real samples, which should be avoided. Could the authors comment on how to avoid such cases? In my understand, as the input is randomly sampled at the beginning, it cannot avoid such cases unless some evaluation is done during training to stop the sampling for these trajectories.


As reinforcement learning continues to drive machine intelligence beyond its conventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommodates sparse reward environment, this paper presents Hindsight Trust Region Policy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, taking the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f -divergence. Such approximation results in the decrease of variance and alleviates the instability during policy update. Experimental results on both discrete and continuous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments.

*Summary* : This paper augments the TRPO policy optimization objective with hindsight data, where the hindsight data is generated from goals based on trajectories. The key contribution of the paper is based on deriving an on-policy adaptation of hindsight based TRPO, that can be useful for sparse reward environments. The paper draws ideas from existing papers such as HPG and considers the IS based variant of HPG for on-policy, similar to TRPO, that can achieve monotonic performance improvements. Furthermore, the authors introduce a logarithmic form of constraint, by re-deriving the KL constraint and leading to a f-divergence based constraint, which is argued to have useful effects in terms of lowering the variance. Experimental results are compared with baselines including HPG and its variants on standard sparse reward benchmark tasks.
*Comments* :
	- The core contribution of the paper is to introduce hindsight based TRPO where end states from the trajectory treated as goals can be useful for generating pseudo-rewards, such that existing TRPO based approaches can be better suited for sparse reward environments. The claim is that existing policy gradient methods cannot sufficiently explore in sparse reward environments.
	- Since incorporating hindsight data can make the approach off-policy in nature, leading to higher variance and instability, the authors propose to augment hindsight approach into on-policy based methods such as TRPO. The key contribution is to develop a theoretically sound approach for hindsight based policy optimization.
	- The hindsight TRPO objective is derived based on goal-conditioned policies with TRPO, based on existing recent work on Hindsight Policy Gradient (HPG) (Rauber et al., 2019). The key difficulty that needs to be tackled is when introducing hindsight experience makes the approach off-policy in nature. The authors derive the HTRPO objective based on an importance sampling approach that can also incorporate the hindsight data.
	- Equation 7 writes out the key quantity, an IS based approach considering the current goal and alternative goals to derive a similar TRPO objective based on IS based Monte-Carlo estimators, while maintaining the trust region guarantees. Equation 8 further shows the gradient of the objective. Theorem 3.1 and 3.2 follows naturally where the key trick in Thm 3.1 is going from equation 19 to 20 to derive the IS based estimator with goal conditioned trajectories. I am not sure why Thm 3.2 needs to be written out explicitly, given that it follows naturally for gradient of the expectation? Is there any key takeaway from it?
	- In TRPO, the expectation is based on states sampled from a state distribution. The authors argue that for hindsight based data, this state distribution in fact can change due to changes in the generated goals (ending states), and hence the KL needs to be in expectation w.r.t to the state occupancy measure. Furthermore, the authors change the KL based constraint into a logarithmic form of a constraint such as to reduce variance and introduce more stability. To achieve this, the paper uses an approximation to the logarithmic form of the constraint, by using an expectation of the square instead of plain differences between the log of policies. The key is that instead of using the KL divergence, the authors introduce f-divergence where the function is convex allowing smooth optimization.
	- The overall contribution of the paper can be summarized from equation 15 - introducing a IS based correction, while remaining on-policy for hindsight based TRPO objective. And since hindsight can change the underlying state distribution, leading to more instability, the paper introduces a different form of constraint (based on the f-divergence) which can have lower variance than the KL form of constraint.
	- Experimental results are demonstrated for few sparse reward benchmarks, comparing to the standard HPG, TRPO and several variants of the proposed HTRPO approach with weighted IS and with the KL constraint. The advantages of HTRPO on these tasks seems clear, mainly in the Fetch Reach and Fetch Push tasks, significantly outperforming the baselines. Even in the continuous tasks, HTRPO seems to outperform the baselines consistently.
*Feedback/Questions* :
	- I am not sure of the significance of Theorem 3.2 - it seems obvious that the gradient of the objective spans out naturally from equation 7?
	- The authors mention about the state occupancy measure instead of considering the state distribution for the KL term. However, the discussion of state occupancy measure seemed to have faded away? What was the significance of mentioning it, or why should it be considered even? There are no assumptions being made on whether the state distribution should be the discounted state occupancy measure or the stationary distribution (if infinite horizon is considered).
	- The introduction of the logarithmic form of constraint, even though shows theoretically to reduce variance, is not well motivated or demonstrated from the suite of experimental results? From the results, it is not obvious whether this form of constraint is indeed having useful effects in terms of reduced variance?
	- The paper seems to adapt from the HPG objective, and does indeed a great job comparing to HPG throughout the suite of experiments. However, in the results, the paper mainly compares to other off-policy based methods including HPG and its variants (official and re-implemented one). I find the comparison of results a bit odd in that sense, since it is comparing the on-policy adaptation of HPG (ie, HTRPO) and the off-policy variants? If run for long enough, does all converge to the same results? If so, then the benefits is mainly in faster learning (assumably due to better exploration in the sparse reward tasks). But then again, these benefits may be because of the on-policy approach compared to the off-policy one?
	- I would encourage the authors to compare to more standard goal-conditioned or reward shaping based baselines for TRPO. For example, does the proposed HTRPO approach perform better compared to other goal-conditioned approaches of TRPO, or for example if a form of reward shaping (based on goals) are used in TRPO? It would then be a more likewise comparison? The current results seem to show benefits of HTRPO, but I think there is a need for stronger baselines where TRPO + exploraton (reward shaping or goal conditioning) performs better?
	- I am not convinced about the arguments with sensitivity of HTRPO with network architecture and parameters. How is this demonstrated from the suite of results?

*Summary of Review and Score* :
Overall, I think the paper has merits, mainly in terms of deriving the on-policy adaptation with hindsight data. The key objectives are derived nicely in the write-up and easy to follow, although there are some confusions that need to be clarified (example : the discussion on state occupancy measure and the significance of it). The paper motivates exploration for TRPO in sparse reward tasks, and considers the hindsight adaptation of existing TRPO. But related works such as HPG have already taken a similar approach for the off-policy case, and this paper's key contribution is in terms of theoretically deriving the objectives for on-policy adaptation. However, I am not convinced about the overall merits and contributions of the paper, especially in terms of demonstrated results and proper comparisons with baselines. I think while the objectives and derivations follow naturally, the contributions of the paper is somewhat marginal.
I would therefore vote to marginally reject this paper - mainly in light of the core novel contribution and due to lack of sufficient results demonstrating the usefulness of the approach. The paper combines several things together - especially discussions of the logarithmic form of the constraint. I doubt whether all these introduced together led to the improvements shown in the experimental results or not. It would be useful to clarify the contributions from each of the components.


In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.

* Summary
The work considers reinforcement learning in partially observed environments, targeting in particular the case when the agent needs to remember some important information over a longer period of time.
The work gives an interesting connection between variational recurrent neural networks and reinforcement learning. The paper is well written and technically sound. The method achieves improved performance in several (simpler) benchmark problems in comparison to strong baselines.
* Detailed discussion
The work allocates latent variables to describe unobserved part of the state and learns the state transition probability model (the action-dependent generator (10)). Learning these state transitions becomes an unsupervised learning task, for which the lower bound on the likelihood of the observed data is optimized. This learning of the state transition model is actually similar to classical model-based RL. The policy and value functions are then learned on top of this model using SoTA techniques. From this perspective I would say this is a classical approach to partially observed reinforcement learning with advanced models for transition matrix and value and policy functions.
The work combines together variational recurrent neural network (VRNN) and soft actor critic (SAC) models into a rather advanced system for reinforcement learning. These parts are somewhat extended to fit together, but it is not clear whether all components are needed. For example, why z and d need to be distinguished, cannot they be combined into one latent variable “zd” in order to simplify the number of connections? Or I am missing some important property necessary for the inference?
In (7),(8) are the expectations over the same state / actions or different? The notation is not clear about it.
After (2), given x_t and d_{t-1} instead of d_t?
The experiments are conducted in the setting that the observations were all relevant measurements for the agent (positions, velocities). From Fig 7 RoboscoolHooper (close loop) we see that actually all measurements can be well predicted by generative model up to 8 steps ahead (in dim 2 that stays zero the noise is amplified). This is a nice and desirable property and is the consequence of the generative modeling and the marginal likelihood ELBO optimization. However the paper seem to argue (Appendix E) that the algorithm does not require the model to make accurate predictions and relies on the encoding capacity, which I find confusing. Furthermore, it would be interesting to see how this approach should scale to the setting when there are more sensory inputs, some less relevant than the other. In the end, is a good generative model to predict the future observation needed or not needed in this approach?
The exploration seems to be addressed just by randomizing the policy,  are there any better options?


In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using bytealigned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5 MB (20× compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26× factor.

The suggested method proposes a technique to compress neural networks bases on PQ quantization. The algorithm quantizes matrices of linear operations, and, by generalization, also works on convolutional networks. Rather than trying to compress weights (i.e. to minimize distance between original and quantized weights), the algorithm considers a distribution of unlabeled inputs and looks for such quantization which would affect output activations as little as possible over that distribution of data. The algorithm works by splitting each column of W_ij into m equal subvectors, learning a codebook for those subvectors, and encoding each of those subvectors as one of the words from the codebook.
The method provides impressive compression ratios (in the order of x20-30) but at the cost of a lower performance. Whether this is a valuable trade-off is highly application dependent.
Overall I find the paper interesting and enjoyable. However, as I am not an expert in the research area, I can not assess how state of the art the suggested method is.
There are a few other questions that I think would be nice to answer. I will try to describe them below:
Suppose we have a matric W_{ij} with dimensions NxM where changing i for a given j defines a column. By definition, linear operation is defined
y_i = sum_j W_ij x_j . Now say each column of matrix W is quantized into m subvectors. We can express W_ij in the following way:
W_ij = (V^1_ij + V^2_ij + ... V^m_ij)x_j where V^m_ij is zero everywhere except for the rows covering a given quantized vector.
For example, if W had dimensions of 8x16 and m=4,
V^2_{3,j}=0, for all j, V^2_{4,j}=non_zero, V^2_{7,j}=non_zero, V^2_{8,j}=0, V^2_{i=4:8,j}=one_of_the_quantized_vectors.
y_i = sum_j W_ij x_j = sum_k sum_j (V^k_ij) x_j =def= sum_k z^k_i where z^k are partial products: z^k_i=0 for i<k*N/m and i>(k+1)N/m
Thus, the suggested solution effectively splits the output vector y_i into m sections, defines sparse matrices V^k_{ij} 1<=k<=m, and performs column-wise vector quantization for these matrices separately.
Generally, it is not ovious or given that the current method would be able to compress general matrices well, as it implicitly assumes that weight W_{ij} has a high "correlation" with weights W_{i+kN/m,j} (which I call "vertical" correlation), W_{i,k+some_number} (which I call "horizontal" correlation) and W_{i+kN/m,k+some_number} (which I call "other" correlation). It is not given that those kind of redundancies would exist in arbitrary weight matrices.
Naturally, the method will work well when weight matrices have a lot of structure and then quantized vectors can be reused. Matrices can have either "horizontal" or "vertical" redundancy (or "other" or neither). It would be very interesting to see which kind of redundancy their method managed to caprture.
In the 'horizontal' case, it should work well when inputs have a lot of redundancy (say x_j' and x_j'' are highly correlated making it possible to reuse code-words horizontally within any given V^k: V^k_ij'=V^k_ij''). However, if thise was the case, it would make more sense to simply remove redundancy by prunning input vector x_j by removing either x_j' or x_j'' from it. This can be dome by removing one of the outputs from the previous layer. This can be a symptom of a redundant input.
Another option is exploiting "vertical" redundancy: this happens when output y_i' is correlated with output y_{i'+N/m}. This allows the same code-word to be reused vertically. This can be a symptom of a redundant output. It could also be the case that compressibility could be further subtantially improved by trying different matrix row permutations. Also, if one notices that y_i' ir correlated with y_i'', it might make sense to permute matrix rows in such a way that both rows would end up a multiple N/m apart. It would be interesting to see how this would affect compressibility.
The third case is when code words are reused in arbitrary cases.
Generally, I think that answering the following questions would be interesting and could guide further research:
1. It would be very interesting to know what kind of code-word reusa patterns the algorithm was able to capture, as this may guide further research.
2. How invariance copressibility is under random permutations of matrix rows (thus also output vectors)?


Pre-trained Deep Convolutional Neural Network (CNN) features have popularly been used as full-reference perceptual quality features for CNN based image quality assessment, super-resolution, image restoration and a variety of imageto-image translation problems. In this paper, to get more insight, we link basic human visual perception to characteristics of learned deep CNN representations as a novel and first attempt to interpret them. We characterize the frequency and orientation tuning of channels in trained image classification deep CNNs (e.g., VGG-16) by applying grating stimuli of different spatial frequencies and orientations as input. We observe that the behavior of CNN channels as spatial frequency and orientation selective filters can be used to link basic human visual perception models to their characteristics. Doing so, we develop a theory to get more insight into deep CNN representations as perceptual quality features. We conclude that sensitivity to spatial frequencies that have lower contrast masking thresholds in human visual perception and a definite and strong orientation selectivity are important attributes of deep CNN channels that deliver better perceptual quality features.

I thank the authors for their detailed response. Some of my questions have been addressed in the rebuttal, but as far as I can tell very few modifications have been made to the paper: mainly an additional super-resolution experiment in the appendix, which goes in a good direction, but currently comes across as quite preliminary. So I think the paper still has most of the weaknesses I mentioned and thus it is not quite fit for publication. But I do encourage the authors to strengthen the experiments (for instance by addressing the points I raised, or by some other means) and resubmit elsewhere.
---
The paper proposes an approach to analyzing the properties of "perceptual metrics" used in deep learning image generation methods. "Perceptual metrics" are computed by measuring distances between images not in the pixel space, but i na feature space of a pre-trained cNN. The proposed analysis method, inspired by studies of human perception, is based on measuring the response of these features to sinusoidal gratings of varying frequency or orientation. Based on these responses, the paper proposes a "Perceptual Efficacy Score" that should measure the importance of certain feature in the feature maps for the performance of a perceptual metric. Experiments show that indeed distances measured between features with high score better correlate with human judgement of image similarity than distances between features with a lower score.
I find the paper quite interesting, but lean towards rejection at this point. This i smainly because the experiments seem somewhat anecdotal and incomplete, see below for further details.
Pros:
1) Application of methods from psychology/neuroscience to artificial neural networks is an interesting avenue of work. Moreover, better unsderstanding of "perceptual metrics" is of wide interest for various image processing applications.
2) The proposed score seems to indeed correlate quite well with the importance of features for human judgement of image similarity.
3) Presentation is mainly clear.
Cons:
1) Experiments are not very exhaustive and at times a bit confusing. For instance:
1a) Results are sometimes presented in a confusing way. In Figure 4 first of all it is not quite clear what points correspond to I guess each point is an image) and, second, it is not very obvious that the correlation is higher i none of the plots. In tables 1 and 2 it is confusing that different percentiles for H and L are used for different networks/layers. Is this based on some tuning? Then the tuning process should be clearly explained. Moreover, it might be useful to report the full curves of performance as a function of the percentage of features used.
1b) There are no baselines and there is not much justification of computing the "Perceptual Efficacy" score the way it is computed. What if one uses only the orientation-based score? Or only the frequency-based? What if one selects the most relevant features in a data-driven way (based on correlation on a training set)? What if one selects subsets of features randomly?
1c) While the method is inspired by methods used for studying natural vision systems, there is no connection to human experiments. It would be interesting to see a comparison of frequency and orientation tuning of features in a CNN to human cells (as I understand, the latter should be available in prior works?).
1d) It would be great to see the selected features be used not only for offline image similarity assessment, but also for training image processing models - in the end, this has been the main use of "perceptual metrics". Do they lead to improved results?
1e) Since the paper is about (subjective) image quality, it might be useful to show some qualitative results, potentially in the appendix if space is an issue.
2) There are some issues with the presentation:
2a) I had a hard time understanding what exactly "Contrast Sensitivity Function" and "contrast masking" are.
2b) Minor issues:
- In the abstract: "trained object detection deep CNNs" - I guess image classification is meant
- Beginning of Section 2: "Section. 2", "convolution layer as collection channels"
- Section 4: "corresponds the the peak:
- Section 5.2 "Berkeley-Adobpe"