Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings. We hope that the Jelly Bean World will prompt new interest in the development of never-ending learning, and more broadly general intelligence.

Summary
This paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features.
Review
The paper highlights many key characteristics of an environment that are challenging to current RL models. This focus on building a benchmark upon which further research can measure performance is important. I find the proposed environment to be incredibly intriguing and would find it valuable to the field of lifelong learning (or continual learning or never-ending learning, etc.). I think the size and scope of the environment generator is impressive, showing a considerable amount of engineering effort has gone into its design.
The largest overarching issue that I would like to point out is the limited study of modelling choices. I am not an expert on applied Reinforcement Learning, so I can make very few claims about the validity of the chosen network architecture or use of the PPO policy-gradient algorithm for this environment. However, it is critical, in my view, that a paper introducing a new environment studies these effects itself; demonstrating how various degrees of learning capacity or wider ranges of learning algorithms behave in the given environment. If a slightly larger network architecture trivially solves each task in this environment, can this still be considered a benchmark task? A key result in the paper that I would like to see further investigated (even with only a different network architecture) would be Figure 6, the comparison between scent, vision, and vision+scent. It is unclear to me why the scent features would be so challenging to learn from and specifically why they would harm the representation so permanently. A deeper study using only the scent features would be valuable to me. In its current state, it appears that these feature provide no additional information and are thus not necessary to include in the environment; breaking one of the primary motivating features of JBW: the multi-modality. I recognize that the paper comments on the orthogonality of the scent playing a role, and notes that further results are included on a not-yet-available website (presumably to maintain anonymity). However, I would like to see these results included in the appendix of the paper so I could better assess the utility of the scent features. Perhaps an additional result showing the average reward versus the cosine distance (or other measure of orthogonality) between "jellybeans" and "onions" would additionally motivate the utility of the scent features.
The paper empirically investigates the use of curriculum learning to accelerate learning for a particular task. The paper then claims that curriculum learning improves learning speed, but ultimately does improve final performance. This demonstration is intended to showcase the use of the proposed environment (JBW) for curriculum learning. However, there are few key issues with this empirical study. First, the paper shows the reward rate of 3 different curricula but does mention the metric used to compare the agents during the time the curricula is active. It is implied that the metric is the reward rate of each individual agent; however, each agent has a unique reward function making comparisons between agents impossible. Curriculum #2 can only receive positive rewards while Curriculum #1 can only receive negative rewards. Naturally this means that Curriculum #2 must have strictly greater or equal reward rate over Curriculum #1. Even in the case that the final objective specifies the metric used, these are still highly non-comparable entities. A suggestion to improve this result would be to run each curricula for 100k as a "pretraining" phase, then to restart the agents to the same state in the environment and measure their performance from there.
The case study measuring the effects of non-stationarity of the rewards does not provide sufficient evidence that the proposed environment contributes a novel ability to investigate non-stationarity. First, the given study of non-stationarity focuses solely on an alternating reward function, clearly demonstrating the problem of catastrophic forgetting. While this is a motivating demonstration, it is not novel and the issue of catastrophic forgetting in our models has been known since at least the 90s (e.g. French 1999 and related). Carefully and scientifically investigating such an issue is best done in a far less complex environment where more precise results can be drawn. Further, the ability to oscillate a reward function in this way is not unique to this environment and can be trivially done in most environments. Secondly, it is unclear if JBW allows for non-stationarity in the transition probabilities in the MDP. This is a critical component to non-stationarity and would be a necessary feature for me to claim non-stationarity is widely supported in the environment.
The paper starts with a motivating conversation about environment complexity, with interesting insights into measuring the complexity of an environment based on the complexity of the policy used to solve that environment. However this conversation is ignored until the conclusion of the paper, where the paper claims to have built an environment of greater complexity than already existing environments. Without any supporting evidence in the body of the paper, it is impossible to verify the validity of this statement, and it is still an open question to me whether this claim is even falsifiable in the first place. As a concrete counter-claim, I would claim that the Minecraft environment (Malmo) has similar or higher complexity to the proposed environment in most aspects. Minecraft has a far greater diversity of objects, a third dimension of movement, adversarial components, hunger and health, etc. each of which adding a large level of complexity not achievable in the proposed environment. This is not to say that I expect the proposed environment to contain these features, but rather to point out that claims of greater complexity may be ill-founded.
Additional Comments (not affecting score)
I do slightly question if ICLR is the appropriate venue for such work. While I recognize that the scope of this conference has shifted considerably over the past few years, this paper (as written) does not further understanding or study of learning representations. I believe a more careful demonstration of the representation induced by characteristics of the environment is within easy reach of the paper, but is not currently presented.
-----------
After the author response, reading other reviews/responses, and looking at the edited draft:
I am convinced of the utility of the domain, the scope of the engineering effort put into building, and the ease with which it can be configured by the user to test many applicable settings (partial observability, stochasticity in transitions and rewards, etc.). I remain slightly skeptical of the amount of benefit the proposed provides over the Malmo environment for any of the settings discussed in the case-studies.
I specifically feel my concerns about the stochasticity in the transitions and environment complexity have been well addressed. My concerns about the curriculum learning demonstration are partially addressed to a point where I am satisfied. My concerns about the modeling choice are also partially satisfied, with one lingering concern. I am unclear if the environment is trivially solvable by using more computation resources (e.g. bigger networks). However, after reconsideration I decided this concern bares less weight than I previously considered.
All this considered, I am changing my rating from 3 -> 6.


Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out-of-the-box for representative models: ResNet-50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors -shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization.

There has been a great deal of interest and research into reduced numerical precision of weights, activations and gradients of neural networks. If, for example, 16 bit floating point can be used instead of 32 bit floating point, then the memory bandwidth is halved along with significant gains in computational performance.
In this work the authors propose an 8-bit floating point format (denoted S2FP8) for tensors. In general, computing activations and gradients with such low precision at training time, has generally proved challenging without a number of tricks such as scaling the loss of each minibatch to within a reasonable range. Such ``tricks'' can be difficult to tune for each problem.
The key idea here is that for each tensor of 8-bit numbers, two 32 bit floating point statistics are recorded as well. These determine (in log-space) a scale and an offset for the 8-bit numbers (eq 1). This means that in this format tensors of significantly different scales can be well-represented (although larger scales necessarily implies low precision).
Matrix multiplications are done in FP32 precision and then converted in S2FP8 format. This requires an additional step to accumulate the summary statistics of each tensor in order to convert from FP32 to S2FP8 (the mean and the max of the tensor elements in log space).
The weights of the network are stored in FP32 and the gradients and activations are computed in S2FP8 and used to update the weights.
They test this approach in ResNet, a small transformer and a MLP for collaborative filtering. They find it reaches similar performance to FP32 where standard FP8 format has worse performance or results in Nan's.
Improvements in computational efficiency, both at training and inference, are active areas of research and this work contributes a novel approach using summary statistics. However, there are a several ways this work could be improved.
1. There is no comparisons with bfloat16, which is becoming a widely used approach to lower precision and is gaining significant hardware support [1].
2. Discussion and analysis regarding the need to gather summary statistics after each matrix multiplication (or other tensor operation). It is claimed that this brings minimal HW complexity, but this doesn't seem well justified. For a large tensor, this additional reduction to compute statistics may be expensive (in memory bandwidth and computation), particularly since this is done with FP32.
3. Even with the current implementation on a GPU, it should be possible with kernel fusions to gain some significant savings in memory bandwidth (and therefore computational speed), but there is no attempt anywhere to show any runtime benefit on current hardware.
Minor issues:
Some captions are very terse and the figures would benefit from a clearer explanation (e.g. figure 6).
[1] https://en.wikipedia.org/wiki/Bfloat16_floating-point_format


The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PAIRNORM, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PAIRNORM is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PAIRNORM makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs.

Summary
It is known that GNNs are vulnerable to the oversmoothing problem, in which feature vectors on nodes get closer as we increase the number of (message passing type graph convolution layers). This paper proposed PairNorm, which is a normalization layer for GNNs to tackle this problem. The idea is to pull apart feature vectors on a pair of non-adjacent nodes (based on the interpretation of Laplace-type smoothing by NT and Maehara (2019)). To achieve this approximately with low computational complexity, PairNorm keeps the sum of distances of feature vectors on all node pairs approximately the same throughout layers. The paper conducted empirical studies to evaluate the effectiveness of the method. PairNorm improved the prediction performance and enabled make GNNs deep, especially when feature vectors are missing in the large portion of nodes (the SSNC-MV problem).
Decision
I want to recommend to accept the paper because, in my opinion, this paper contributes to deepening our understanding of graph NNs by giving new insights into what causes the oversmoothing problem and which types problem (deep) graph NNs can solve.
The common myth about graph NNs is that they cannot make themselves deep due to the oversmoothing. Therefore, oversmoothing is one of the big problems in the graph NN field and has been paid attention from both theoretical and empirical sides. This paper found that the deep structures do help to improve (or at least worsen) the predictive performance when the significant portion of nodes in a graph does not have input signals. To the best of our knowledge, this is the first paper that showed the effectiveness of deep structures in citation network datasets (Deep GCNs [Li et al., 2019] successfully improved the prediction performance of (residual) graph NNs using as many as 56 layers for point cloud datasets). The proposed method is theoretically backboned, easy to implement, and applicable to (theoretically) any graph NNs. Taking these things into account, I would like to judge the contribution of this paper is sufficiently significant to accept.
Minor Comments
	- Table 3. Remove s in the entry for GAT-t2 Citeseer 0%.
Questions
	- Can we interpret PairNorm (or the optimization problem (6)) from the viewpoint of graph spectra?
	- Although the motivation of Centering (10) is to ease the computation of TPD, I am curious how this operation contributes to performance. Since the constant signal does not have information for distinguishing nodes, eliminating it by Centering might result in emphasizing the signal component for nodes classification tasks. From a spectral point of view, Centering corresponds to eliminating the lowest frequency of a signal.
	- Figures 3 and 7 have shown that GCN and GAT did not perform well compared to SGC when the layer size increases. The authors discussed that this is because GCN and GAT are easier to overfit. However, SGC chose the hyperparameter $s$ from $\{0.1,1,10,50,100\}$, whereas the authors examined a single $s$ for GCN and GAT. Therefore, I think there is another hypothesis that simply the choice $s$ was misspecified. If this is the case, I am interested in the effect of $s$ on predictive performance.
[Li et al., 2018] Li, Qimai, Zhichao Han, and Xiao-Ming Wu. "Deeper insights into graph convolutional networks for semi-supervised learning." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.


Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyper-parameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks.
(1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyper-parameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of finetuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning.

This submission studies the problem of transfer learning and fine tuning. This submission proposes four insights: Momentum hyperparameters are essential for fine-tuning; When the hyperparameters satisfy some certain relationships, the results of fine-tuning are optimal; The similarity between source and target datasets influences the optimal choice of the hyperparameters; Existing regularization methods for DNN is not effective when the datasets are dissimilar. This submission provides multiple experiments to support their opinion.
Pros:
+  This submission provides interesting facts that are omitted in previous research works.
+  This submission examines the previous theoretical results in empirical setting and finds some optimal hyperparameter selection strategies.
+  This submission provides many experiment results of fine-tuning along with its choice of hyperparameters that could be taken as baselines in future researches.
Cons:
-	All experiments results are based on same backbone, which makes all discoveries much less reliable. More experiments on other backbones are necessary. Furthermore, this submission claims that the regularization methods such as L2-SP may not work on networks with Batch Normalization module. But there is no comparison on networks without BN.
-	Providing a complete hyperparameter selecting strategy for fine-tuning could be an important contribution of this submission. I suggest authors to think about it.
-	This submission claim that the choice of hyperparameters should depend on similarity of different domains. But this submission does not propose a proper method for measure the similarity or provide detailed experiments on previous measurements.
-	It seems that the MITIndoors Dataset is not similar with ImageNet from the semantic view. This submission does not provide similarity measurement between these datasets. Why the optimal momentum is 0?
-	The effective learning rate and ‘effective’ weight decay are not first given in this submission. This makes the novelty of this submission relatively weak. Authors only test these strategies in fine-tuning setting and find that they also work with a different initialization.
-	It seems that merely searching for learning rate and weight decay hyperparameters (as Kornblith et al. (2018) did) on a fixed momentum is Ok if there is a most effective relationship between learning rate and momentum. So the discoveries in the first part that a 0 momentum can be better is based on a careless search of learning rates?
-	This submission omits that Kornblith et al. (2018) also referred to the fact that the momentum parameter of BN is essential for fine-tuning and provided a strategy in section A.5. Discussion about this strategy will make this submission more complete.
This submission gives important discoveries about the hyperparameter choice in the fine-tuning setting. But there are several flaws in this submission. I vote for rejecting this submission now but I expect authors to improve the submission in the future version.


Gradient perturbation, widely used for differentially private optimization, injects noise at every iterative update to guarantee differential privacy. Previous work first determines the noise level that can satisfy the privacy requirement and then analyzes the utility of noisy gradient updates as in non-private case. In this paper, we explore how the privacy noise affects the optimization property. We show that for differentially private convex optimization, the utility guarantee of both DP-GD and DP-SGD is determined by an expected curvature rather than the minimum curvature. The expected curvature represents the average curvature over the optimization path, which is usually much larger than the minimum curvature and hence can help us achieve a significantly improved utility guarantee. By using the expected curvature, our theory justifies the advantage of gradient perturbation over other perturbation methods and closes the gap between theory and practice. Extensive experiments on real world datasets corroborate our theoretical findings.

This paper proposes a quantity called expected curvature to analyze the convergence of gradient perturbation based methods that achieves differential privacy. Comparing to minimum curvature, which was used in previous convergence analyses, expected curvature better captures the properties of the optimization problem, and thus offers an explanation for the advantage of gradient perturbation based methods over objective perturbation and output perturbation.
Using expected curvature is a pretty interesting idea, and having more refined convergence bound is useful. I have the following questions.
1. It seems to me that the convergence bound is similar to previous bound, except that \mu is replaced by \nu and one log(n) disappears. Could you explain more intuitively how hard the new analysis is? Is it similar to just replacing any \mu by \nu in the previous analysis? And why do they differ by log(n)?
2. How are the experiments (in terms of setup and results) differ from those in Iyengar et al? It seems to me the paper is proposing a method for convergence analysis and the DP algorithms remain the same. I feel like in Iyengar et al, there is no clear difference between gradient perturbation and objective perturbation. Maybe I was wrong about that, but could you elaborate more?
3. I agree that the expected curvature better captures the convergence of gradient-based DP methods. Yet I don’t see clearly how this can be used to show that they have more advantages than objective perturbation. Is it possible that the analyses of objective perturbation can also be improved (maybe with other techniques)? Since all we have are upper bounds, I feel like it is a bit early to conclude that it is less powerful. You mentioned that “That is because DP makes the worst-case assumption on query function and output/objective perturbation treat the whole learning algorithm as a single query to private dataset. ” I didn’t follow this part. I feel like DP is always making a worst-case assumption; even for gradient perturbation, you need to add noise to protect the worst case. Could you elaborate more on that?
4. My understanding is that for different dataset the curvatures would be different. I think it might be interesting if you plot something similar to Figure 1 for other datasets and compare how they match with the training curve. Do you expect them to look very different on different dataset / optimization problem?


We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x, y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may be used and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, and out-of-distribution detection while also enabling our models to generate samples rivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and present an approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-art in both generative and discriminative learning within one hybrid model.

This work is an attempt to bridge the gap between discriminative models, which currently obtain the state of the art on most classification problems, and generative models, which (through a model of the marginal p(x)) have the potential to shine on many tasks beyond generalization to a hold-out set with minimal shift in distributions: out of distribution detection, better generalization out of distribution, unsupervised learning etc.
While much of the current work is related to normalizing flows / invertible neural networks, the authors here propose a quite simple but appealing method: A standard neural classifier is taken and the softmax is layer chopped off and replaced by an energy based model, which models the joint probability p(x,y) instead of the posterior p(y|x). The advantage is an additional degree of freedom in the scale of the logit vector, which is would have been otherwise normalized by the softmax layer and now can now model the data distribution. The downside is the loss in ease of training. Whereas (discriminative) deep networks can be easily trained by gradient descent on a cross-entropy objective, the partition function in the energy model makes this un tractable. This is addressed through sampling, similar to (Welling & Teh, 2011).
One of the biggest achievements reported by the authors is that the performance on discriminative tasks is not hurt (much) by adding the generative model. There is only a 3 point gap between Wide-ResNet and the proposed model (92.9% vs. 95.8%) … but on what dataset? 3 datasets are mentioned in the experimental section, but table 1 does not mention on which datasets the accuracy is reported. My guess is that this is a mean or mixture, since GEM performances of 96.7% and 72.2% are reported for SVHN and CIFAR10, respectively, but this should be made clearer.
On out of distribution detection, could the authors comment on the histograms in table 2, in particular the difference between the new measure (AM JEM) compared to JEM log p(x) on CelebA? The proposed measure does not seem to fare well here.
Although the method does not outperform the gold standard of adversarial training, I found the models robustness to adversarial examples quite appealing, given that it was not trained for this objective (which also means that it does not require an adaptation to a norm).
I was very impressed by Figure 6 showing distal adversarial initialized from random images, showing pretty clear images of the modelled class. The modelled variations require more investigation to verify whether we have a collapse for each class, but the results look very promising.
The paper is well written and easy to understand. A couple of details on the training procedure are missing in the experimental part. It is stated that, both, p(y|x) and the generative part p(x), are optimized, but how are these exactly integrated? Given the difficult in training this model reported in the paper, this seems to be particularly important.
I also appreciated the description of the limitations of the algorithm, and the details in the appendix (ICLR should go back to unlimited paper lengths, btw.).
More information on complexity (training times etc.) should also be helpful.


It is valuable yet remains challenging to apply neural networks in logical reasoning tasks. Despite some successes witnessed in learning SAT (Boolean Satisfiability) solvers for propositional logic via Graph Neural Networks (GNN), there haven't been any successes in learning solvers for more complex predicate logic. In this paper, we target the QBF (Quantified Boolean Formula) satisfiability problem, the complexity of which is in-between propositional logic and predicate logic, and investigate the feasibility of learning GNN-based solvers and GNN-based heuristics for the cases with a universal-existential quantifier alternation (so-called 2QBF problems). We conjecture, with empirical support, that GNNs have certain limitations in learning 2QBF solvers, primarily due to the inability to reason about a set of assignments. Then we show the potential of GNN-based heuristics in CEGAR-based solvers, and explore the interesting challenges to generalize them to larger problem instances. In summary, this paper provides a comprehensive surveying view of applying GNN-based embeddings to 2QBF problems, and aims to offer insights in applying machine learning tools to more complicated symbolic reasoning problems.

This paper explores how graph neural networks can be applied to test satisfiability of 2QBF logical formulas. They show that a straightforward extension of a GNN-based SAT solver to 2QBF fails to outperform random chance, and argue that this is because proving either satisfiability or unsatisfiability of 2QBF requires reasoning over exponential sets of assignments. Instead, they show that GNNs can be useful as a heuristic candidate- or counterexample- ranking model which improves the efficiency of the CEGAR algorithm for solving 2QBF.
This is a clear, well-written, and well-structured paper, and I support accepting it to ICLR. That being said, I am not as familiar with the literature on neural solvers for logic problems, so I base my review on the content within the paper more than its context in the field.
I can’t find much to fault with the writing and arguments. The GNN architecture for 2QBF (Section 2) is simple, elegant, and well-motivated as a minimal extension of successful SAT solvers. The arguments in Section 3 are convincing, and make a good case for why an algorithm such as CEGAR is necessary. Finally, the metrics in Section 4 are clearly interpretable and well-justified.
A couple questions and concerns:
In Section 3, The amount of training data (up to 160 pairs of formulas for predicting satisfiability) seems to be very small for a machine learning problem. By comparison, Selsam et al. 2019 says they train their GNN SAT solver on “millions of problems” (Section 5). Is there a good reason for using a much smaller dataset, given that 2QBF is a harder class of problem?
Section 4.2: how are the TraunU, TrainS, TestU, and TestS datasets generated?
In Section 4.6, are the models re-trained on these new distributions, or on the data described in Section 4.2? (If the latter, how does the GNN perform if re-trained on the larger-spec data?)
And minor points on clarity:
* “-” for the baseline seems a bit awkward; consider spelling out “vanilla”?
* Are all the numbers in the tables iteration counts, unless specified otherwise? It would help to restate this in the captions. Similarly, I wonder if there could be more informative names for GNN1, GNN2, GNN3, and GNN4?


An important property of image classification systems in the real world is that they both accurately classify objects from target classes ("knowns") and safely reject unknown objects ("unknowns") that belong to classes not present in the training data. Unfortunately, although the strong generalization ability of existing CNNs ensures their accuracy when classifying known objects, it also causes them to often assign an unknown to a target class with high confidence. As a result, simply using low-confidence detections as a way to detect unknowns does not work well. In this work, we propose an Unknownaware Deep Neural Network (UDN for short) to solve this challenging problem. The key idea of UDN is to enhance existing CNNs to support a product operation that models the product relationship among the features produced by convolutional layers. This way, missing a single key feature of a target class will greatly reduce the probability of assigning an object to this class. UDN uses a learned ensemble of these product operations, which allows it to balance the contradictory requirements of accurately classifying known objects and correctly rejecting unknowns. To further improve the performance of UDN at detecting unknowns, we propose an information-theoretic regularization strategy that incorporates the objective of rejecting unknowns into the learning process of UDN. We experiment on benchmark image datasets including MNIST, CIFAR-10, CIFAR-100, and SVHN, adding unknowns by injecting one dataset into another. Our results demonstrate that UDN consistently outperforms state-of-the-art methods at rejecting unknowns -up to 20 point gains in accuracy, while still preserving the classification accuracy.

This paper is about a novel method to detect unknown samples which are of a different class than the trained ones. The idea is to use an output subnet which use a fully connected layer and a binary tree which encode the product relationship instead of the sum currently used in state of the art method (particularly the softmax with low confidence). The binary tree is made of split nodes which are responsible to produce a probability distribution from the root to each leaf. The max path i.e. the path with the largest probabilities determines the class of the input and can be used to measure how confident the classifier is about the classification decision. Combine multiple subnets and the tool obtained is able to do complex predictions and maintain a good generalization performance. The method also uses an information theory based regularization which decrease the probability of having subnets with uniform probability distribution i.e. a large entropy. Experiments on CIFAR-10 and MNIST against CIFAR-100, SVHN show that the method has an improved rejection accuracy while maintaining a good classification accuracy on the test set.
The topic of the paper is interesting and the approach seems to be solid, however the experiments are not so convincing. They are limited on two very easy datasets and do not show if the method is able to scale when more difficult and more realistic amount of classes are considered (like in CIFAR-100, SVHN or maybe ImageNet). Given that a simple dataset like these require 9 hour of training, it is also not clear how much the method is able to scale computationally and if it is applicable realistically. Moreover the presentation could be improved as figure 1 and its section 2 are complex and not easy to follow at several points. Hence, I’m leaning towards rejecting this paper.
In particular:
- It would be interesting to see experiments where the number of classes is higher than the ones in MNIST and CIFAR-10. CIFAR-100 and SVHN would be a good testbed for such case.
- How is the complexity of the method and how does it scale with the number of training classes?
- It is stated that the method bring a 25% percentage points in the accuracy of unknown rejection detection, however table 1 shows a large improvement only in the case of SVHN. Hence the claims seems a bit off compared to the measured data. Moreover, using only CIFAR-10 is insufficient to back up the claim.
- Why the related work section is at the end of the paper? It confuses the reader and would be more useful to be put after the introduction section.


Training neural networks to be certifiably robust is critical to ensure their safety against adversarial attacks. However, while promising, state-of-the-art results with certified training are far from satisfactory. Currently, it is very difficult to train a neural network that is both accurate and certifiably robust. In this work we take a step towards addressing this challenge. We prove that for every continuous function f , there exists a network n such that: (i) n approximates f arbitrarily close, and (ii) simple interval bound propagation of a region B through n yields a result that is arbitrarily close to the optimal output of f on B. Our result can be seen as a Universal Approximation Theorem for interval-certified ReLU networks. To the best of our knowledge, this is the first work to prove the existence of accurate, interval-certified networks.

The paper aims to show that there exist neural networks that can be certified by interval bound propagation. It claims to do this by showing that for any function f, there is a neural network g (close to f) such that interval bound propagation on g obtains bounds that are almost as good as one would get by applying interval bound propagation to f.
There are a couple issues with this paper. The first is that the main theorem does not imply the claimed high-level take-away in the paper in any practically relevant regime. This is because while the paper does show that there is a network that approximates f well enough, the size of the network in the construction is exponential (for instance, one of the neural network components in the construction involves summing over all hyperrectangles lying within a grid). The result is only plausibly practically relevant if there is a polynomially-sized network for which the bound propagation works.
The second issue is the comparison to related work, which omits both several key techniques in the literature and two related papers on universal approximation.
On universal approximation papers, see these two: https://arxiv.org/abs/1904.04861 and https://arxiv.org/abs/1811.05381. They consider a different proof strategy but more plausibly yield networks of reasonable size.
On key techniques in the literature: "typically by employing methods based on mixed integerlinear programming, SMT solvers and bound propagation" ignores several major techniques in the field: convex relaxations (SDP + LP), and randomized smoothing. Similarly, "specific training methods have beenrecently developed which aim to produce networks that are certifiably robust" again ignores entire families of techniques; Raghunathan et al. train SDP relaxations to perform well, while Cohen et al. train randomized smoothing to work well. Importantly, randomized smoothing is not about creating an over-approximation of the network but explicitly constructs networks that are smooth.
"some of the best results achieved on the popular MNIST (LeCun et al., 1998) and CIFAR10(Krizhevsky, 2009) datasets have been obtained with the simple Interval approximation (Gowalet al., 2018; Mirman et al., 2019)"
-Is this actually true? I just looked at Mirman et al. and it doesn't seem to explicitly compare to any existing bounds. My impression is that randomized smoothing (Cohen et al.) currently gets the best numbers, if we're allowed to train the network. If not allowed to train the network, Raghunathan et al. (Neurips 2018) performs well and is not compared to in the Mirman paper.
These omissions must be addressed as the introduction and related work is misleading in its current state.
Finally, some writing issues:
>>> While the evidence suggests "no", we prove that for realisticdatasets and specifications, such a network does exist and its certification can beestablished by propagating lower and upper bounds of each neuron through thenetwork
One cannot "prove" something for "realistic datasets", since "realistic datasets" is not a formal assumption. Please fix this.
>>> "the most relaxed yet computationally efficient convex relaxation"
What does most relaxed mean? Most relaxed would most straightforwardly mean outputting the entire space as the set of possibilities at each point, which is clearly not intended.


Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the conatural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low-resource scenario.

This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.
Although the idea seems reasonable and interesting, I feel like this paper needs work both in the theory and experiments. Figure 1 is a nice visualisation of the key take-away point of the paper.
Theory: the authors take the natural-gradient updates from batch learning and just modify it so that the KL term is now for the previous task(s) instead of the current one. Although this may seem reasonable, I would appreciate some analysis as to what this implies or means.
Experiments:
- Split CIFAR from Chaudhry et al. (2018b) uses 10 tasks, why does this paper use 20 tasks?
- Previous works usually find that for EWC, large values of the \lambda hyperparameter provide best results. This corresponds to lower forgetting of previous tasks. The hyperparameter range in Appendix A.2.3 is only over small values of \lambda (by orders of magnitude).
- Why do the authors only allow 1 epoch per task for Split CIFAR? This probably results in early stopping: the new tasks are not able to reach their new optimal points (with or without regularised trajectories). This seems to go against the intuition provided by Figure 1, where the authors are showing that changing the trajectory results in a better local minimum being found.
In fact, by adding another regularisation term, it is unsurprising that co-natural gradient updates have less forgetting, as the extra regularisation term probably means the trained parameters are even closer to the previous parameters.
-------------------
EDIT: Score changed to 'Weak Accept' following discussion with the authors.


We consider the challenge of estimating treatment effects from observational data; and point out that, in general, only some factors based on the observed covariates X contribute to selection of the treatment T , and only some to determining the outcomes Y . We model this by considering three underlying sources of { X, T, Y } and show that explicitly modeling these sources offers great insight to guide designing models that better handle selection bias. This paper is an attempt to conceptualize this line of thought and provide a path to explore it further. In this work, we propose an algorithm to (1) identify disentangled representations of the above-mentioned underlying factors from any given observational dataset D and (2) leverage this knowledge to reduce, as well as account for, the negative impact of selection bias on estimating the treatment effects from D. Our empirical results show that the proposed method (i) achieves state-of-the-art performance in both individual and population based evaluation measures and (ii) is highly robust under various data generating scenarios.

Summary:
   The authors consider the problem of estimating average treatment effects when observed X and treatment T causes Y. Observational data for X,T,Y is available and strong ignorability is assumed. Previous work (Shalit et al 2017) introduced learning a representation that is invariant in distribution across treatment and control groups and using that with treatment to estimate Y. However, authors point out that this representation being forced to be invariant still does not drive the selection bias to zero. A follow up work (Hassanpour and Greiner 2019) - corrects for this by using additional importance weighting that estimates the treatment selection bias given the learnt representation. However, the authors point out even this is not complete in general, as X could be determined by three latent factors, one that is the actual confounder between treatment and outcome and the other that affects only the outcome and the other that affects only the treatment. Therefore, the authors propose to have three representations and enforce independence between representation that solely determines outcome and the treatment and make other appropriate terms depend on the respective latent factors. This gives a modified objective with respect to these two prior works.
The authors implement optimize this joint system on synthetic and real world datasets. They show that they outperform all these previous works because of explicitly accounting for confounder, latent factors that solely control only outcome and treatment assignment respectively.
Pros:
  This paper directly addresses the problems due to Shalit 2017 that are still left open. The experimental results seems convincing on standard benchmarks.
I vote for accepting the paper. I don't have many concerns about this paper.
Cons:
  - I have one question for the authors - if T and Y(0),Y(1) are independent given X is assumed, then how are we sure that the composite representations (of the three latent factors) are going to necessarily satisfy ignorability provably ?? I guess this cannot be formally established. It would be great for the authors to comment on this.


Recent work has shown that quantization can help reduce the memory, compute, and energy demands of deep neural networks without significantly harming their quality. However, whether these prior techniques, applied traditionally to imagebased models, work with the same efficacy to the sequential decision making process in reinforcement learning remains an unanswered question. To address this void, we conduct the first comprehensive empirical study that quantifies the effects of quantization on various deep reinforcement learning policies with the intent to reduce their computational resource demands. We apply techniques such as post-training quantization and quantization aware training to a spectrum of reinforcement learning tasks (such as Pong, Breakout, BeamRider and more) and training algorithms (such as PPO, A2C, DDPG, and DQN). Across this spectrum of tasks and learning algorithms, we show that policies can be quantized to 6-8 bits of precision without loss of accuracy. We also show that certain tasks and reinforcement learning algorithms yield policies that are more difficult to quantize due to their effect of widening the models' distribution of weights and that quantization aware training consistently improves results over post-training quantization and oftentimes even over the full precision baseline. Additionally, we show that quantization aware training, like traditional regularizers, regularize models by increasing exploration during the training process. Finally, we demonstrate usefulness of quantization for reinforcement learning. We use half-precision training to train a Pong model 50% faster, and we deploy a quantized reinforcement learning based navigation policy to an embedded system, achieving an 18× speedup and a 4× reduction in memory usage over an unquantized policy.

This paper investigates the impact of using a reduced precision (i.e., quantization) in different deep reinforcement learning (DRL) algorithms. It shows that overall, reducing the precision of the neural network in DRL algorithms from 32 bits to 16 or 8 bits doesn't have much effect on the quality of the learned policy. It also shows how this quantization leads to a reduced memory cost and faster training and inference times.
I don't think this paper contributes with many novel results in the field, with most results being known or expected. The result that is interesting, in my opinion, is not properly explored.  The paper is well-written but it is a bit repetitive. It seems to me that the first 3 pages could be compressed in 1, as the same information is introduced over and over again.
With respect to the results being known, quantization is known to succeed in supervised learning tasks. In a deep reinforcement learning algorithm, when you apply post-training quantization in a deep reinforcement learning algorithm, mainly when that algorithm uses a value function (e.g., A2C or DQN), the problem is reduced to a regression problem. It is no different than a supervised learning problem. One has the original network’s prediction and they need to match that prediction. The complexities introduced in the reinforcement learning problem (bootstrapping, exploration, stability) don’t exist anymore as they arise during training. Thus, it doesn’t seem to me that these results are novel or surprising. In a sense it is neat to see that eventual errors do not compound, but that’s it. If I were to write this paper I would make this set of experiments much shorter just as a sanity check. One thing that I feel is missing is a notion of the impact of the quantization not in the rewards accumulated but in the policy/value function. How often does the quantized agent take a different action than the original agent, for example? Does it happen often but only when it doesn’t matter, or is it rare?
The quantization during training is potentially interesting. It was not properly explored though. I wonder if the quantization during training has a regularization effect, which is known to improve agent’s performance in reinforcement learning (e.g., Cobbe et al., 2018, Farebrother et al., 2018). Does the agent generalize better when using a network with fewer bits of precision? How does this change impact training? These are all questions that could potentially make the results in this paper novel (i.e., quantization as a form of regularization), but as it is now, the results are not that surprising.
Importantly, there are important details missing in the paper that make it hard for me to evaluate the validity of the results presented. Are the results reported over multiple runs? What is the version of the Atari games used, is it the one with stochasticity? How much variance do we have if we replicate this process over different networks that perform well? These are questions I would like to see answered because they also inform us about the impact of the proposed idea. For example, if by repeating this experiment multiple times one observe a high variance, it might mean that different models might be impacted in different ways.
The results in the “real-world” (Pong is not real-world) are not that surprising as well. Basically they show that if one uses a network with lower precision training and inference are faster, which, again, is not surprising.
There’s also an important distinction in the results that is not discussed in the paper: DQN estimates a value function while methods such as PPO directly estimate a policy. The reason DQN might have a wider distribution is exactly because it is estimating a different objective. These are important details that should be acknowledged and discussed in the paper. In my opinion, for this paper be relevant, it should have a very thorough evaluation of these different dimensions of reinforcement learning algorithms, with explicit discussions about it. Variance, the impact of quantization during learning, the distinction between parametrizing policies versus value functions, etc.
Finally, there are some aspects of the presentation of this paper that could also be improved. Aside from typos, below are some other comments on the presentation.
- There’s no such thing as Atari environment, it is either Arcade Learning Environment (Bellemare et al., 2013) or Atari games.
- I’d introduce/explain quantization in the beginning of the second paragraph of the Introduction for those not familiar with the term.
- No references are provided for the environments used. You should refer to Bellemare et al.’s (2013) work as well as Brockman et al.’s (2016).
- Is it really necessary to explain Fp16 quantization as it is done now, with even a picture of two bytes? I’d expect most readers are familiar with how numbers are represented in a computer.
- The equation for Uniform Affine Quantization is pretty much the same as the one in the Section Quantization Aware Training. All these “repetitions”, or discussions that are common-knowledge give the impression that the paper is trying to fill all the pages without necessarily having enough content.
- The references are not standardized (e.g., sometimes names are shortened, sometimes they are not) and the paper “Efficient inference engine on compressed deep neural network” is cited twice.
References:
Marc G. Bellemare, Yavar Naddaf, Joel Veness, Michael Bowling: The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res. 47: 253-279 (2013)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba: OpenAI Gym. CoRR abs/1606.01540 (2016)
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, John Schulman: Quantifying Generalization in Reinforcement Learning. CoRR abs/1812.02341 (2018)
Jesse Farebrother, Marlos C. Machado, Michael Bowling: Generalization and Regularization in DQN. CoRR abs/1810.00123 (2018)
------
>>> Update after rebuttal: I stand by my score after the rebuttal.
The rebuttal did acknowledge some points I made to me the paper took a gradient update towards the right direction. I don't think the paper is quite there yet though. It is repetitive, spending too much time with basic concepts, and it still ignores small details that matter (e.g., calling it Atari Arcade Learning). I strongly recommend the authors to follow my recommendations closely and then submit the paper again to a next conference. The discussion about generalization is potentially interesting, going beyond the regularization for exploration aspect. A better discussion about quantization during learning is also essential. The first three pages could probably be compressed by half.


Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of the VAE. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without having to force it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data points, we introduce an ex-post density estimation step that can be readily applied to the proposed framework as well as existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.

This paper propose an extension to deterministic autoencoders. Motivated from VAEs, the authors propose RAEs, which replace the noise injection in the encoders of VAEs with an explicit regularization term on the latent representations. As a result, the model becomes a deterministic autoencoder with a L_2 regularization on the latent representation z. To make the model generalize well, the authors also add a decoder regularization term L_REG. In addition, due to the encoder in RAE is deterministic, the authors propose several ex-post density estimation techniques for generating samples.
The idea of transferring the variational to deterministic autoencoders is interesting. Also, this paper is well-written and easy to understand. However, in my opinion, this paper needs to consider more cases for autoencoders and needs more rigorous empirical and theoretical study before it can be accepted. Details are as follow:
1. The RAEs are motivated from VAEs, or actually CV-VAEs as in this paper. More precisely, the authors focus on VAEs with a constant covariance Gaussian distribution as the variational distribution and a Gaussian distribution with the identity matrix as the covariance matrix as the model likelihood. However, there might be many other settings for VAEs. For example, the model likelihood can be a Gaussian distribution with non-constant covariance, or even some other distributions (e.g. Multinomial, Bernoulli, etc). Similarly, the variational distribution can be a Gaussian distribution with non-constant covariance, or even some more complicated distributions that do not follow the mean-field assumption. Any of these more complex models may not be easily transferred to the RAE models that are mentioned in this paper. Perhaps it is better if the authors can consider RAEs for some more general VAE settings.
2. Perhaps the authors needs more empirical study, especially on the gain of RAE over CV-VAE and AE.
a) As the motivated model (CV-VAE) and the most related model in the objective (AE), they are not appearing in the structured input experiment (Section 6.2). It will be great if they can be compared with in this experiment.
b) The authors did not show us clearly whether the performance gain of RAE over VAE, AE and CV-VAE is due to the regularization on z (the term L_z^RAE) or the decoder regularization (the term L_REG) in the experiments. In table 1, the authors only compare the standard RAE with RAE without decoder regularization, but did not compare with RAE without the regularization on z (i.e. equivalent to AE + decoder regularization) and CV-VAE + decoder regularization. The authors would like to show that the explicit regularization on z is better than injecting the noise, hence the decoder regularization term should appear also in the baseline methods. It is totally possible that perhaps AE + decoder regularization or CV-VAE + decoder regularization perform better than RAE.
c) The authors did not show how they tune the parameter \sigma for CV-VAE. Since the parameter \beta in the objective of RAE is tunable, for fair comparison, the authors needs to find the best \sigma for CV-VAE in order to get the conclusion that explicit regularization is better than CV-VAE.
d) Although the authors mention that the 3 regularization techniques perform similarly, from Table 1, it is still hard to decide which one should we use in practice in order to get a performance at least not too much worse compared to the baseline methods. RAE-GP and RAE-L2 perform not well on CelebA while RAE-SN perform not well on MNIST, compared to the baseline methods. We know that the best performance over the 3 methods is always comparable to or better than the baselines, but not none of the single methods do. It is better if the authors can provide more suggestions on the choice for decoder regularization for different datasets.
3. The authors provided a theoretical derivation for the objective L_RAE (Equation 11), but this is only for the L_GP regularization. Besides, this derivation (in Appendix B) has multiple technique issues. For example, in the constraints in Equation 12, the authors wrote ||D_\theta(z1) - D_\theta(z2)|| < epsilon for all z1, z2 ~ q_\phi(z | x), this is impossible for CV-VAE since this constraint requires D_theta() to be bounded while q_\phi(z | x) in CV-VAE has an unbounded domain. Moreover, in the part  (||D_\theta(z1) - D_\theta(z2)||_p=\nabla D_\theta(\tilde z)\cdot ||z_1-z_2||_p) of Equation 13, \nabla D_\theta(\tilde z) is a vector well the other two terms are scalars, which does not make sense. There are many other issues as well. Please go through the proof again and solve these issues.
Questions and additional feedback:
1. Can the authors provide more intuitions why do you think the explicit regularization works better compared to the noise injection? Can you provide a theoretical analysis on that?
2. Can the authors provide some additional experiments as mentioned above? Also, can the authors provide more details about how do they tune the parameters \beta and \lambda?
========================================================================================================
After the rebuttal:
Thanks the authors for the detailed response and the additional experiments. I agree that the additional experiment results help to support the claims from the authors, especially for the CV-VAE for the structured data experiments and the AE + L2 experiment. So I think now the authors have more facts to support that RAE is performing better compared to the baseline methods.
Therefore, I agree that after the revision, the proposed method RAE is supported better empirically. So I am changing my score from "weak reject" to "weak accept". But I still think the baseline CV-VAE + regularization is important for Table 1 and the technical issues in the theoretical analysis needs to be solved. Hope the authors can edit them in the later version.


Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning. In this paper, we propose ScoringAggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions with sparse reward and then plan on unseen tasks in zero-shot condition. The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning. Many previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve the similar goal with pure off-policy imperfect data. Instantiating our framework results in a generalizable policy to unseen tasks. Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.

The paper proposes a framework (Scoring-Aggregating-Planning (SAP)) for learning task-agnostic priors that allow generalization to new tasks without finetuning. The motivation for this is very clear - humans can perform much better than machines in zero-shot conditions because humans have learned priors about objects, semantics, physics, etc. This is achieved by learning a scoring function based on the final reward and a self-supervised learned dynamics model.
Overall, the paper is very clear and easy to follow.
The presented task is realistic and important, and the paper seems to address it in a reasonable approach.
However, the evaluation seems lacking to me - the evaluation convinced me that SAP works, but I am not convinced that it works better than existing approaches (see below), and especially did not convince me that it is better in the zero-shot test environment.
The (anonymized) website contains nice videos that support the submission.
Questions for the authors:
1. Page 3, 3rd paragraph of Section 3: the paper says that "The proposed formulation requires much less information and thus more realistic and feasible" - I agree that this is more realistic, but is it really more feasible? The requirement of much less information makes the proposed formulation much more sparse.
2. A basic assumption in the SAP framework is that a local region score is a sum of all the sub-regions. As phrased in the paper: "in the physical world, there is usually some level of rotational or transnational invariance". I'm not sure that this assumption makes sense neither in the Mario case or in other tasks, e.g., robotics. Doesn't it matter if you have a "turtle" right in front of you (which means that the turtle is going to hit you), or below you (which means that you are going hit the turtle)?
3. A question about the planning phase - page 5 says: "We select the action sequence that gives us the best-aggregated score and execute the first action". Do you select the entire sequence of actions in the new environment in advance? Can the agent observe the new state after every action, and decide on the next action based on the actual step that the action has reached, rather than on the state that was approximated in advance?
In other words - what happens if the first action in the new test environment yields an unexpected state, that was not predicted well by the dynamics model; does the agent continue on the initial planned trajectory (that ignores the "surprise"), or does it compute its next action based on the unexpected state?

4. Experiments: in Gridworld and Mario - are there any stronger baselines in the literature, or reductions of known baselines to the zero-shot scenario? Are the chosen "Human Priors", BC-random and BC-SAP just strawmen?
Since the main goal of this paper is the zero-shot task, what would convince me is a state-of-the-art model that does possibly *better than SAP on the training level*, but *worse than SAP in generalizing to the new level*. Additionally, are there other baselines that specifically address the zero-shot task in the literature?
Minor (did not impact score):
Page 2, 1st paragraph: "... we show that how an intelligent agent"...
Page 3, 3rd paragraph: "... in model-free RL problem" - missing an "a" or "problem*s*"?
Page 3, 3rd paragraph: ". Model based method ..." - missing an "a" as well?
Page 4, 1st paragraph:: "... utilizing the to get the ..."
Page 4, last row: missing a dot after the loss equation, before the word "In".
Page 7, Table 1: "BC-random" is called "BC-data" in the text. Aren't they the same thing?


Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps. Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly. In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data. In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner. On three real-world datasets, including one our newly collected NYCtraffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.

This paper proposes a Transformer-based model with cross-dimensional self-attention for multivariate time series imputation and forecasting. The authors consider time series with observations collected at different locations (L), for different measurements (M), and at different timestamps (T). The authors describe 4 different self-attention mechanisms based on how the three dimensions, and it turns out the proposed Decomposed approach achieves the best performance and has moderate model complexity among the four. Experiments on several traffic and air quality datasets show the superiority of the proposed model.
Overall the problem and the proposed model are well motivated. Handling time series data with missing values is quite important, and the authors design the novel cross-dimensional attention mechanism which is reasonable and performs well. Especially, the authors compare several recent RNN-based models. The proposed method outperforms these strong baselines in imputation and long-term forecasting tasks.
However, I do have a few questions and concerns.
It would be quite helpful to see the training time and model size comparisons with baselines, which is to validate the claim in the introduction that `replacing the conventional RNN-based models to speed up training`.
The proposed model treats all three dimensions equally, with direct attention of every two variables, and is independent with the order. Though effective and mathematically clean, I am not sure the temporal/spatial smoothness and dependencies in time series are properly modeled in this way -- as time series is not the same as embedded sequences in NLP. This may explain why the performance on short-term forecasting is relatively unsatisfying.
It seems that the proposed model is designed for missing completely at random (implied by the statement `... due to unexpected sensor damages or communication errors` from the introduction, and the experimental settings on adding missing values). Many missing variables in time series may be missing at random or even not at random.
About the experiments:
Two datasets of the three mentioned in the main paper have M=1, which degrade the proposed model from 3-dimensional to 2-dimensional.
Why forecasting experiment is conducted on METR-LA, while using the imputation for forecasting is conducted on a different dataset and without comparing other forecasting baselines?
What are the metrics used in Tables 6, 7, 9? Several metrics are used (RMSE, MSE, MAPE, MAE, MRE) while results on different datasets are shown in different but not all metrics. Is there any reason to cherry-pick metrics for different experiments?
In Table 4, the proposed method's results in RMSE is consistently better than shown in MAE compared with other baselines. Any explanations would be useful.
The overall idea is relatively easy to follow, while some detailed descriptions should be added or clarified.
When taking health-care data as an example of geo-tagged time series, could you explain or provide references?
Figure 2 only demonstrates 3 attention mechanisms, and the Shared should also be included.
S(i,j) is used in Section 3.1 without explicit formal definition.
Please clarify the sentence about \sigma below Equation (4).
Please refer to the section number explicitly in Supp if used. (E.g., on Pages 5 and 7.)
On which dataset and what settings are the results in Table 1 computed? The numbers are helpful, but it would be better if the results are computed based on the hyperparameters (e.g., T, L, M, d_V, etc).
Minor typos:
Page 3, Paragraph of RNN-based data imputation methods: `...indistinguishable. so that...`
Page 3, Section 3.1, `..where Then, ...`
Page 14, A' is used to denote the reshaped tensor, while \tilde is used in the main paper.
Page 16, `During testing,`


Turing complete computation and reasoning are often regarded as necessary precursors to general intelligence. There has been a significant body of work studying neural networks that mimic general computation, but these networks fail to generalize to data distributions that are outside of their training set. We study this problem through the lens of fundamental computer science problems: sorting and graph processing. We modify the masking mechanism of a transformer in order to allow them to implement rudimentary functions with strong generalization. We call this model the Neural Execution Engine, and show that it learns, through supervision, to numerically compute the basic subroutines comprising these algorithms with near perfect accuracy. Moreover, it retains this level of accuracy while generalizing to unseen data and long sequences outside of the training distribution.

This paper investigates an interesting problem of building a program execution engine with neural networks. The authors proposed a transformer-based model to learn basic subroutines, such as comparison, find min, and addition, and apply them in several standard algorithms, such as sorting and Dijkstra’s.
Pros:
1. The method achieves generalization towards longer sequences than the sequences in the training set in several algorithms.
2. The method represents numbers in binary form and the visualization shows that it learns embeddings from fixed-range integer numbers in a well-structured manner.
3. The learned NEE subroutines are tested in a variety of standard algorithms, such as multiple sorting algorithms and Dijkstra’s shortest path algorithm. The experiments further demonstrate that several NEE subroutines can be composed together in complex algorithms.
Cons:
1. NEE mostly focuses on learning low-level subroutines such as number comparison or addition. Therefore, it has to be used along with conventional subroutines, and cannot completely replace the full execution in complex algorithms, which have sophisticated control logic, such as if/else and for-loops. When the transformer model is used alone in the sorting task (Sec. 4.1), the performance degrades substantially as the sequence length gets longer.
2. Although the method achieved some degree of strong generalization, it lacks a formal way to verify the correctness of the learned subroutines, as opposed to prior work on program synthesis (Cai et al. 2017) that can prove the generalization of their model with recursion.
3. The method relies on detailed execution traces for supervised learning which can be costly to obtain.
Questions:
1. Confusing sentence: “Can we retain high attention resolution by restricting the model to only observe the first scenario repeatedly?” Can you elaborate on what you meant here?
2. From Figure 1, it seems that the model with dot product attention generalizes better in longer sequences than the one with all modifications. What’s the reason?
3. I would like to better understand the limitation of these learned NEE subroutines in long sequences. For instance, in Figure 8 and Figure 9, how would the model perform beyond the lengths of the sequences tested here? Would the performance maintain at 100% or decrease gradually as the sequences get even longer?
4. I am curious to know how this method could be extended to support more complex number systems, such as float numbers, and more complex data structures beyond sequences, such as binary trees and priority queues. I’d love to hear what the authors have to say about this.
5. I'd also like to know if the number embeddings learn in different algorithms would exhibit different structures (by examining the visualization of number embeddings learned in different tasks).
6. As NEE focuses on learning the basic subroutines while NPI aims to learn the high-level program executions, I think that it’d be very interesting to see how these two can combine their complementary strengths to build a complete neural-based execution engine.
Typos:
Select he node --> Select the node


In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image generation is split into 2 stages. In the first stage a generator network outputs 3D objects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to overcome them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.

I thank the authors for the rebuttal and the additional experiments. The additions do partially address my concerns, although not entirely. For instance, the experiments on non-face classes are very preliminary and it is unclear if they work at all (no other views shown). I hope the authors are right that the method will work on other classes after some tuning, but this is not demonstrated in the paper. Overall, I am quite in a borderline mode. I think the paper looks promising and after further improving the experimental evaluation it can become a great publication. But for now the experiments, especially the new ones, look somewhat incomplete and rushed, more suitable for a workshop paper. Therefore, I still lean towards rejection.
---
The paper proposes an approach to learning the 3D structure of images without explicit supervision. The proposed model is a Generative Adversarial Network (GAN) with an appropriate task-specific structure: instead of generating an image directly with a deep network, three intermediate outputs are generated first and then processed by a differentiable renderer. The three outputs are the 3D geometry of the object (represented by a mesh in this work), the texture of the object, and the background image. The final output of the model is produced by rendering the geometry with the texture and overlaying on top of the background. The whole system can be trained end-to-end with a standard GAN objective. The method is applied to the FFHQ dataset of face images, where it produces qualitatively reasonable results.
I am in the borderline mode about this paper. On one hand, I believe the task of unsupervised learning 3D from 2D is interesting and important, and the paper makes an interesting contribution in this direction. On the other hand, the experimental evaluation is quite limited: the results are purely qualitative, on a single dataset, and do not contain much analysis of the method. It would be great if the authors could add more experiments to the paper during the discussion phase.
More detailed comments:
Pros:
1) The paper is presented well, is easy to read. I like the detailed table with comparison to related works, and a good discussion of the limitations of the method and the tricks involved in making it work. I also like section 4 clearly discussing the assumptions of the work, although I think it could be shortened quite a bit.
2) The proposed method is reasonable and seems to work in practice, judging from the qualitative results.
cons:
1) The experiments are limited.
1a) There are no quantitative results. I understand it is non-trivial to evaluate the method on 3D reconstruction, although one could either train a network inverting the generator, or, perhaps simpler, apply a pre-trained image-to-3D network to the generated images. But at least some image quality measures (FID, IS) could be reported.
1b) The method is only trained on one dataset of faces. It would be great to apply the method to several other datasets as well, for instance, cars, bedrooms, animal faces, ShapeNet objects. This would showcase the generality of the approach. Otherwise, I am worried the method is fragile and only applies to very clean and simple data. Also, if the method is only applied to faces, it makes sense to mention faces in the title.
1c) It would be very helpful to have more analysis of the different variants of the method, ideally with quantitative results (again, at least some image quality results). Figure 3 goes in this direction, but it is very small and does not give a clear understanding of the relative performance of diferent variants.
2) A missing very relevant citation of HoloGAN by Nguyen-Phuoc et al.  [1]. It is not yet published, but has been on arXiv for some time. I am a bit unsure about the ICLR policy in this case (this page https://iclr.cc/Conferences/2019/Reviewer_Guidelines suggests that arXiv paper may be formally considered prior work, in which case it should be discussed in full detail), but at least a brief mention would definitely be good.
[1] HoloGAN: Unsupervised learning of 3D representations from natural images. Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang. arXiv 2019.


We propose precision gating (PG), an end-to-end trainable dual-precision quantization technique for deep neural networks. PG computes most features in a low precision and only a small proportion of important features in a higher precision. Precision gating is very lightweight and widely applicable to many neural network architectures. Experimental results show that precision gating can greatly reduce the average bitwidth of computations in both CNNs and LSTMs with negligible accuracy loss. Compared to state-of-the-art counterparts, PG achieves the same or better accuracy with 2.4× less compute on ImageNet. Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.8× computational cost reduction on LSTM on the Penn Tree Bank dataset. Precision gating has the potential to greatly reduce the execution costs of DNNs on both commodity and dedicated hardware accelerators. We implement the sampled dense-dense matrix multiplication kernel in PG on CPU, which achieves up to 8.3× wall clock speedup over the dense baseline.

This paper introduces Precision Gating, a novel mechanism to quantize neural network activations to reduce the average bitwidth, resulting in networks with fewer bitwise operations. The idea is to have a learnable threshold Delta that determines if an output activation should be computed in high or low precision, determined by the most significant bits of the value. Assuming that high activations are more important, these are computed at higher precision.
I agree that the following three key contributions listed in the paper are (slightly re-formulated):
1. Introducing Precision Gating (PG), the first end-to-end trainable method that enables dual-precision execution of DNNs and is applicable to a wide variety of network architectures.
2. Precision gating enables DNN computation with a better average bitwidth to accuracy tradeoff than other state-ofthe-
art quantization methods. Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware.
3. Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which reduces the computational cost for both passes.
These contributions are novel and experimental evidence is provided for multiple networks and datasets. The paper is well-written and provides insightful figures to showcase the strengths of the present method. Related work is adequately cited. The paper does not contain much theory, but wherever possible equations are provided to illustrate in detail how the method works.
Experimental results are shown for the datasets CIFAR-10 with ResNet-18 and ShiftNet-20, and ImageNet with ShuffleNet V2 0.5x. On both datasets, PG outperforms uniform quantization, PACT, Fix-Threshold and SeerNet in terms of top-1 accuracy and average bitwidth.
What I am missing is information about the variability of results, since there are no error bars. Are the results averaged over multiple trials (if yes how many?), and is there a difference in variance between the methods? I realize that adding standard deviations to all results in the tables might be infeasible, but a qualitative statement would be interesting. In particular, the random initialization of the hb bits could play a bigger role than lb bits.
The two variants of PG, with and without sparse backpropagation are also investigated, showing that sparse backpropagation leads to more sparsity. To show that the resulting lower average bitwidth gained with PG leads to increased performance, the authors implement it in Python (running on CPU) and measure the wall clock time to execute the ResNet-18 model. Speedups $> 1$ are shown for every layer when using PG. Evidence from other papers is cited to argue that similar speedups are expected on GPUs.
Even though at the moment it is unclear to me how statistically significant the results are, and I strongly recommend commenting on this in the paper, I think the idea of PG and the demonstrated benefits make the paper interesting enough to be accepted at ICLR.
I also have a few questions that I could not get completely from the paper:
1. I am a bit confused by what you call features. Fig. 2 shows by example how the method works for an input $I$. Is $I$, a single number, i.e. a single entry of your input vector, or do you mean the complete input vector?
2. Could you give a bit more insight, how you tuned your hyperparameters, especially $\delta$ and $\sigma$?
3. What exactly does e.g. $\delta=-1$ mean? The network ideally should compute at high precision, when the result when only considering the most significant bits is above -1?
From a hardware point of view, the paper focuses on GPU implementations. I would have hoped for a discussion of suitable custom hardware that could support PG most efficiently.
Minor comments that I would be interested in but did not influence my score
- It seems to me that on the top-left image of Fig. 3, one blue circle (the second largest) is too much? First part shows 8 dots, middle and right only seven?
- Can you please cite a source that DNN activations have outliers (Sec. 3.4)?
- You could also define e.g. one $\delta$ and $\Delta$ per layer, couldn't you? Would be interesting to see if  e.g. thinning out the precision over depth is possible / has advantages.


We show that for any convex differentiable loss, a deep linear network has no spurious local minima as long as it is true for the two layer case. This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately implies the powerful result by Kawaguchi (2016). Further, with the recent work by Zhou & Liang (2018), we can remove all the assumptions in (Kawaguchi, 2016). This property holds for more general "multi-tower" linear networks too. Our proof builds on the work in (Laurent & von Brecht, 2018) and develops a new perturbation argument to show that any spurious local minimum must have full rank, a structural property which can be useful more generally.

The motivation of this paper is training deep neural network seems to not suffer from local minima, and it tries to explain this phenomenon by showing that all local minima of deep neural network is global minima. The paper shows that for any convex differentiable loss function, a deep linear neural network has no so called spurious local minima, which to be specific, are local minima that are not global minima, as long as it is true for two-layer Neural Network. The motivation is that combining with existing result that no spurious local minima exists for quadratic loss in two-layer Neural Network, this relation connecting between two-layer and deeper linear neural network immediately implies an existing result that all local minima are global minima, removing all assumptions. The result also holds for general “multi-tower” linear networks.
Overall, this paper could be an improvement of existing results. It is well written and the proof step is clear in general. However, there’re some weakness need clarifications on the results, especially on the novelty. Given reasonable clarifications in response, I would be willing to change my score.
For novelty, it is unclear if the results from Lemma 1 to Theorem 1 and 2 are both being stated as novel results. The first part of proof of Theorem 1 is obvious and straightforward, and the other direction has been used before for multiple times as claimed in the paper, what is your novelty exactly here? For the key technical claim of Lemma 1, it looks like this perturbation technique already exists in (Laurent & Brecht, 2018), why do you claim it as a novel argument?
Besides novelty, there are also some other unclear pieces in this paper needs clarification:
1)	Is the main result which is “no spurious local minima for deep neural network” holds for any differentiable convex loss other than quadratic loss? How will Theorem 1 help us understand the mystery of neural network?
2)	How does the result help us understand non-linear deep neural network, which is commonly use in practice?
3)	The paper should give some explanations about why the results help training neural networks.


For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photorealistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-rasterscan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks. PyTorch code is provided at https://drive.google.com/file/d/1UFQ7yS6Lobg ZGAxIwXS40vzWO8zwySV/view.

This paper proposes a combined architecture for image-text modeling. Though the proposed architecture is extremely detailed, the authors explain clearly the overarching concepts and methods used, within limited space. The experimental results are extremely strong, especially on sub-domains where conditional generative models have historically struggled such as images with angular, global features - often mechanical or human constructed objects. "Computers" and "cars" images in Figure 2 show this quite clearly. The model also functions for tagging and annotating images - performing well compared to models designed *only* for this task.
The authors have done a commendable job adding detail, further analysis, and experiments in the appendix of the paper. Combined with the included code release, this paper should be of interest to many.
My chief criticisms come for the density of the paper - while it is difficult to dilute such a complex model to 8 pages, and the included appendix clarifies many questions in the text body, it would be worth further passes through the main paper with a specific focus on clarity and brevity, to aid in the accessibility of this work.
As usual, more experiments are always welcome, and given the strengths of GAN based generators for faces a text based facial image generator could have been a great addition. The existing experiments are more than sufficient for proof-of-concept though.
Finally, though this version of the paper includes code directly in a google drive link it would be ideal for the final version to reference a github code link - again to aid access to interested individuals. Being able to read the code online, without downloading and opening locally can be nice, along with other benefits from open source release. However the authors should release the code however they see fit, this is more of a personal preference on the part of this reviewer.
To improve my score, the primary changes would be more editing and re-writing, focused on clarity and brevity of the text in the core paper.