Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? How to set the range for learning rate tuning? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the Conjugate Kernel, CK, (also called the Neural Network-Gaussian Process Kernel), and the Neural Tangent Kernel, NTK. Roughly, the CK and the NTK tell us respectively "what a network looks like at initialization" and "what a network looks like during and after training." Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks. By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks. We believe the computational tools we develop here for analyzing the spectra of CK and NTK serve as a solid foundation for future studies of deep neural networks. We have open-sourced the code for it and for generating the plots in this paper at github.com/jxVmnLgedVwv6mNcGCBy/NNspectra.

Updates:
Thanks for the updates.
I find the new theoretical results interesting and potentially useful,  which shows, in the large $d$ setting, spectrums of CKs/NTKs  for boolean cube, sphere and isotropic Gaussian are closed to each other in some sense. Thus, I raise my score to weakly accepted but lower down my confidence level since I am not that familiar with Boolean cube literature.
------------------------------------------------------
The study of extremely over-parameterized networks (i.e. infinitely width networks) has become one of the most active research directions in theory deep learning. The key objects in understanding such networks are the conjugate kernel [1, 2] (CK defined in the paper) and the Neural tangent kernels [3] (NTK). The CK characterizes how the network looks like at initialization (connection to Gaussian processes as well) and the NTK is very useful to characterize the gradient descent training dynamics of large width networks in the kernel regime. Understanding properties of such kernels, in particular, their spectra distribution and eigenspace, could be potentially an important step towards a finer-gained understanding of generalization in neural networks.
The main contribution of this paper is the development of the spectral theory of CK and NTK on boolean cube (similar or weaker results on uniform distribution in spheres and Gaussian distribution in R^n). More precisely, the authors show that, over the space of boolean cube, the CK/NTK could be diagonalized using the Fourier basis and the eigenvalues depend only on the frequency (i.e. the degree of the monomials); Thm 3.1. The authors also develop some computation tools to compute the spectra; Lemma 3.2.   Using the tools developed in this paper, the authors are able to clarify some of the interesting observations found by other researchers. Most noticeably, the authors show that the observation in [4] 'neural network is biased towards simple functions' is NOT universal. Whether this statement is correct or not depends heavily on the choice of activation function (e.g. Relu v.s. Erf) and hyper-parameters (e.g. weight variance, depths).  There are also some other interesting empirical findings: the optimal depth of a neural network depends on the complexity (i.e. degree in the boolean cube setting) of the function to learn, CK (i.e. training only the last layer) tends to be more useful for learning less complex functions, etc.
Overall, this is a nice paper. I am leaning for a weakly accept.
[1] Amit Daniely, Roy Frostig, and Yoram Singer. Toward Deeper Understanding of Neural Networks:
The Power of Initialization and a Dual View on Expressivity. arXiv:1602.05897 [cs, stat], February
2016.
[2] Jaehoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha
Sohl-dickstein. Deep Neural Networks as Gaussian Processes. In International Conference on
Learning Representations, 2018.
[3] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and
Generalization in Neural Networks. arXiv:1806.07572 [cs, math, stat], June 2018. 00000
[4] Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because
the parameter-function map is biased towards simple functions. arXiv:1805.08522 [cs, stat], May
2018.


Graph representation learning for hypergraphs can be used to extract patterns among higher-order interactions that are critically important in many real world problems. Current approaches designed for hypergraphs, however, are unable to handle different types of hypergraphs and are typically not generic for various learning tasks. Indeed, models that can predict variable-sized heterogeneous hyperedges have not been available. Here we develop a new self-attention based graph neural network called Hyper-SAGNN applicable to homogeneous and heterogeneous hypergraphs with variable hyperedge sizes. We perform extensive evaluations on multiple datasets, including four benchmark network datasets and two single-cell Hi-C datasets in genomics. We demonstrate that Hyper-SAGNN significantly outperforms the state-of-the-art methods on traditional tasks while also achieving great performance on a new task called outsider identification.
Hyper-SAGNN will be useful for graph representation learning to uncover complex higher-order interactions in different applications.

Summary:
This work is tackling the problem of predicting variable-sized heterogeneous hyperedges in hypergraphs. The authors do so by introducing a multi-head attention mechanism that takes in a set of node features and predicts the probability of this set of nodes forming a hyperedge.
- Quality and clarity:
The quality of the paper is good, the writing is clear and the reasoning is relatively easy to follow. The problem the authors are trying to tackle is also clearly defined. The motivation of the dynamic and static embedding choice could be explained better. The intuition that the pseudo-distance tells us how well the static embedding
of a node can be approximated by the features of the neighbour within the tuple somewhat makes sense, but I also have the impression that this is just one possible interpretation and what happens in practice could actually be quite different. A more formal treatment and analysis of this intuition would be very helpful.
- Originality:
The main contribution here is to propose a more flexible formulation to improve over the fixed MLP used by DHNE. This in itself is a good idea, but overall is an incremental improvement.
- Significance:
The results are convincing, DHNE is clearly outperformed on the benchmarks. Depending on the ease of training the model compared to DHNE, this work might be a useful contribution for the ICLR community.
Other points
The related work section should be expanded by discussing relationships to:
Bai, Song, Feihu Zhang, and Philip HS Torr. "Hypergraph Convolution and Hypergraph Attention." arXiv preprint arXiv:1901.08150 (2019).
Feng, Yifan, et al. "Hypergraph neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
------------------
Post-rebuttal
------------------
I would like to thank the authors for their in-depth response. I found the additional analysis on the dynamic node embeddings insightful and it is reassuring that it confirms the claims. Overall I think the rebuttal addresses my major concerns and I will raise my score.


Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized floating point representations have received recent interest due to their hardware efficiency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy. We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arithmetic. We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss. The method makes no assumptions regarding the underlying parameter distributions. We evaluate our method on pre-trained image classification models on the CIFAR-10 and ImageNet datasets. For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of significance from the Monte Carlo trials. Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects.

Summary:
The paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network.
Decision:
Overall, this is an interesting paper with interesting results. However, I think there is considerable room for improvement, and that more details are needed in order to assess the significance of the results, as I detail in the rest of my review. For these reasons, I recommend weak reject for now, but I encourage the authors to continue working on improving the paper and to provide more details in the updated version.
Contribution:
The paper considers an important problem, that of quantizing the weights and activations of a neural network in order to reduce computational and memory cost, while maintaining the machine-learning performance as high as possible.
In my opinion, the main contribution of the paper is the experimental findings, and in particular that the sensitivity of the training loss with respect to the precision of the weights and activations correlates with the accuracy of the network. It seems to me that these results may relate to work on Bayesian neural networks, sharp vs flat minima, and minimum-description length approaches to variational inference. Work on these areas has also shown that sensitivity of the training loss with respect to the precision of the weights (which intuitively happens when the network is at a "sharp" local minimum vs a "flat" one) is related to poor generalization performance, and vice versa. I would encourage the authors to explore the potential relationship of their work with these areas, and possibly discuss them in an updated version of the paper.
Originality:
The paper describes a method for assessing the sensitivity of a neural network with respect to the precision of the weights and activations. The method is a straightforward application of Monte Carlo Arithmetic (MCA) to neural networks. I believe that the application of MCA to neural networks for this particular purpose is novel, and that the results are original. However, the introduction of the paper gives the impression that the proposed method is brand new, and even uses the acronym MCA to refer to the proposed method, which can be confusing to readers. I would suggest to the authors to rewrite the introduction so as to reflect more accurately that the contribution is not a brand-new method, but rather the application of an existing method (MCA) in a novel way.
Writing quality:
The paper is generally easy to read, but there is considerable room for improvement. There are mistakes, and often the writing is sloppy and imprecise. I give some more specific suggestions on what to improve later on.
Technical quality:
The method is well motivated and the experiments seem reasonable. However, there is very little detail on the experiments, which makes it hard to assess their correctness/significance. I would suggest to the authors to rewrite the experimental section with full detail, or put more details in an appendix. In particular:
- Is each Monte Carlo trial done on the same batch of training images or a different one? If different, how are the trials averaged, and does that mean that the standard deviation over trials also includes a contribution due to different batches?
- In sections 5.1 and 5.2, how were the results for different t combined/aggregated? Did you use linear-regression analysis as in section 5.3?
- When you say "accuracy", do you mean accuracy on the training set, validation set, or test set? This is particularly important for assessing the significance of the results, and is something that is currently missing from the description of the experiments.
- How was the quantization of the neural networks performed? It would be good to explain this at least on a high level, in addition to citing Wang et al., (2018).
- In section 5.2, how was the model selection for each method performed exactly? In the baseline method, was the model to be quantized selected based on validation performance before quantization or after quantization?
Specific suggestions for improvement:
The citation format, i.e. (Smith et al., (2019)), is unusual and uses unnecessarily many parentheses. Use \citep for (Smith et al., 2019), and \citet for Smith et al. (2019).
The illustration of fig. 1 is not fully convincing as a motivation for floating-point arithmetic. Even though it makes the case that Float(7, 7) is more efficient than Float(8, 8) and Float (9, 9), the comparison between Float(7, 7) and Fixed(12, 12) is hard to interpret, as we can't conclude whether the efficiency gain is due to reducing the number of bits or to switching from fixed-point to floating-point arithmetic. A more convincing illustration would compare fixed-point with floating-point arithmetic using the same number of bits.
It would be better if fig. 1 were 2D, as 3D doesn't add anything but makes it harder to compare sizes visually.
Please avoid exaggerations, such as "exquisitely sensitive" or "extremely sensitive", when "sensitive" would suffice.
Section 2 is grammatically sloppy:
- arihtmetic --> arithmetic
- Last line of page 2 seems to be missing a verb.
- this has lead --> this has led
The related-work section is too short and in many cases it doesn't explain what previous work has actually done. For example, "rounding of inexact values to their nearest FP approximation has been studied in several publications" is vague: what exactly these publication have done? This lack of detail makes it hard to assess the originality of the current paper, and how it differs from existing work.
Section 3 is often unclear with imprecise mathematical notation:
- "e is the base-2 exponent of x in binary floating point arithmetic": surely, the exponent is represented as an integer?
- (bs, be1, be2, ..., bex, bm1, bm2, ..., bmx) is sloppy, as it indicates that the indices run from 1 to x.
- Bx = sx + ex + mx  is also sloppy; what is meant here is the number of bits to represent sx, ex, mx and not the values themselves.
- F(x) = x(1 + θ), shouldn't θ be δ?
- "which is typically the cause of horrific numerical inaccuracy from numerical analysis literature", the phrase "from numerical analysis literature" doesn't make much sense here.
- In eq. (4), substituting the expression for x from eq. (1) doesn't yield the same result.
- "The number of trials is an important consideration because [...] it can produce adverse effects on results". What is meant by "adverse effects"? Do you mean that with few trials Monte Carlo doesn't give accurate results? Please be more specific.
- "we can determine the expected number of significant binary digits available from a p-digit FP system as p ≥ −log2(δ)". I'm unable to follow this statement, please explain further. Also, from applying logs to δ ≤ 2^{−p} one gets an inequality that doesn't match the one in p ≥ −log2(δ).
- "The relative error of an MCA operation is, for virtual precision t, is δ ≤ 2^-t", "is" is used twice here.
- "the expected number of significant binary digits in a t-digit MCA operations is at least t", operations --> operation. Also, shouldn't it be at most t, otherwise K becomes negative?
-  is discussed in the section --> is discussed in the next section.
Some mistakes in section 4.1:
- y = (x; w) --> y = f(x; w)
- Eq. (8) is sloppy, it uses X for both the set and its size. Use |X| or something similar for the size.
- In the caption of fig. 2, baes --> base
I'm not convinced by the second bullet point in section 4.1, that the averaging over many images used to obtain the accuracy is the reason why MCA doesn't work well. Surely, the training loss (cross entropy) is also an average over many images? To me it would seem more plausible that the main reason MCA works with training loss but not accuracy is because accuracy is discrete, whereas training loss is continuous.
Fig. 3 would be much easier to read if the axes were labelled, and if the axes had the same range (so that different plots can be compared visually).
Fig. 5 would be easier to read if the networks were sorted with respect to K.
In section 5, CIFAR-10 is sometimes written as CIFAR10.
Appendix A is empty, so it should be removed.


Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective to investigate if they can deliver on their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs. Our theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice in the literature, our results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.

Update: I thank the authors' for their response, and have read the other reviews.
This paper demonstrates some theoretical and practical limitations on the use of likelihood based generative models for detecting adversarial examples. They construct a simple counterexample showing that there are adversarial examples for an arbitrarily accurate model (as measured by KL) that are not detectable by diminished likelihood of the model (as the dimension increases). Extending the work of Gilmer et al, this proves that there can be no general robustness guarantee for conditional generative models (Bayes classifiers). They provide compelling empirical evidence that while conditional normalizing flows trained on MNIST can be effective in detecting and defending adversarial attacks, these models trained on CIFAR10 are not. Surprisingly, it is shown that linear interpolations between images of different classes yield higher likelihoods for the CIFAR10 models and that class has little impact on model likelihoods. This goes some way in explaining why the detection is not effective on CIFAR, but questions still remain.
The paper makes fairly modest claims, but does a good job at demonstrating them and shedding some light on the issue. The experiments are thorough and fit into a growing body of evidence that the likelihoods of normalizing flows and other image based likelihood models may not be that informative or well calibrated, where past work has focused on out-of-distribution detection. My only major complaint with the paper is that it is not clear to what extent the theoretical and practical problems are related. As mentioned in the paper, the counterexample construction depends on the geometry of the data rather than the learning model. It could be that for both the MNIST and CIFAR10 datasets, the geometry is such that robustness garauntees are possible, and that the discrepancy in detection and interpolation arises because the normalizing flow has modeled the MNIST distribution much better than the CIFAR10 distribution. In this case we might hope that using conditional likelihood models for adversarial detection can be made effective, but that effort needs to be placed into improving the modeling capability. It's not obvious how to probe this distinction, but it would be good if this was given some thought in the paper. Also it would be good to see the attack detection numbers on BG-MNIST.
Comments:
Difficulty in training conditional generative models:
I believe in the two papers you cite the models do not use the label as input, but rather there is a separate model for each class? The overfitting is likely why the models had slightly lower conditional likelihood. As an aside, there are a couple of other examples of conditional normalizing flow models on images that use a mixture of Gaussians in the latent space [1], [2].
eq. 4: In the paper it is said that the second term in eq 4 is at most log(C), because the uniform distribution would have this value and that therefore this is negligibly small in comparison to the other term. Why exactly is this the case, couldn't the data entropy term be smaller in principle even if it's larger in practice? Or is the argument that the data entropy term scales with the dimensionality, but the label term does not leading to an imbalance? This could use some clarification.
A3: What is meant by ‘While the ground-truth likelihoods for the padded and un-padded datapoints are the same due to independence of the uniform noise and unit density of the noise’ in the appendix section A3? Wouldn't the ground truth negative log likelihoods would increase by the entropy of the uniform noise? Also, then in the bits per dimension calculations is the dimension the number of unpadded dimensions or the padded dimensions?
[1] Izmailov, Pavel, et al. "Semi-Supervised Learning with Normalizing Flows." Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.
[2] Atanov, Andrei, et al. "Semi-Conditional Normalizing Flows for Semi-Supervised Learning." Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.


Existing local explanation methods provide an explanation for each decision of black-box classifiers, in the form of relevance scores of features according to their contributions. To obtain satisfying explainability, many methods introduce ad hoc constraints into the classification loss to regularize these relevance scores. In this paper, we discuss some shortcomings of these methods and address them by a simple but effective mask predictor. Specifically, we present the concept of distribution controllers and integrate it with a neural network to directly guide the distribution of relevance scores. Then we introduce the classification loss to optimize the proposed predictor. The benefit of this strategy is to enable discriminative scores over supporting features, and facilitate the setting of involved hyperparameters. The experimental results demonstrate that the proposed method also outperforms others in terms of faithfulness and explainability. The code is available at https://github.com/iclrlocal/.

Paper Summary:
This paper proposed a method to produce instance-wise saliency map for image classification tasks. The proposed method develop a U-net-based generator for saliency mask, where the important modifications are (1) a skewness-inducing activation function for mask generation (i.e. controller), which is either a ReLU function or a scaled sigmoid function raised to a certain power. Authors argue that the proposed activation function leads to output saliency score to have right-skew distribution, which leads to more distinguished saliency map. (2) a smoothness-inducing mechanism where the saliency mask is generated at coarser scale then up-sampled with a bilinear operation so the generated map is smooth. Finally, the training is performed by minimizing solely on the cross-entropy loss with respect to the original model predictions, which is expected to improve faithfulness in mimicking the original black-box classifiers.
I'm leaning toward rejecting this paper in its current form. While I think this paper proposed an interesting strategy in improving the faithfulness of the explainer (i.e. training on the cross-entropy loss with respect to original classifier), the rest of the two modification either already exists in the literature (generate mapping at coarser resolution is an idea from Dabkowski & Gal (2017) / Du (2018) as pointed out by the author) or suffers potential technical issues that can benefit from further methodology improvement/empirical justification (please see Major Comments). Furthermore, empirical experiments seem to suggest that the proposed method sometimes fail to produce more smooth and more discriminative maps compared to its MGnet baseline (e.g. Figure 5, top right row), calling into the question of whether training solely on a cross-entropy loss is enough to ensure the generated saliency map is of high-quality. I still like the overall idea of this paper, and I think this work can be made more rigorous and informative by including a careful ablation study about the loss/gain of different regularization terms and loss functions (e.g. the effect of TV(M) term on smoothness, the effect of f(\phi(I, 1-M)) term on discriminativeness, and the trade-off between cross-entropy v.s. the original negative log likelihood loss as used in Dabkowski & Gal (2017)).
Major Comments:
(1) Custom Transformer
The derivation of the new activation function assumes that the distribution of the hidden-layer output $m^g_{ij}$ is strictly Gaussian (i.e. unimodal, symmetric, and with sufficiently light tail), and author proposed to build this Gaussian distribution from raw hidden-unit output through standardization (Section 3.2.2). However, this approach may not be valid in practice. This is because there is no guarantee that the distribution of $m^g_{ij}$ is strictly unimodal and symmetric. If the output is skewed/multimodal in anyway, the normalization operation won't produce a Gaussian distribution, since a linear transformation (i.e. normalization) of a skewed distribution is still a skewed distribution. As a result, I am worried that the hyperparameter derived in Section 3.2.3 may not always lead to right-skewed distribution as expected. If author would like to ensure the outcome distribution is right-skewed, it may be good to tune these activation function parameters for each instance so the empirical distribution of $m^d_{ij}$ satisfy certain skewness criteria (e.g., maybe tune the parameter in a way such that the pearson's coefficient of skewness of the empirical distribution is sufficiently small).
In addition, I find the author's claim about the benefit of right-skewed distribution (i.e. producing more discriminative maps) not yet well supported by the empirical evidence. For example, in Figure 4 the produced maps in the first three rows are relatively fuzzy, and the difference between the irrelevant features and the important features does not seem to be very high. The situation is only improved after the instance-wise finetuning. I think there are two possible reasons, either (a) the right-skewness of the actual distribution of the saliency map score in those images were not guaranteed (as mentioned earlier, I believe the current approach cannot guarantee the pre-activation units are Gaussian distributed, therefore the actual skewness of the post-activation values may vary depending on the input). (b) the right-skewness in the outcome distribution alone is not enough to guarantee good discrimininative behavior.
To clarify this (and provide more evidence for author's claim that skewness <-> better discrimination), author can consider visualizing the saliency map with varying degree of skewness (you can measure the skewness of the empirical distribution of the saliency-map score using pearson's coefficient of skewness), and show that the "discriminativeness" and skewness of the empirical distribution are correlated.
(2) Smoothness of the produced map
In the original MGnet work, the authors used two mechanisms to guarantee smoothness of the saliency map: (1) generate map at coarser level and (2) TV penalty on loss function. In this work author used only (1). As a result, the produced map in some cases (e.g. Figure 5, top right row for BF After, also Figure 7, left fifth row for BF after) appear jagged and not smooth. I'm wondering if adding TV penalty back would help somewhat. Again, a careful ablation study would be beneficial in clarifying this.
Minor Comments
(1) There's some minor technical issue in the description for Equation (6). Assuming $z_{ij}$ is Gaussian, it is $\eta$ AND sigmoid() COMBINED to "approach the uniform distribution", it is important to mention the sigmoid function since the uniform transformation used here is based on the Probability Integral Transform,   (i.e. if a random variable $z \sim \Phi(.)$ follows Gaussian distribution with CDF $\Phi(.)$, then $\Phi(z) \sim Unif(0, 1)$, i.e. you are transforming a Gaussian random variable using Gaussian CDF to get uniform distribution), and equation (6) is using $\eta$ AND sigmoid() combined together to approximate this Gaussian CDF [1].
(2) Equation (14) on page 13. There should be an $z^2$ term within the exponential function.
Reference:
[1] Gary R.Waissi, Donald F.Rossin (1996) A sigmoid approximation of the standard normal integral.


Recent work in graph neural networks (GNNs) has led to improvements in molecular activity and property prediction tasks. Unfortunately, GNNs often fail to capture the relative importance of interactions between molecular substructures, in part due to the absence of efficient intermediate pooling steps. To address these issues, we propose LaPool (Laplacian Pooling), a novel, data-driven, and interpretable hierarchical graph pooling method that takes into account both node features and graph structure to improve molecular understanding. We benchmark LaPool and show that it not only outperforms recent GNNs on molecular graph understanding and prediction tasks but also remains highly competitive on other graph types. We then demonstrate the improved interpretability achieved with LaPool using both qualitative and quantitative assessments, highlighting its potential applications in drug discovery.

This paper proposes a graph down-sampling component named LaPool. It uses the graph signal to dynamically select some "important" centroids and learn a sparse assignment matrix for clustering the remaining nodes.
This paper should be rejected due to the following reasons.
1. Strange new results compared with its previous version
The paper has a previous arxiv version. While the method does not change, the performance in this current submission has dramatic change compared with previous version: the proposed model seems much improved, while some important baselines (that outperform the proposed model) only have 50% of it previous performance in this submission.
For example, for Table 1 in the current version, the proposed model reached almost perfect scores in both F1 and AUC. However in previous version for the same setting and experiment (in Table 2 and 3 of its arxiv version), the performance are much lower especially the Structural alert prediction results.  Their previous results, which show the model does not perform better than DiffPool .
For the structural alert prediction results on DiffPool, below I copied and pasted the results from the arxiv version

Table 3: Structural alert prediction results
------------------------------------------------------------------------------------------------------------
                                             Tox21                   |               ChEMBL
                       F1-macro F1-micro ROC-AUC F1-macro F1-micro ROC-AUC
------------------------------------------------------------------------------------------------------------
GIN                     78.9        68.3             72.6           93.6         76.7          59.2
DiffPool              79.2        68.0             75.6           94.5        83.3          59.3
Graph U-net      71.1        47.6             67.9           92.9        68.1          59.3
LaPooldistance 80.6        74.2             73.5           95.2        81.3          59.5
LaPoolunreg     81.3        72.8             74.1           94.1        75.8          58.9
LaPool3hop       79.1        71.6             74.8           93.8        75.0          59.1
------------------------------------------------------------------------------------------------------------
One example:
(1) in the ICLR version, The F1 score for DiffPool is only 48.638 ± 9.916 on ChEMBL data (about 50% of its previous level) but F1 for the proposed method is improved. Why is that?
(2) same as baseline  Graph U-net , the ICLR version reports F1 37.585 ± 2.978, why is that?
(3) same as GIN, in the ICLR version, F1 is only 31.759 ± 3.728, less than 50% of its previous level in arxiv version.
The author needs to justify this dramatic change.
2. Although LaPool can dynamically select centroids, for a dense graph such as a complete graph, only one centroid will remain there since there is only one node which has larger signal variation than all its neighbors, as shown in Eq. 4. This consequently hurts the model performance on more dense graphs. That may be why in Table 2, on dense data "DD", LaPool performs much worse than baseline DiffPool. Also on another dense data "FRANKENSTEIN", LaPool does not performs significantly better than DiffPool.

3, the evaluation of interpretability is not convincing. This paper considers  "interpretability as the degree to which a human (in this context, a medicinal chemist) can understand the cause of the model’s decision". Therefore, the conclusion that this model is more interpretable is based on only one person's subjective judgement.  Even so, from the scores the model does not outperform baseline "GIN" that much.


Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from both demonstrations and trial-anderror experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on a set of challenging, vision-based control tasks.

The paper introduces a meta-learning approach that can learn from demonstrations and subsequent reinforcement learning trials. The approach works in two phases. First, multiple demonstrations are collected for each task and a meta-learning policy is obtained by embedding the demonstrations of a single task in a context vector that is given to the actor policy.  Subsequently, the meta-policy is used to collect trajectories which are also evaluated with a (sparse) reward. These trajectories, along with the demonstrations, are again used by a second meta-learning policy (with a concatenated context from demonstrations and trajectories) to obtain the final policy. The algorithm is tested on a simple point reaching task as well as on a more complex 3d physics simulation that contains pick-and-place tasks, pushing tasks and button pressing.
The approach seems really interesting and I think combining demonstrations with reinforcement learning for meta learning is a very promising approach. This is also underlined by the experimental results presented in the paper. The only concern I have is the presentation of the paper. The algorithmic description is very short and many things are left unclear. I would also like to see a bit more ablation studies in the paper. More comments below:
- Just from the text it is not clear how the meta-learning actually works and you have to study the figures in detail to understand that a context is extracted from the demonstrations as well as the RL trajectories. The context should be better introduced (even though this has been shown already in different papers, I think it is a good strategy to write the
in a self-contained way).
- It is unclear to me how the reward signal is used to learn the meta-policy is phase 2. I understand that it is concatenated to the embedding before forming the context (again this is not really described in text but only in the figure), but it is quite unclear to me how the reward is used as optimality criteria. How do you reinforce good trajectories and decrease the influence of poor trajectories? This is maybe one of the most important parts of the paper and needs to be described in much more detail.
- The reward functions are not properly explained in the paper, even not for the toy task (reaching). This should at least be done in the appendix.
- I would like to see more ablation studies. For example, it is mentioned that 40 observations are used from the demonstrations and the trajectories. How is this number picked and how does it influence the performance? Also, typically, how long is a single trajectory? It is also unclear to me, if we take 40 random observations and the reward signal is sparse, wouldn't it be quite likely that the observations do not contain any reward values?
- How does K influence the performance of the algorithm?


Many challenging prediction problems, from molecular optimization to program synthesis, involve creating complex structured objects as outputs. However, available training data may not be sufficient for a generative model to learn all possible complex transformations. By leveraging the idea that evaluation is easier than generation, we show how a simple, broadly applicable, iterative target augmentation scheme can be surprisingly effective in guiding the training and use of such models. Our scheme views the generative model as a prior distribution, and employs a separately trained filter as the likelihood. In each augmentation step, we filter the model's outputs to obtain additional prediction targets for the next training epoch. Our method is applicable in the supervised as well as semi-supervised settings. We demonstrate that our approach yields significant gains over strong baselines both in molecular optimization and program synthesis. In particular, our augmented model outperforms the previous state-of-the-art in molecular optimization by over 10% in absolute gain.

(post rebuttal) I appreciate the authors for detailed rebuttal. I stay with the original score based on the following reason.
In RAML you sample from exponentiated reward distribution and do maximum likelihood (minimizing forward KL in classic control as inference framework). How you sample depends on the problem assumption. In the original paper, they used a heuristic based on edit distance from ground truth, but in theirs they did not have assumption that you can evaluate correctness (as used in external filter). In this paper's context, the sampling naturally comes down to rejection sampling. Therefore I keep the original stance regarding the similarity between this work and RAML.
---
The paper proposes an iterative data augmentation approach based self-generation and filtering with success criteria. The authors justify the algorithm as an EM procedure of maximizing \log p^*(y|x), where p^*(y|x) \propto p(y|x) * p(c=1|x,y). They demonstrate that the iterative data augmentation procedure can provide significant gains to SOA models in molecule generation and outperform MLE+RL method in program synthesis datasets.
Strengths:
- A simple approach that can be added to any seq2seq translation where success metric can be evaluated efficiently
- Demonstrated results on multiple applications
Weaknesses:
- The method requires being able to evaluate constraints
- The approach is simple and does not seem to present significant novelty over prior methods. Particularly, the approach could be considered as nesting RAML [1] updates with (1) low temperature, (2) self-generated trajectories.
Other comments:
- Some missing references [1, 2]. [2] also studies molecule generation and proposes approaches similar to RL + MLE.
[1] Norouzi, Mohammad, et al. "Reward augmented maximum likelihood for neural structured prediction." Advances In Neural Information Processing Systems. 2016.
[2] Jaques, Natasha, et al. "Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.


Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. Finally, we apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.

*Summary*
Authors proposed a variety of metrics to measure the reliability of an RL algorithm. Mainly looking at Dispersion and Risk across time and runs while learning, and also in the evaluation phase.
Authors have further proposed ranking and also confidence intervals based on bootstrapped samples. They also compared the famous continuous control and discrete actions algorithms on Atari and OpenAI Gym on the metrics they defined.
*Decision*
I believe the paper is discussing a very important issue, and some possible solutions to it, even if not perfect it's an important step toward paying more attention to maybe similar metrics. I am in favor of the paper in general,  but I have some concerns.
1. My main concern is that why authors think that community will adopt these metrics and report them? I like how authors have proposed different metrics, but having one or two easy to compute metric is much more likely to be adopted, than 6 different metrics, which I’m not sure how easy it is to use the python package? It’s of main importance, because if community don’t use these metrics in the future, the contribution of the paper is minimal.
2. There is no question of the importance of reliability of RL algorithms, but we need to be careful that RL algorithms are not optimizing for metics like CVaR, so maybe a better learning algorithm (in the sense of expectation learning) might not have better reliability metrics because it is not the main objective.
So following this, how would authors think their metrics can be used to design a more reliable algorithms? For example there is good literature on CVaR learning for safe policies. Do you think there exists a proxy for metrics you introduced that can be used to for the objective of the optimization?
3. Another main concern is the effect of exploration strategy: All these metrics can be highly affected  by different exploration strategy in different environments. For example if an environment has a chain like structure, then given the exploration strategy you may have an extremely high CVaR or IQR. How do authors think they can strip off this effect? (Running all algorithm with the same exploration strategy is not sufficient, since the interplay of learning algorithm and exploration may be important)
4. Generalizability: How do authors think these metrics are generalizable. For example if algorithm A has better metrics than algorithm B on open AI Gym task for continuous control, how much we expect the same ranking applies while learning on a new environment. I am asking this, because to me, some of these metrics are very environment dependent, and being reliable in some environments may not imply reliability in other environments.
*Note*:
Code and modularity of it: The main contribution of this paper will be shown when other researchers start using it and report the metric, if the code is hard to use, the contribution of the paper is hard to be significant.
==== Post Rebuttal ====
Thanks for the responses authors posted, I think there is a good chance that the community will benefit from this experimental metrics in the future, so I increase my rating to accept.


Message-passing neural networks (MPNNs) have been successfully applied to representation learning on graphs in a variety of real-world applications. However, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data: losing the structural information of nodes in neighborhoods and lacking the ability to capture long-range dependencies in disassortative graphs. Few studies have noticed the weaknesses from different perspectives. From the observations on classical neural network and network geometry, we propose a novel geometric aggregation scheme for graph neural networks to overcome the two weaknesses. The behind basic idea is the aggregation on a graph can benefit from a continuous space underlying the graph. The proposed aggregation scheme is permutation-invariant and consists of three modules, node embedding, structural neighborhood, and bi-level aggregation. We also present an implementation of the scheme in graph convolutional networks, termed Geom-GCN, to perform transductive learning on graphs. Experimental results show the proposed Geom-GCN achieved state-of-the-art performance on a wide range of open datasets of graphs.

GEOM-GCN: GEOMETRIC GRAPH CONVOLUTIONAL NETWORKS
The paper introduces a novel GCN framework, whose purpose is to overcome weaknesses of existing GCN approaches, namely loss of structural neighbor information and failure to capture important dependencies between distant nodes. The paper uses a mapping from nodes to an embedded space and introduces a second type of a neighborhood: a proximity in the embedded space. In the embedded space, a set of relations of nodes is defined. For each node v, the paper uses a 2-stage convolution scheme: 1) for each neighborhood type, the nodes in the same relation with v are combined; 2) the resulting nodes are again combined into a new feature vector. This approach allows one to overcome the issues described above. The experiments show that in most cases the approach outperforms the existing GCN solutions, sometimes with a large gap.
I have the following concerns about the paper:
-- My main concern is the learning time, which is an issue for a straightforward GCN implementation. There were multiple attempts to decrease it (GraphSAGE, FastGCN, etc.). Therefore, I would like to see running times on the presented graphs as well as on relatively large graphs (see e.g. https://arxiv.org/pdf/1902.07153.pdf for candidates). If some techniques were used to make the implementation faster, I would be good to include them in the paper (or, if they are standard, they should be referenced). At the very least, I believe it should be prioritized as a future direction.
-- It’s unclear why we should use the same latent space and the same τ for both N_g and N_s. I would expect that mapping into different spaces could provide better results: the two neighborhood types seem very different, and I don’t see why the neighbors should be aggregated in the same way. If using different spaces doesn’t provide an improvement, an explanation for this would be very useful.
-- α and β are defined and shown in Table 2, but they are never used (as it stands now, α and β can simply be removed). If the results in Table 3 correlate with them, then this dependence should be highlighted. In such case, it would also be better to move α and β to Table 3.
-- The paper uses 3 different node embedding strategies. These strategies can be combined in q with different weights (which can be learned as hyperparameters). Will it produce the best of 3 (or better) result?
-- “We use an embedding space of dimension 2 for ease of explanation” But what τ is used in the real implementation?
-- There are various GCN implementations; however, the comparison is performed with only 2 of them. I would like to see either comparison with more implementations, or the explanation why the comparison with the given two suffices.
-- Is it possible to make the implementation available?
While there are a lot of possible improvements, I believe that some of them can be addressed in a future research, and the paper’s novel approach is noteworthy in itself. My current verdict is 5/10, and I’ll be happy to improve it if the above issues are fixed.
Presentation issues:
-- The notation used in definition of m_v^l is unclear.
-- Why τ is a part of each node’s structural neighborhood? It’s a global function, isn’t it?
-- Introduction: I believe that the exact problems which GCNs solve (e.g. node classification) should be mentioned.
-- The flow in Section 2.1 is a bit weird. Namely, it says “To overcome the first weakness”, but the first wickness wasn’t stated in the previous paragraph (of course, one can deduce it, and it also was defined long ago, but it’s disturbing for a reader).
-- Figure 1B is confusing: it looks like the nodes from N_g(v) lie in a small region around v.
-- I think that splitting Figure 1C into 2 figures would make it clearer.


We investigate neural network training and generalization using the concept of stiffness. We measure how stiff a network is by looking at how a small gradient step on one example affects the loss on another example. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate. We experiment on MNIST, FASHION MNIST, and CIFAR-10 using fully-connected and convolutional neural networks. Our results demonstrate that stiffness is a useful concept for diagnosing and characterizing generalization. We observe that small learning rates reliably lead to higher stiffness at a given epoch as well as at a given training loss. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of a dynamical critical length that characterizes the distance over which datapoints react similarly to gradient updates. The dynamical critical length decreases with training and the higher the learning rate, the smaller the critical length.

This paper introduces the concept of stiffness: a measure of the change in the loss of sample A due to a gradient step based on sample B. It analyses the expected dynamic for A, B samples from the same and different classes, as well as, samples from the train and test sets.
To better understand the dynamics of optimization in neural networks is an open and important problem and the paper is clearly motivated in this regard. The proposed method is straight forward and I am not aware of a similar method.
In addition to that, the paper also introduces "dynamical critical length ξ" which is the stiffness of A, B samples based on the cosine similarity of the respective inputs (section 2.4). A linear estimator of when this length becomes 0 is also introduced. Confusingly this is also called the "dynamical critical length ξ" in section 4.2. Later on the term "dynamical scale ξ" and "dynamical critical scale ξ" seem to be used interchangeably. Figure 6 mentions the "critical length χ" on the y-axis which seems to be a typo as no such measure was introduced.
The equivalence between eq. 2 and the two parts of eq. 3 is not obvious. We'd appreciate if the authors would provide a proof of such.
Overall, the paper is written in a simple language but paragraphs remain surprisingly hard to understand. An example of such is e.g. section 4.4: What do the authors mean by "characteristic distance" between two input points? What is "the typical scale of spatial variation" of a function? etc.
The paper concludes that:
1.) there is a link between generalization and stiffness
2.) stiffness decreases with the onset of overfitting
3.) "general gradient updates with respect to a member of a class help to improve loss on data points in the same class"
4.) "The pattern breaks when the model starts overfitting to the training set, after
which within-class stiffness eventually reaches 0"
5.) This is observed for different models on different datasets
6.) "we observed that the farther the datapoints and the higher the epoch of training, the less
stiffness exists between them on average"
7.) "the higher the learning rate, the smaller the ξ"
Verdict: Reject
The conclusions are self-evident. The paper fails to demonstrate the usefulness of stiffness and most results are expected and provide little to no insights into the optimization dynamics of deep neural networks. In fact, the reasoning in this paper is almost tautological (conclusions 1-6).
E.g. if the A, B samples used to compute stiffness are separately drawn from the train and test set then stiffness is a proxy for the difference between the train error and the test error after another gradient step. The authors then compute stiffness at different points of the optimization procedure and conclude that stiffness decreases when the network starts to overfit. Since overfitting is the point in training where train error and test error diverge it is obvious that this can also be observed with regards to "stiffness". Hence, the reasoning is circular.
Conclusion 7 is slightly different in that it observes that larger learning rates result in smaller ξ which, given the previous paragraph, we can rewrite into the statement "larger learning rates generalise better". This is a well known empirical observation and has been discussed thoroughly (e.g. on connection with flat and sharp minima or learning rate decay schedules).
Disclaimer: This review was done on short notice.


In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query. Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism. This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion. The embeddings are trained with convolutional neural networks or the word2vec model. We demonstrate the performance of this model with image retrieval and text querying data sets.

In this paper, the authors propose to use attention to combine multiple input representations for both query and search results in the learning to rank task. When these representations are embeddings from differentiable functions, they can be jointly learned with the neural network which predicts rankings. A limited set of experiments suggest the proposed approach very mildly outperforms benchmark approaches.
Major comments
To the best of my knowledge, this is the first paper to apply attention to the learning to rank problem. However, the main methodological innovation seems to be the use of attention to create and train an ensemble of models; this has been previously explored in the literature (e.g., [Kim et al., ECCV 2018]).
The paper is also missing important context in that it omits developments in using deep learning for the learning to rank problem (e.g., [Pang et al., CIKM 2017; Ai et al., WWW 2018]). The experimental evaluation does not include any other deep methods; thus, it is not clear if the (very minor) improvement in performance are due to the deep models or the proposed attention approach.
The datasets used in the experiments are not appropriate for evaluating learning to rank algorithms. A variety of learning to rank datasets are available, and these should be used rather than (or in addition to) the toy datasets considered here. Examples: http://arogozhnikov.github.io/2015/06/26/learning-to-rank-software-datasets.html, http://quickrank.isti.cnr.it/istella-dataset/, https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/,
Minor comments
Concerning Section 3.3, in what sense is SGD used to “calibrate” the model? It seems as though the authors just mean it is used to “train” the model. However, is there some other meaning of calibration (e.g., in the sense of a Brier score) here?
In Table 1, what is the meaning of a dropout p value of 1? In most deep learning frameworks (e.g., Keras and PyTorch), this would mean all nodes are dropped out.
In what sense are the “5 randomized runs” for the experiments randomized? Are different train, test splits used? or just different random seeds? or something else?
How is it that the error rates are higher when using superclasses for evaluation?
Typos, etc.
The paper has several significant problems with the “\cite”s and “\ref”s in the paper. First, the “\cite”s should presumably be “\citep”s or something since the references are not set off from the rest of the text. Second, the paper includes references to equation numbers which are not present in the paper, such as “equation (12)”. It seems as the equations are in the paper, but are included in some unnumbered environment (“\begin{align*}” or some such). This makes it very difficult to track down to which equations the authors intend to refer. Third, the reference numbers to figures and tables in the text is wrong. For example, the text refers to “Tables 8 and 9” for 20 newsgroups (at the end of Section 4). Clearly, this is supposed to be Tables 6 and 7. It seems like the authors moved the CIFAR-10 discussion to the appendix but did not update the references in the text.
Tables 2 and 4 are exactly the same.
Figure 4 is not referenced in the text.
It would be helpful to put Figure 2 a bit closer to where it is discussed in the text.
The references are not consistently formatted.
“components of for each” -> “components for each”
Please define acronyms like MAP at least once.


Variational inference is a fundamental problem in Variational Auto-Encoder (VAE). By virtue of high-dimensional geometry, we propose a very simple algorithm completely different from existing ones to solve the inference problem in VAE. We analyze the unique characteristics of random variables on spheres in high dimensions and prove that the Wasserstein distance between two arbitrary datasets randomly drawn from a sphere are nearly identical when the dimension is sufficiently large. Based on our theory, a novel algorithm for distribution-robust sampling is devised. Moreover, we reform the latent space of VAE by constraining latent random variables on the sphere, thus freeing VAE from the approximate optimization pertaining to the variational posterior probability. The new algorithm is named as Spherical Auto-Encoder (SAE), which is in essence the vanilla autoencoder with the spherical constraint on the latent space. The associated inference is called the spherical inference, which is geometrically deterministic but is much more robust to various probabilistic priors than the variational inference in VAE for sampling. The experiments on sampling and inference validate our theoretical analysis and the superiority of SAE.

Summary
This paper considers the L2 normalization of samples “z” from a given prior p(z) in Generative Adversarial Netowks (GAN) and autoencoders. The L2 normalization corresponds to projecting samples onto the surface of a unit-hypersphere. Hence, to attempt to justify this normalization, the authors rely on some already established results regarding high dimensional hyperspheres. In particular, the focus is on the fact that, the Euclidean distance between any given point on a hypersphere and another randomly sampled point on the hypersphere tends to a constant, when the number of dimensions goes to infinity. This result is then used to show that the Wasserstein distance between two arbitrary distributions on a hypersphere converges to a constant when the number of dimensions grows. Based on this result, the authors claim that projecting the latent samples onto the surface of a hypersphere would make GAN less sensitive to the choice of the prior distribution. Moreover, they claim that such normalization would also benefits inference, and that it addresses the issue of variational inference in VAE.
Main comments.
This paper is hard to follow and requires substantial improvements in terms of writing, owing to several grammatical and semantic issues. Moreover, there is a lack rigor; some important claims are supported neither by experiments nor by theoretical analysis. Experiments in the main paper are also weak. I can therefore not recommend acceptance. My detailed comments are below.
- An important claim in this paper is that the proposed approach “alleviates variational inference in VAE”. However, this requires clarification as well as theoretical/empirical justifications.
- In the introduction, it is stated that generated samples from VAE may deviate from real data samples, because “the posterior q(z|x) cannot match  the prior p(z) perfectly”. However, in VAE we do not expect the posterior to match the prior perfectly, as this would result in useless data representations or inference. Generation issues in VAE may rather be explained by the fact that, in this context we optimize a lower bound on the KL-divergence between the empirical data distribution and the model distribution. The latter objective does not penalize the model distribution if it puts some of its mass in regions where the empirical data distribution is very low or even zero.
- Theorem 2 (on the convergence of the Wasserstein distance (W2) on high dimensional hyperspheres) does not seem to hold if, for instance, P and P’ are empirical distributions with overlapping supports. Further, even when the above Theorem holds, the W2 distance may be relatively high since it is proportional to the square root of the number of samples.
- Moreover, why and how would Theorem 2 justify improved inference when projecting latent samples onto a hypersphere?
- Please consider revising the following statement in the introduction: “The encoder f in VAE approximates the posterior q(z|x)”. The encoder “f” in VAE parametrizes the variational posterior.
- Some typos,
	- Abstract, “… by sampling and inference tasks”  -- “on sampling …”
	- Introduction second paragraph after eq 2. “… it also causes the new problems” – “ … causes new problems”
	- Section 2.1, “For convenient analysis …” – “For a convenient …”
	- Second paragraph after Theorem 1. “… perform probabilistic optimizations … ” – “… optimization …”
	- Section 5.2, second paragraph. Is it Figure 9?
The main recommendations I would make are as follows.
- Consider revising the paper to improve its writing.
- Provide rigorous theoretical analysis and discussions to support the main claims.
- Improve experiments by including more datasets and baselines (e.g., hyperspherical VAE [1]), as well conduct more targeted experiments to give more insights regarding the effect of the L2 normalization on inference and generation.
[1] Davidson, Tim R., et al. "Hyperspherical variational auto-encoders." UAI, 2018.


In contextual bandits, an algorithm must choose actions given observed contexts, learning from a reward signal that is observed only for the action chosen. This leads to an exploration/exploitation trade-off: the algorithm must balance taking actions it already believes are good with taking new actions to potentially discover better choices. We develop a meta-learning algorithm, MÊLÉE, that learns an exploration policy based on simulated, synthetic contextual bandit tasks. MÊLÉE uses imitation learning against these simulations to train an exploration policy that can be applied to true contextual bandit tasks at test time. We evaluate on both a natural contextual bandit problem derived from a learning to rank dataset as well as hundreds of simulated contextual bandit problems derived from classification tasks. MÊLÉE outperforms seven strong baselines on most of these datasets by leveraging a rich feature representation for learning an exploration strategy.

•	Summary
    This paper introduced a meta-learning algorithm for the contextual bandit problem, MELEE, which learns an exploration policy based on simulated and synthetic contextual bandit tasks. The training is mainly divided into two steps. In step one, they proposed to train a policy optimizer, which maps features and actions to rewards. This policy optimizer could be used to reveal the most valuable action to take according to the modeled reward. All possible actions and their corresponding values are revealed to the policy optimizer because of the existing ground-truth labels in the synthetic dataset. The policy optimizer would then suggest which action to take. The algorithm takes the action in an  greedy fashion, i.e. with probability  it will follow the suggestion and with probability  it will sample it uniformly at random. The policy optimizer, historical actions and the taken actions are appended to the training set for training the exploration policy  in the next step. The procedure in step one is proposed to be done in  rounds. In step two, the training set is used for training an exploration policy  . During testing, the contexts are drawn from the real world, the policy optimizer will first evaluate the whole history and the exploration policy will generate actions with the input from the policy optimizer and the context. The algorithm suggests the action to explore in an  greedy fashion. The proposed algorithm is evaluated on a dataset for learning to rank, and 300 synthetic datasets. It shows better performances in most cases.
    I am critical about the paper  because 1) the experiments show that all more recently published exploration methods are worse than weak baselines, for which it lacks enough justification and convincing explanations. 2) the experiments are difficult to understand with only citation to publications. Detailed information of the datasets, tasks, procedures is missing.
•	Main arguments
    The paper is in general hard to follow because of too many citations of the previous works without simple explanations. It refers to the imitation algorithm, AggreVate, which is an instantiate of meta-learning for contextual bandits. Meanwhile, they failed to clarify the difference between the proposed algorithm and the existing one, making the training algorithm part confusing. The major concern lies in the experiment section, I can not see big performance difference between the proposed method and the  greedy based methods in Figure 1 (left) as 1) the variances are large 2) there are overlapping . It is surprising to see all recently published method are worse than the classical  greedy method. These results may require deeper investigations. In addiction, the used datasets in experiments are mentioned without any details and task definitions, which makes the experiments part unclear and the result not that convincing. Below are some other inconsistencies in the paper:
1.	I had a hard time to understand what the function  means. In Section 2,  is used to map user feature to predicted rewards for actions, i.e.  is a function with user features and actions as input and reward as output. However, in equation 1, it only takes user feature as input. In Section 5.1,  is called a classifier (I regard it as a variant of function  ). Both are not consistent with the mapping definition when first formally defined. This point confuses me so that I cannot fully understand what the POLOPT does as the function  is the output of it.
2.	By the end of section 2, the paper claims that they used direct method for it simplicity and unbiased property. However, as verified in [1], the direct method is biased with low variance whereas IPS is unbiased with high variance.
[1] Dudık, Miroslav, John Langford, and Lihong Li. “Doubly Robust Policy Evaluation and Learning.”
•	Things to improve that did not impact the score
a.	clear definition of the introduced notations, including the ones in algorithms 1.
•	Questions:
a.	What is  ?
b.	What is POLOPT?
c.	Why do you choose AggreVate?
d.	Will the performance change if using any other methods instead of direct method for policy optimizer?
e.	What is roll-out policy in line 8 of Algorithm 1? (In text only the roll-out value is defined.)
f.	What will happen if the roll-in action is different from the behavior in test time?
g.	How is exploration policy trained?


Detection of photo manipulation relies on subtle statistical traces, notoriously removed by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design a lightweight trainable lossy image codec, that delivers good rate-distortion performance, comparable with the popular hand-crafted BPG, and has low computational footprint on modern GPU-enabled platforms. Our results show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/storage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20).

The paper describes a pipeline for image compression which allows to reliably detect specific manipulation patterns in compressed images.  The results show that it is possible to learn image compression that performs similarly to a modern image compression algorithm while in the same time is optimized to reveal specific kinds of manipulations. The authors build upon (Korus & Memon, 2019), but use a learnable codec instead of differentiable JPEG.
The idea to regularize entropy of the latent representation of images is interesting. A method to train a well-performing image compression system which can also follow additional constraints (such as ability to reveal certain manipulations) is very valuable for practice. Unfortunately, there are already available trainable compression methods and the authors do not compare to these methods. However, in my opinion to detect manipulation in the image one should prove that visual content in some area of the image was significantly changed with respect to some original, while in the other parts of the image it was not changed. Otherwise it becomes impossible to distinguish in-camera filtering and secondary postprocessing. Basically, the authors present a method to detect whether a very particular configuration of some basic image processing filters (Gaussian blur, median filter, resampling)  was applied to the image. Therefore the particular problem formulation looks very artificial.
With regards to the experiments in the paper, I was somewhat lost. Compare the Fig. 5 and the Fig. 8. In the Fig. 8, we see a big set of possible system configurations having different manipulation detection accuracy, image quality and compression performance.  In the Fig. 5, we see a compression efficiency-image quality dependency. However, it remains unclear how do the systems represented at these two graphs relate to each other, or, in the other words, what is he mapping between points of these graphs. Next, poor performance of JPEG manipulation detection by the proposed  network does not prove that JPEG manipulation cannot be detected, it just shows that the proposed architecture does not perform well in this problem. A comparative study which relates a new system to a current state of the art is required to claim that a proposed approach is better. Finally, SSIM is not a standard way to compute image quality. MS-SSIM and PSNR are also popular, and a user study is usually recommended to claim that some method generates images of better visual quality.
Summarizing, the authors do not provide a new best-performing image compression algorithm, and neither solve a problem of image manipulation detection, but show that it is possible to learn an image compression system with some additional constraints. I believe it is an interesting contribution, and I hope the authors can improve presentation of the experiments.


Reinforcement learning algorithms have been shown to effectively learn tasks in a variety of static, deterministic, and simplistic environments, but their application to environments which are characteristic of dynamic lifelong settings encountered in the real world has been limited. Understanding the impact of specific environmental properties on the learning dynamics of reinforcement learning algorithms is important as we want to align the environments in which we develop our algorithms with the real world, and this is strongly coupled with the type of intelligence which can be learned. In this work, we study what we refer to as ecological reinforcement learning: the interaction between properties of the environment and the reinforcement learning agent. To this end, we introduce environments with characteristics that we argue better reflect natural environments: non-episodic learning, uninformative "fundamental drive" reward signals, and natural dynamics that cause the environment to change even when the agent fails to take intelligent actions. We show these factors can have a profound effect on the learning progress of reinforcement learning algorithms. Surprisingly, we find that these seemingly more challenging learning conditions can often make reinforcement learning agents learn more effectively. Through this study, we hope to shift the focus of the community towards learning in realistic, natural environments with dynamic elements.

Summary
This paper discusses the value of creating more challenging environments for training reinforcement learning agents. Specifically, the paper focuses on three characteristics of the environment that the paper claims are necessary for developing intelligent agents. The first of these properties is stochasticity in the environment transitions, specifically stochasticity that is independent of the action taken by the agent. The next is sparsity of rewards; discontinuing the use of reward shaping to define desired behavior. Finally the paper argues that environments should not be episodic, that natural environments are continuing tasks so research focus should be around solving continuous tasks.
Review
My primary concern about this paper is the largely insufficient literature review. Many of the claims made in the motivation of this paper are not novel to this paper and are, in fact, incredibly vibrant sub-communities of study within the field of Reinforcement Learning. A more careful literature review should have easily found these communities and more nuanced claims could have been made. I will give concrete examples of the most important cases of missing literature in the following paragraphs, but this list is not exhaustive.
The paper claims that most standard RL environments include detailed reward functions that unnecessarily shape learning and inject bias into the learning process. While I agree that this is problematic, I disagree that this paper provides any novel insights towards this problem. The problem of learning from sparse rewards is well-known in the RL community and is a hot-topic of study. Even in the standard environments cited by the paper we have Montezuma's Revenge and Pitfall, two environment notorious for their difficulty due to sparse reward; each of which with its own host of literature surrounding only the single environment (for instance, I encourage the authors to investigate the highly controversial Go-Explore paper by Uber and its references). Other environments not considered by this paper include the Malmo (Minecraft) learning environment, around which NeurIPS 2019 hosted an extremely sparse reward competition. Another (overlapping) community of RL research interested in the sparse reward setting is the intrinsic reward community, one such paper being Riedmiller and Hafner et al. 2018.
This paper claims that all standard environments are episodic. Of the environments listed as "standard" by this paper, this claim does not even hold. However, there is a large chunk of the RL community that is not represented here. The continual learning and life-long learning communities are focused exclusively on the problem of non-episodic learning. Some example environments used by the community include Malmo, MuJoCo, DeepMind's Lab environment, and many smaller toy domains designed to showcase individual problems including Cart-Pole, RiverSwim, Pendulum, and Acrobot; with the smaller environments from OpenAi Gym cited by this paper. Another smaller community to investigate would be the average reward formulation of the RL problem, which fairly exclusively focuses on the continual learning problem.
Finally, this paper seems vaguely reminiscent of a few particular environments that I have seen in the literature previously. For example, Berkeley's robot task discovery playpen (see for example Singh, Yang, Hartikainen, Finn, Levine 2019). Or an even more similar simulated environment being the Playroom environment by Singh, Barto, Chentanez 2005. Finally, Malmo has been used in a similar way as the tool-building examples mentioned in this paper.
There were a few key issues with the experiments discussed in this paper. The first of which being "Hypothesis 1" which states: "Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem." This hypothesis alone does not appear to be uniformly true. In fact, imagine a simple 5-state random walk markov chain environment without termination. Each state is visited infinitely many times, so the chain is ergodic and there are no non-stationary points. The empirical section uses meta-parameters that were ill-motivated with no discussion about meta-parameter selection. It is critical to point out that the stepsize used for an episodic problem will likely not be the optimal stepsize for the corresponding non-episodic problem, as the magnitude and variance of the considered returns are necessarily different (in response to Figure 3). Further, evaluating over 3 random seeds simply is not sufficient to make any statistically significant claims when comparing any of these curves (for instance in Figure 5).
Additional Comments (do not influence rating)
For a paper exclusively introducing a new control environment, it is critical to include a discussion about the exploration problem. At the very least, I would appreciate seeing sensitivity curves for values of epsilon.
This paper makes many strong claims about the nature of intelligence that are neither supported in the work or are accepted in the community. While it is intuitive that the environment plays a critical role in developing intelligence, the lack of universal definition of intelligence makes this a non-falsifiable claim. Although I appreciate the point the authors are trying to make, which is that RL research frequently is done in the realm of toy simulated domains, I do not think that this paper includes the appropriate supporting evidence to validate such lofty claims.
It would be interesting to change the exploration method from epsilon-greedy to sampling according to the softmax action distribution. This can have dramatically improved performance on non-adversarial exploration problems, and reduces the need for scheduled epsilon decay. It additionally reduces the need for two extra meta-parameters, allowing the empirical claims to be made more strongly without performing some sweep over parameters.
--------------------
Edit after reading discussion, rebuttal, and edits to manuscript.
The paper's intended contribution was different than I had realized during the initial review phase. I appreciate the effort the author put into the response and the changes made to the literature review section of the paper. I think these help to demonstrate the scope and placement of the work considerably.
I still believe the paper over-states the novelty of the results and I still find it difficult to understand their utility. I believe that the entirety of the paper falls under the domain of exploration in RL, but this is not made clear through the introduction or related works section (though the updates to the related works section help considerably).
Stochasticity in the environment helping the agent to learn is not a surprising finding at all if the exploration method is insufficient. To give a small example, imagine an agent wandering in a tabular gridworld. If the agent has no method of exploration, and the environment has sparse reward, then it is not surprising that the agent would get stuck in a corner. If the environment was modified so that each transition had a 5% chance of randomly going another direction (e.g. the "right" action has a 5% chance of becoming an "up" action), then we've effectively encoded epsilon-greedy exploration through the environment dynamics. All of this comes down to say, it is difficult to separate the dynamics of the environment from the agent's exploration method. I think it would require a careful study that considers these aspects more explicitly.
To summarize, I think the paper could easily be accepted to a future conference, but I think it is important to:
- Make connection between the insights and exploration clear, specifically designing the introduction, lit review, and experiments around this connection.
- Make sure the contributions are extremely clear in the writing. Demonstrate those contributions directly in the empirical section.
- Tone down the claims in the writing. Many of the claims about the state of the field in RL are demonstrably incorrect. I agree with the sentiment trying to be expressed, but the absolutism makes it difficult to separate that sentiment from a deep understanding of the current RL literature with only the most important papers cited, or a fundamentally insufficient lit review. To make claims about the state of a field the lit review should be rather extensive.


Capsule networks are constrained by the parameter-expensive nature of their layers, and the general lack of provable equivariance guarantees. We present a variation of capsule networks that aims to remedy this. We identify that learning all pairwise part-whole relationships between capsules of successive layers is inefficient. Further, we also realise that the choice of prediction networks and the routing mechanism are both key to equivariance. Based on these, we propose an alternative framework for capsule networks that learns to projectively encode the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer. This is done using a trainable, equivariant function defined over a grid of group-transformations. Thus, the prediction-phase of routing involves projection into the SOV of a deeper capsule using the corresponding function. As a specific instantiation of this idea, and also in order to reap the benefits of increased parameter-sharing, we use type-homogeneous group-equivariant convolutions of shallower capsules in this phase. We also introduce an equivariant routing mechanism based on degree-centrality. We show that this particular instance of our general model is equivariant, and hence preserves the compositional representation of an input under transformations. We conduct several experiments on standard object-classification datasets that showcase the increased transformation-robustness, as well as general performance, of our model to several capsule baselines.

In this work, a method was proposed to train capsule network by projectively encoding the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer. Thereby, the proposed method aims to improve equivariance of capsule nets with respect to translation (rotation and scaling).
The proposed method is interesting and the initial results are promising. However, there are various major and minor problems with the work:
- There are various undefined functions and mathematical notation, such as the following:
- Please give formal and precise definitions of groups and group representations for readers who are not familiar with mathematical groups.
- What are GetWeights and Agreement used in Algorithm 1?
- Please define “routing among capsules” more precisely.
- How do you calculate Pool() more precisely?
- In the paper, the results are given for a general class of groups. However, it is not clear how these results generalize even for some popularly employed groups, such as Z^n, S_n, SO(n), SE(n) etc., with different symmetry properties, base space, and field type.
- Please check the following paper for a detailed discussion on group equivariant CNNs with different group structures, and elaborate the theoretical results for particular groups (e.g. at least for p4m used in experiments) :
T.S. Cohen, M. Geiger, M. Weiler, A General Theory of Equivariant CNNs on Homogeneous Spaces, NeurIPS 2019
- Please define the norms used in Algorithm 2.
- How do you calculate accuracy of models? Are these numbers calculated for a single run, or for an average of multiple runs? If it is the former, please repeat the results for multiple runs, and provide average accuracy with variance/standard deviation. If it is the latter, please provide the variance/standard deviation as well.
- Have you performed analyses using larger datasets such as Cifar 100 or Imagenet? It would be great to provide some results for larger datasets to explore their scalability.
- Please define accuracy given in tables more precisely, use dot "." at the end of sentences in captions.
- There are several typo/grammatical errors, such as the following:
-- Homegenous -> homogeneous
--  for with prediction networks
-- Please proof-read the paper in detail and fix the typo etc.
After the discussions:
Most of my questions were addressed and the paper was improved in the discussion period. Therefore, I increase my rating.
However, some parts of the paper still need to be clarified. For instance;
- GetWeights: To ensure that the predictions are combined in a meaningful manner, different methods can be used to obtain the weights. The role of GetWeights is to represent any such mechanism.
-> Please define these methods in detail and more precisely, at least in the Supp. mat.
- Agreement : The Agreement function represents any means of evaluating such consensus.
-> This is also a very general concept, which should be more precisely defined.
- Our theoretical results (Theorem 2.1 and Theorem 2.2) hold for all general groups, and the particular group representation  defined in the main paper.
-> Could you please give a concrete discussion on generalization of these results and the proposed algorithms for discrete and continues groups? For instance, how do these algorithms and results generalize with Z^n and SO(n)?


Quantization reduces computation costs of neural networks but suffers from performance degeneration. Is this accuracy drop due to the reduced capacity, or inefficient training during the quantization procedure? After looking into the gradient propagation process of neural networks by viewing the weights and intermediate activations as random variables, we discover two critical rules for efficient training. Recent quantization approaches violates the two rules and results in degenerated convergence. To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training. We also analyze the quantization error introduced in calculating the gradient in the popular parameterized clipping activation (PACT) technique. Through SAT together with gradient-calibrated PACT, quantized models obtain comparable or even better performance than their full-precision counterparts, achieving state-of-the-art accuracy with consistent improvement over previous quantization methods on a wide spectrum of models including MobileNet-V1/V2 and PreResNet-50.

Quick Summary: Based on semi-quantitative analysis the paper first proposes two rules for quantization of DNNs, then extends previous methods based on these rules to propose specific technique for quantizing activations and weights. Experimental results are provided on MobileNet V1/VB2 and ResNet50 and compare favourably against baselines (which are old and unfortunately do not represent recent developments in the literature). Overall I found novelty low and also there are several recent papers already published before this submission which provide results which are at least as good or even better
Details
Specifically the rules are:
1. To prevent logits from entering saturation region of the cross entropy loss, the effective weight in the last fully connected layer should be small.
2.To keep the gradient of weights in the same scale across the whole network, either BN layers should be used after linear layers such as convolution and fully- connected layers, or the variance of the effective weights should be on the order of the reciprocal of the number of neurons of the linear layer (n_l).
They use DoReFa for weight quantization and PACT for activation quantization based on the above observations/rules. However the contributions were not really very novel in my opinion. See especially the paper "LEARNED STEP SIZE QUANTIZATION" I linked below when explaining novelty
This reviewer feels that the authors were perhaps unaware of several important contributions to the literature this year which are substantially better than the baselines compared against in the paper. I list a few below for the authors to compare against, and to explain their novelty in contributions in the rebuttal phase. Unfortunately this makes me feel the paper should be a clear reject in its current state, unless the authors can convince us otherwise.
https://arxiv.org/pdf/1902.08153.pdf
https://arxiv.org/pdf/1903.08066.pdf
https://arxiv.org/pdf/1905.11452.pdf
https://arxiv.org/pdf/1905.13082.pdf


A general graph-structured neural network architecture operates on graphs through two core components: (1) complex enough message functions; (2) a fixed information aggregation process. In this paper, we present the Policy Message Passing algorithm, which takes a probabilistic perspective and reformulates the whole information aggregation as stochastic sequential processes. The algorithm works on a much larger search space, utilizes reasoning history to perform inference, and is robust to noisy edges. We apply our algorithm to multiple complex graph reasoning and prediction tasks and show that our algorithm consistently outperforms state-of-the-art graph-structured models by a significant margin.

This paper introduces policy message passing: a graph neural network with multiple message types and an inference mechanism that assigns messages of a particular type to edges in a recurrent fashion. Experiments on visual reasoning tasks and on a noisy citation network classification task indicate competitive performance.
There are many interesting ideas in this paper and I believe that the overall approach/idea has merit and is interesting for the community. In its current state, however, the paper has issues in terms of clarity of writing, technical and notational mistakes, unclear experimental settings and unclear implementation of baseline models, and needs a major revision, hence I recommend reject. This feedback is mostly meant as an encouragement to “polish” the ideas outlined in the paper and their presentation, as I believe that these can be quite impactful, if presented well and once all technical issues have been addressed.
Discussion of the main idea outlined in this paper:
———————————————————————
The proposed dynamic message passing procedure (termed “reasoning agents” in the paper) can be understood as an extension of Neural Relational Inference (NRI; Kipf et al., ICML 2018), where a Gumbel softmax-based attention mechanism in parallel softly assigns a particular message function per edge in the graph. The main difference to NRI is that this is done sequentially over multiple message passing steps in the graph, whereas NRI (in the training phase) only performs one single such step (for all edges in the graph).
Getting this to work well in practice is a challenging problem, as sequential sampling steps can accumulate biases in the approximate inference scheme. The authors propose to address this issue in a number of ways: the attention mechanism per edge is coupled over time steps via a local hidden state and coupled to all other edges via a global hidden state (this is novel), a vector of ones is added to the action probabilities and subsequently re-normalized to act as a regularizer (this appears to be a hack), they use a learned prior distribution instead of a fixed prior as in NRI, and lastly, the authors introduce a mixed sampling strategy which mixes samples from the (learned) prior distribution for edge types and the proposal distribution while re-weighing the objective with importance sampling weights — for this last contribution (mixing + reweighing) the description in the paper misses some details (how precisely is the mixing done over the course of training?).
Clarity, technical correctness, experimental evaluation
———————————————————————————
* The paper contains a lot of spelling mistakes and grammatical errors, making it difficult to read.
* Notational issues: Eq 3 is unclear (no node/edge indices provided, function arguments unspecific — all three functions have the same arguments); Eq. 8/9 switches between posteriors p, true posteriors \pi^* (what’s the difference from p?) and priors \pi without explanation— a derivation would be helpful. Eq. 10 only places a prior on the first time step — where does this come from? A derivation would be helpful.
* “Belief propagation or other neural message passing frameworks define a delta function where the prior is deterministic and fixed” -> this is not the case for NRI
* What temperature is used for the Gumbel softmax distribution? Does the forward pass use “hard” (straight-through trick) or soft activations?
* Eq. 5: Why condition B on the indices i and j? This will likely not generalize to new edges
* Eq. 6: What is the motivation for adding a vector of ones? Wouldn't it be possible to achieve the desired regularization effect by increasing the softmax temperature?
* How are the edge and global hidden states initialized?
* The explanation for the mixed sampling procedure is unclear, please elaborate (potentially use an appendix)
* What are the train/val/test splits, how are hyper parameters optimized in all experiments?
* Why implement attention-based GNN baselines with sigmoid gates even though you cite Transformers and Graph Attention Networks (GAT) in this paragraph? I suspect that multi-head dot product attention (default for Transformers/GAT) instead of sigmoid gating would perform much better.
* Please explain the architecture / experimental setting used in the GNN and RRN baselines in an appendix. Please also clarify how you use NRI (recurrent version, Markovian version etc.)
* “Where am I” task: the model setup paragraph is unclear
* Figure 3a: the example provided is not d x d (but 3x4 in terms of # puzzle pieces)
* Ablation studies: please consider including experiments that analyze how the proposed model performs without global state, without the added prior (vector of ones), without the mixed sampling procedure, without recurrence and with different numbers of functions per edge.
Other comments:
* Please consider familiarizing yourself with the \citep command in LaTeX for parenthetical citations to avoid sequences of closing parentheses.