Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, spurred discussion, and lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs. In this paper we show that the widely used "non-saturating" training scheme can also be interpreted in this way, specifically as minimizing a particular reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

######## Updated Review #########
I would like to thank the author(s) for their rebuttal, which I have carefully read. I also appreciate the effort made to improve the paper. My overall evaluation of the paper stands unchanged.
#############################

#### My review is based on the updated paper downloaded from the anonymous link. ####
This paper discusses alternative training strategies for f-GANs. While the discussion has some interesting points, the presentation needs to be much improved. It is not easy to follow this paper in its current form, and the main results are not properly emphasized. As such, I am not certain of its real contribution. f-GANs are not routinely used in practice (except for the vanilla JSD and RKL), and as far as I can tell the saturating gradient issue is no longer a central concern (it has been well addressed years ago, with, e.g. WGANs). I am voting to reject this submission, but I am willing to re-evaluate this paper if the author(s) significantly improves their writing.
- In Section 2, what does it mean by "the Fisher metric of the family"? The concept of Fisher metric is defined anywhere in the text, and there is no reference to it.
- It is not is intuitive why the second derivative of f_R takes the form $u^{-3} f''(u^{-1})$ (given above Eqn (2)). Please elaborate.
- Please avoid the use of subjective phrases such as "imagining", "get a feel", etc. I am guessing the author(s) are trying to suggest taking a visual inspection of the discrepancies projected on the log-likelihood ratio axis (which is 1D) and figure out which f-div might be more appropriate.
- Fig. 3 needs legends. There are two solid (dotted, resp) lines in the Figure, and I am guessing one of them is for the saturating and the other for the non-saturating gradient. This needs to be specified because the line specs are identical.
- After going through the entire paper, I would highly recommend the author(s) to take a course in academic writing. The main contributions are not highlighted and some of the key concepts are not even properly defined. For example, analysis of the non-saturating gradient, which is supposedly the main result of this submission, appeared in pp. 7, by which time most readers have exhausted their patience. The vanilla version of the non-saturating scheme never appeared in the main text. The notation system is also non-standard, where the notation $\bar{\lambda}$, normally reserved for average/expectation, has been used to denote the gradient wrt $\lambda$. The writing can be very unprofessional at times, for example, "the answers to these questions are yes, yes and no respectively".
- I do not know why the author(s) inserted one toy experiment in the paper, as it serves no purpose. Each model converges to their respectively optimal, as expected. There is no discussion of how to choose an appropriate f-div or (f,g) hybrid in practice.
- I totally agree with the author(s) on the point that this is a note (sec 7, second paragraph), rather than an academic paper. There is a lot of derivations in the paper, but insightful discussions are limited in the sense that it is not clear how to connect these results to improve practice. The section titles also read like bullet points in a note.
- Not certain about the significance of this paper. The derivations are fairly standard and I can not find any useful proposals that might benefit the practice of f-GAN training. There is a number of papers discussing non-saturating GAN training (see sec 8), and I do not think this paper adds too much value to this discussion.


Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map. The precise form of this representation is often considered to be a metric representation of space. An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks. Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment. To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output. We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state. These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities. By combining two types of networks in a modular structure, we could get better performance for both regularities. Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -which can be shaped by pre-training and analyzed using dynamical systems methods. Furthermore, we demonstrate that non-metric representations are useful for navigation tasks, and their combination with metric representation leads to flexibile multiple-task learning.

## Overview
This paper explores how pre-training a recurrent network on different navigational objectives confers different benefits when it comes to solving downstream tasks. First, networks are pretrained on an objective that either emphasizes position (path integration) or landmark memory (identity of the last wall encountered). This pretraining generates recurrent networks of two classes, called PosNets and MemNets (in addition to no pre-training, called RandNets). Surprisingly, the authors found that pre-training confers different benefits that manifests as differential performance of PosNets and MemNets across the suite. Some evidence is provided that this difference has to do with the requirements of the task. Moreover, the authors show how the different pretraining manifests as different dynamical structures (measured using fixed point analyses) present in the networks after pre-training. In particular, the PosNets contained a 2D plane attractor (used to readout position), whereas the MemNets contained clusters of fixed points (corresponding to the previously encountered landmark).
Overall, I thought this was a very interesting paper--it is one of the first papers I have seen that demonstrates how different pre-training requirements both change network dynamics (as measured by fixed points), and how those differences can yield different benefits on downstream navigational tasks.
## Major comments/concerns
- I think the presentation the pretraining objective (eq 3) could be clearer. Is eq 3 what is minimized during pre-training? How are \alpha, \beta, and \gamma chosen? \alpha is used to separate the two types of networks (MemNet from PosNet), which is the critical difference studied in the paper, so it would helpful to go into more detail about what \alpha controls and how it was chosen.
- For the first task, I am surprised that the agent is able to navigate the environment using only the eight neighboring locations. What is the size of the arena? What fraction of the states are simply surrounded on all sides by empty space? It would be informative to show some trajectories of agents solving the basic task.
- For Fig 3A and 3B, it would be nice to show the other network's performance (i.e. show the PosNet on the scaling task in 3A, and the MemNet on the bar task in 3B).
- How come there are no networks that are able to solve both sets of tasks? That is, how come there are no networks in the upper right region of Fig 3C? Does this suggest that an agent needs to combine two separate RNNs to solve the whole suite of tasks?
- What happens if you train recurrent networks with more sophisticated cell architectures (e.g. a GRU or an LSTM)? These are typically easier to train (and using automatic differentiation techniques are also amenable to fixed point analysis).
## Minor comments
- In eq. (1), use `\left(` and `\right)` to make the first set of parentheses have an appropriate height.
- Typo on the first line after eq. (6) (matrices)
- Relevant reference on comparing networks using dynamics around approximate fixed points: https://arxiv.org/abs/1907.08549.


In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully-trained model. Our source code can be found in the appendix.

In this paper, the authors performed an empirical study on the importance of neural network weights and to which extent they need to be updated. Some observations are obtained such as from the third epoch on, a large proportion of weights do not need to be updated and the performance of the network is not significantly affected.
Overall speaking, the qualitative result in the paper has already been discovered in many previous work, although the quantitative results seem to be new. However, there is large room to improve regarding the experimental design and the comprehensiveness of the experiments. Just name a few as follows:
1)	For different models and different tasks, the quantitative results are different. There is no deep discussion on the intrinsic reason for this, and what is the most important factor that influences the redundancy of weight updates. The authors came to the conclusion that from the third epoch on, no need to update most of the weights. “3” seems to be a magic number to me. Why is it? No solid experiments were done regarding this, and no convincing analysis was made.
2)	The datasets used in the experiments are not diverse enough and are not of large scale. For example, the CIFA-10 and MNIST datasets are relatively of small scale. What if the datasets are much larger like ImageNet. In such more complicated case, will the weight updates still be unnecessary? Will the ratio and the epoch number change? What is the underlying factor determining these? For another example, there are many NLP datasets for language understanding and machine translation, which are of large scale. Why choosing an image captioning dataset (which I do not agree to be real-life experiments when compared with language understanding and machine translation)? Can the observations generalizable to more complicated tasks and datasets?
3)	The models studied in the paper are also a little simple, especially for the text task. Why just using a single-layer LSTM? Why not popularly used Transformer?
As a summary, for an empirical study to be convincing, the tasks, datasets, scales, model structures, detailed settings, and discussions are the critical aspects. However, as explained above, this paper has not done a good job on these aspects. Significantly more work needs to be done in order to make it an impactful work.
*I read the author rebuttal, but would like to keep my rating unchanged.


Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

First, I would like to note that the claim that SGD with momentum is a special case of Adam with large epsilon is technically wrong because Adam also includes the bias-corrected momentum estimates which SGD with momentum does not consider. It might seem like a small difference, however it is a form of learning rate schedule which most users of Adam are not aware of. In practice, however, Adam with large epsilon can approximate SGD with momentum. Just don't claim the equivalent since it is not there.
I have some difficulties understanding the contribution of the paper. For example
"When tuning all available metaparameters under a realistic protocol at scales common in deep learning,
we find that more general update rules never underperform their special cases."
In practice you do adjust hyperparameter search spaces to fit your conclusions, e.g., "We found that searching over (epsilon, alpha0/epsilon) was more efficient than searching over (epsilon, alpha)." Again, this alone invalidates your experimental setup since you biased it in order to fit your conclusion: "was more efficient" was found after running some prior experiments.
Another situation where your experimental setup is unfairly tuned is when you used different hyperparameter ranges for similar hyperparameter, e.g. see D.2 for ResNet-32 on CIFAR-10 where 6 orders of magnitute difference was used for the initial learning of Momentum and 3 orders of magnitude difference for the initial learning rate of Adam. Similarly, there is a difference of 10x for ImageNet experiments.
The paper suggests that 16 experiments is enough to produce good results. First, one should not forget the  special arrangements (see above) done for hyperparameter search space. Second, for any person working in black-box optimization it is clear that 16 experiments is next to nothing. It should give you something good in 1D, possibly in 2D if your search range is narrow. This is absolutely nothing in larger dimensions (providing that your benchmark in not super trivial and your hyperparameter search space is not absolutely boring when you already narrowed it around the optimum). After 16 evaluations you get pretty bad settings for most algorithms.
Update:
The paper uses a naive hyperparameter optimizer and runs it for a very small budget. The latter likely affects the conclusion of the paper that different training algorithms perform similarly. The authors seem to accept it by mentioning that this is the case for their tuning protocol/budget.
If we would like to compare different training algorithms, we should optimize them on a set of problems using 2-3 state-of-the-art hyperparameter optimizers. Then, we should study how the best seen solutions so far and their robustness  change as a function of the computation budget (the maximum budget should be large enough). Then, one would see that the results are not that different for small budgets (a boring result) and somewhat different for larger budgets. Showing only the boring part seems more misleading than useful.
Update#2:
As I mentioned in my review, Adam with large epsilon is not equivalent to momentum SGD but only approximates the latter. This is because the original Adam has a bias correction term and even if the same *global* learning rate schedule is used both for Adam with large epsilon and momentum SGD, they are not equivalent. In order to obtain the exact equivalence, one would need to either
1) drop the bias correction term of Adam and thus modify the algorithm in order to satisfy the claimed equivalence
or
2) set a particular learning rate *for each batch pass* of Adam to simulate the effect of the bias correction term, this leads to a large number of hyperparameters - as many as the number of batch passes, this is intractable (the setup of the authors does not optimize such batch-wise hyperparameters, they are defined by a global scheduler as a function of batch/epoch index).
If you avoid these modifications, then you can't claim the equivalence but only an approximation. If you don't have the equivalence of the two approaches and so momentum SGD is not a particular case of Adam, then the following sentence from the abstract is false: " As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum)". Again, strictly speaking, it is false that "Adam should never perform worse than momentum" because momentum SGD is not a particular case of the original Adam *unless* you drop the bias correction term or simulate it with tons of hyperparameters, one learning rate value per batch pass. Any global learning rate schedule *used for both* algorithms will not solve the issue because the bias correction term will remain. If you don't modify the learning rate schedule of Adam but only of momentum SGD, then you basically adjust your SGD by moving some part of Adam in it to claim the equivalence of the two, such actions can make pretty much every second algorithm equivalent to another.
My main concern is described in the first Update. It is trivial that a more general optimizer is capable to perform at least as good as its particular case. What is not trivial is to clarify the interplay of computational budgets spent on hyperparameter tuning vs number of hyperparameters vs performance over time.


Neural network pruning techniques have demonstrated it is possible to remove the majority of weights in a network with surprisingly little degradation to top-1 test set accuracy. However, this measure of performance conceals significant differences in how different classes and images are impacted by pruning. We find that certain examples, which we term pruning identified exemplars (PIEs), and classes are systematically more impacted by the introduction of sparsity. Removing PIE images from the test-set greatly improves top-1 accuracy for both sparse and non-sparse models. These hard-to-generalize-to images tend to be mislabelled, of lower image quality, entail abstract representations, atypical examples or require fine-grained classification.

The paper claims that neural network pruning methods have different impact on accuracy in class-wise and sample-wise. To this end, the paper performs statistical tests to identify such classes (and samples) from a population of neural network models of different levels of pruning. In particular, the identified samples, called PIEs, are shown to have significantly lower test accuracy, i.e., the PIEs are much harder to classify, and a user study is performed to support this claim. The paper also demonstrates that pruned models tend to have lower accuracy against adversarial attacks or common corruptions.
In overall, the paper addresses an important problem of investigating the effects of pruning. The experiments performed here seems fairly extensive. One of my primary concerns, however, is that I am still not convinced whether the empirical findings presented here is indeed significant, as some reader might feel those results are not that surprising. I personally feel that it is much more likely that pruning gives heterogeneous effects over the classes, since current pruning schemes do not enforce the "uniform" brain damage, i.e., they simply aim to minimize total accuracy drops. Also, the accuracy differences in Table 1 do not appear to be that large in my opinion. The questions in what follows may help to resolve such concerns:
- Section 2.1: For ImageNet dataset, data imbalance in training set might be the reason of such disparate impact? As CIFAR-10 dataset is balanced, I wonder if CIFAR-10 results could be also presented in more details.
- What if the whole training and pruning is re-performed after excluding the classes which the accuracy is decreased more by pruning? Would it lead to better pruning results, or better overall accuracy?
- I feel there should be more explanation about the actual statistical implication of the use of Welch's t-test: Is it ok that S^c_t may not be independent to S^c_0? What is the key benefits of not just picking outlier classes among S^c_t with respect to c?
- What would happens if we exclude the PIEs in training set, and performs training & pruning from scratch? The overall claims may be strenghten if this could improve the pruning efficiency.
- Figure 7: I think the pure ImageNet top-1 accuracy should be also presented in each plot as a baseline for fair comparison, as this accuracy will be decreased with respect to the sparsity as well.


Neural networks show great vulnerability under the threat of adversarial examples. By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled. One intriguing property of the adversarial examples is transferability. This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world. The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods. Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box attack. However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods. We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks. Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information. Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples. We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function. Our method is generally applicable to most gradient based attack methods.

The paper suggests to use temperature scaling in adversarial attack design for improving transferability under black-box attack setting. Based on this, the paper proposes several new attacks: D-FGSM, D-MIFGSM, and their ensemble versions. Experimental results found that the proposed methods improves transferability from VGG networks, compared to the non-distillated counterparts.
In overall, I liked its novel motivation and simplicity of the method, but it seems to me the manuscript should be improved to meet the ICLR standard. Firstly, the presentation of the method is not that clear to me. The mathematical notations are quite confusing for me as most of them are used without any definitions. I am still not convinced that the arguments in Section 3.1 and 3.2 are indeed relevant to the actual practice of black-box adversarial attacks, which usually includes extremely non-smooth boundaries with multiple gradient steps. Even though the experiments show effectiveness partially on VGGNets, but the overall improvements are not sufficient for me to claim the general effectiveness of the method unless the paper could provide additional results on broader range of architectures and  threat models.
- I feel Section 2.3 is too subjective with vague statements. The following statement was particularly unclear to me: "The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations.": Does the term "effectiveness" indicate some relative effectiveness compared to other methods, e.g. optimization-based attacks? Is this really a general phenomenon in gradient-based attacks? Also, please elaborate more on "So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons ..."
- Regarding that the softmax is the problem, one could try to directly minimize the logit layers skipping the softmax, i.e., gradient on logits? This is actually one of common techniques and there are many simple tricks in the context of adversarial attack, so the paper may include comparisons with such of tricks as well.
- It is important to specify the exact threat model used throughout the experiments, e.g. perturbation constraints and attack details. Demonstrating the effectiveness on a variety of threat models could also strengthen the manuscript.
- Table 1 and 2 may include other baseline (black-box attack) methods for comparison. This would much help to understand the method better.


Hierarchical graph representation learning is an emerging subject owing to the increasingly popular adoption of graph neural networks in machine learning and applications. Loosely speaking, work under this umbrella falls into two categories: (a) use a predefined graph hierarchy to perform pooling; and (b) learn the hierarchy for a given graph through differentiable parameterization of the coarsening process. These approaches are supervised; a predictive task with ground-truth labels is used to drive the learning. In this work, we propose an unsupervised approach, BAYESPOOL, with the use of variational Bayes. It produces graph representations given a predefined hierarchy. Rather than relying on labels, the training signal comes from the evidence lower bound of encoding a graph and decoding the subsequent one in the hierarchy. Node features are treated latent in this variational machinery, so that they are produced as a byproduct and are used in downstream tasks. We demonstrate a comprehensive set of experiments to show the usefulness of the learned representation in the context of graph classification.

The authors propose in this paper a new unsupervised graph representation learning method. The method leverages recent advances in graph coarsening, mainly Loukas' method. The key idea of the method consists in using a reconstruction target that is not the classical one in an auto-encoder setting. More precisely, the encoder takes as an input the original adjacency matrix and node features but the decode only aims at reconstructing the coarse adjacency matrix (obtained via Loukas' method).
The experimental evaluation is quite thorough and shows that the method performs quite well, especially considering it is unsupervised but is compared to supervised representation methods. It would be nice to include statistical tests to assess the significance of the differences in cases were accuracies are very close one to another. A missing part would be to explore the relevance of the learned representation for other tasks (i.e. to use a multi task data set). Of course as the representation is learned in an unsupervised way, one can argue that the current evaluation is already providing an answer.
Overall, I find the paper clear, but the variational bayes part could be much clearer. In fact I'm not sure why this is presented as variational bayes and not only variational. I do not see any prior distribution over parameters, for instance. I understand that the recent "tradition" in variational auto-encoder is to use this terminology, but as a (part time) bayesian, this is a bit annoying.


A Non-constrastive loss for Unsupervised Graph Embeddings
We present Deep MinCut (DMC), an unsupervised approach to learn node embeddings for graph-structured data. It derives node representations based on their membership in communities. As such, the embeddings directly provide interesting insights into the graph structure, so that the separate node clustering step of existing methods is no longer needed. DMC learns both, node embeddings and communities, simultaneously by minimizing the mincut loss, which captures the number of connections between communities. Striving for high scalability, we also propose a training process for DMC based on minibatches. We provide empirical evidence that the communities learned by DMC are meaningful and that the node embeddings are competitive in different node classification benchmarks.

Although this paper seems to only combine existing techniques in community detection and node embedding into a co-train process. The idea is simple and easy understood and the paper is well-written. Theoretical analysis is provided for the approximation error for the sampling strategy. However, major concerns are:
1. Experimental results show that co-training node embedding and community detection can improve the performance for node classification. The improvements may result from the assumption that papers with the same class label are associated with the same community in the citation graph. However, in the dataset, there are many cases that there are not dense connections among the same labeled papers. The authors should check the correlation between the detected communities and the original paper labels.
2. No comparison with other community-preserving node embedding methods, such as "Community Preserving Network Embedding" in AAAI17
3. Since this paper aims to combine community detection and node embedding process, a set of baseline should be considered. For example, if considering the downstream node classification of node embedding as an evaluation task, then how about the performance of the following two-step method. We can first detect communities based on the node features then do graph node embedding by considering the communities' membership and node features together (e.g. simply concatenating both community membership features and node features).
4. Efficiency and scalability evaluations are needed. Spectral clustering has a scalability issue when meeting big graphs. Since the spectral process is also applied in the proposed method, efficiency and scalability evaluations are encouraged to provide, especially for big graphs which are not covered in the selected datasets in this paper.
5. In Sec 5.3 and Fig 2, it's mentioned that trends of the three datasets are different. For the increasing trend, how about the performance for an extreme case where all nodes are considered in one batch. On the other hand, adding more nodes in one minibatch could provide more information, but why there exists a decreasing trend? Though the authors provide a reason in Sec 5.3, it's better to analyze the reason directly from the datasets.


In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules. In this paper, we extend the L2L framework to zeroth-order (ZO) optimization setting, where no explicit gradient information is available. Our learned optimizer, modeled as recurrent neural network (RNN), first approximates gradient by ZO gradient estimator and then produces parameter update utilizing the knowledge of previous iterations. To reduce the high variance effect due to ZO gradient estimator, we further introduce another RNN to learn the Gaussian sampling rule and dynamically guide the query direction sampling. Our learned optimizer outperforms hand-designed algorithms in terms of convergence rate and final solution on both synthetic and practical ZO optimization tasks (in particular, the black-box adversarial attack task, which is one of the most widely used tasks of ZO optimization). We finally conduct extensive analytical experiments to demonstrate the effectiveness of our proposed optimizer.

The paper proposes a zeroth-order optimization framework that employs an RNN to modulate the sampling used to estimate gradients and a second RNN that models the parameter update. More specifically, query directions are sampled from a Gaussian distribution with a diagonal covariance whose evolution is determined by an RNN (QueryRNN). The resulting gradient estimates are used by an RNN (UpdateRNN) that learns the parameter update rule. This framework has the stated advantage that, unlike existing work in ZO optimization, it does not rely upon hand-designed strategies to reduce the variance common to ZO gradient estimators. The paper evaluates the proposed framework on MNIST and CFAR tasks, as well as a synthetic binary classification task. The results demonstrate faster convergence compared to baseline zeroth-order optimization algorithms, while ablations indicate the contributions of the different model components.
The primary contribution of the proposed method is the inclusion of a second network that learns to adapt the covariance of the Gaussian from which  query directions are sampled. The use of an RNN to model the parameter update rule is borrowed from previous work. The ablation studies show that the adaptive sampling strategy noticeably improves convergence as does the inclusion of the RNN update (the contribution of UpdateRNN is more significant in one ablation study, while the contribution of QueryRNN is greater in the other).
Given that a stated advantage of QueryRNN is reducing variance, it would be beneficial to compare against baselines such as Liu et al., 2018a,b and/or Guo et al., 2016 which similarly seek to reduce variance.
The paper includes a large number of typos (e.g., CIFAR-->CIAFR) and grammatical errors, but is otherwise clear.
ADDITIONAL COMMENTS/QUESTIONS
* The related work discussion would benefit from a discussion of how Liu et al. 2018 and Guo et al. 2016 reduce variance
* The computational complexity of the proposed method as compared to the baselines is unclear as is the scalability with dimensionality. At various points, the paper comments that other methods scale poorly with the dimensionality of the query space, which is true of the proposed method unless the operations are parallelized. Is this not possible with the baseline methods?
* The paper makes hand wavy claims to the fact that modulating the diagonal covariance matrix allows the method to focus on certain subspaces. It would be helpful to make these claims more formal, particularly in light of the fact that the mean does not change.
* The method relies upon a bit of a hack that samples from a standard Gaussian at random times. How important is this to performance? How sensitive is convergence to the frequency with which standard Gaussian sampling is used?
* The discussion of Eqn. 5 as it relates to Eqn. 1 is unclear.
UPDATE AFTER AUTHOR RESPONSE
I appreciate the authors' thorough response, which resolved my primary questions/concerns, including comparisons to existing variance reduction methods (which should be incorporated into the main paper).


Neural Networks for Principal Component Analysis: A New Loss Function Provably Yields Ordered Exact Eigenvectors
In this paper, we propose a new loss function for performing principal component analysis (PCA) using linear autoencoders (LAEs). Optimizing the standard L 2 loss results in a decoder matrix that spans the principal subspace of the sample covariance of the data, but fails to identify the exact eigenvectors. This downside originates from an invariance that cancels out in the global map. Here, we prove that our loss function eliminates this issue, i.e. the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix. For this new loss, we establish that all local minima are global optima and also show that computing the new loss (and also its gradients) has the same order of complexity as the classical loss. We report numerical results on both synthetic simulations, and a real-data PCA experiment on MNIST (i.e., a 60, 000×784 matrix), demonstrating our approach to be practically applicable and rectify previous LAEs' downsides.

This paper proposes and analyzes a new loss function for linear autoencoders (LAEs) whose minima directly recover the principal components of the data. The core idea is to simultaneously solve a set of MSE LAE problems with tied weights and increasingly stringent masks on the encoder/decoder matrices. My intuition is that the weights that touch every subproblem are the most motivated to find the largest principal component, the weights that touch all but one find the next largest, and so forth; I found this idea clever and elegant.
That said, I lean towards rejection, because the paper does not do a very good job of demonstrating the practical or theoretical utility of this approach. As I see it, there are two main claims that one could make to motivate this work:
1. This is a practical algorithm for doing PCA.
2. This is a step towards better understanding (and perhaps improving) nonlinear autoencoders, which do things that PCA can't.
Claim (2) might be compelling, but the authors do not make it, and it isn't self evident.
I do not find claim (1) convincing on the basis of the evidence presented. PCA is an extremely well studied problem, with lots of good solutions such as randomized SVD (Halko et al., 2009). A possible advantage of using LAEs to address the PCA problem is that they play nicely with SGD, but again, the claim that the SGD-LAE approach is superior to, say, randomized SVD on a data subsample requires evidence. Also, even if one buys the claim that LAEs are a good way to solve PCA, one can always recover the eigenvectors/eigenvalues by a final decomposition step; the authors claim that an advantage of their approach is that it does not require such "bells and whistles", but this seems like a pretty minor consideration; implementing the proposed loss function seems at least as complicated as making a call to an SVD solver, and it's hard for me to imagine a situation where the cost of that final SVD isn't dominated by the cost of solving the MSE LAE problem.
In summary, I think this paper proposes a clever and elegant solution to a problem that doesn't seem to be very important. I can't recommend acceptance unless the authors can come up with a stronger argument for why it's not just interesting but also useful.


We propose Support-guided Adversarial Imitation Learning (SAIL), a generic imitation learning framework that unifies support estimation of the expert policy with the family of Adversarial Imitation Learning (AIL) algorithms. SAIL addresses two important challenges of AIL, including the implicit reward bias and potential training instability. We also show that SAIL is at least as efficient as standard AIL.
In an extensive evaluation, we demonstrate that the proposed method mitigates the reward bias and achieves better performance and training stability than other baseline methods on a wide range of benchmark control tasks.

**Summary of the paper:
The paper proposes an IL method named support-guided adversarial IL (SAIL), which is based on generative adversarial IL (GAIL) (Ho and Ermon, 2016) and random expert distillation (RED) (Wang et al., 2019). The key idea of SAIL is to construct a reward function by multiplying reward functions learned by GAIL and RED. This multiplication yields two benefits; 1) it handles the issue of biased reward in GAIL, since state-action pairs outside the expert’s support are assigned low reward values. 2) SAIL’s reward is more reliable than RED’s reward for state-action pairs inside the expert’s support. The authors show that SAIL is at least as fast as than GAIL in terms of the sample complexity. Experiments on continuous control benchmarks show that SAIL is overall more stable than GAIL.
**Rating:
The paper proposes a simple but effective combination of existing methods. The proposed method is well motivated and performs well on benchmarks. Still, the paper has some issues regarding justification, clarity, and evaluation, which should be addressed (see below). I vote for weak acceptance.
**Major comments/questions:
- No guarantee of the optimality of the learned policy.
Can it be guaranteed that SAIL learns the expert policy? (assuming the expert policy is realizable). Propositions 1 and 2 show the convergence of the support estimation, but these results are not related to the optimality of a policy learned with the reward function. This is an important point for justifying SAIL, since SAIL does not perform distribution matching to learn the expert policy, and it also does not perform IRL to learn the reward function. Therefore, SAIL lacks the optimality guarantee from both distribution matching and IRL perspectives. Please address and clarify this point.
- Clarity in the theoretical analysis.
In the theoretical analysis, the paper assumes a rate of GAIL for support estimation. This is quite confusing, since GAIL performs distribution matching and does not estimate the support. Also, given that r_gail = -log D(s,a), the reward’s upper-bound (R_gail) is infinity and the bound in Eq. (9) is not informative.
- The reward r_red is constant at the optimal.
Eq. (2) and Eq. (3) imply that, for state-action pairs from the expert’s state-action distribution, r_red is constant at the optimal. Specifically, the optimal solution of Eq. (2) is \hat{\theta} = \theta, which yields to a constant value of r_red(s,a) in Eq. (3). In this scenario, SAIL is equivalent to GAIL for the expert state-action distribution. This means that Eq. (2) should not be optimized until optimal, and some early stopping criteria are required. Does this scenario (constant value of r_red) occur in the experiments?
- IRL baseline methods.
The paper should compare SAIL to methods which aim to handle the bias in reward function, e.g., DAC (Kostrikov et al. 2019). While DAC requires the time limit, this time limit is known in the benchmark tasks. Also, IRL methods such as AIRL (Fu et al., 2018) should be compared, since IRL methods are better than GAIL at handling bias in reward function (Kostrikov et al. 2019).
**Minor comments/questions:
- Typos: "offline RL algorithms" should be "off-policy RL algorithms". Line 5 of Algorithm 1 should perform gradient ascent instead of gradient descent. An expectation over state-action distribution of expert is missing from Eq. (2).
- What are the bold numbers in table 1 and 2 indicating? Why does the Hopper task have two bold numbers, but the other tasks have only one?
--After author response--
I have read the author response and other reviews. I thank the authors for including additional experiments. However, the authors' arguments regarding optimality do not fully address my comments (see below). I will keep the vote of weak acceptance.
The authors argue that "In this asymptotic case, SAIL is equivalent to performing distribution matching via GAIL with the additional *constraint* that candidate distributions need to have the same support of the expert distribution". However, the support of the expert distribution may coincide with the entire state-action space, which makes the additional constraint uninformative in the asymptotic case. Specifically, the expert distribution coincides with the state-action space when the expert policy has an infinite support (e.g., the expert policy is Gaussian). Assuming the asymptotic case, the support estimation in RED will give an indicator function over an entire state-action space, and the support constraint in SAIL is always satisfied. In other words, SAIL is exactly equivalent to GAIL in this case. For these reasons, the authors' arguments regarding optimality do not fully address my comments. I think additional assumptions are required, e.g., the expert policy needs to have a finite support or be deterministic.


Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Fine-tuning protects a victim against a specific adversary, but the attack method can be successfully reapplied to find a new adversarial policy. Videos are available at https://attackingrl.github.io/.

The authors start with the valid observation that for controlled systems the threat model of explicitly flipping pixels or otherwise changing the inputs directly is less relevant than the case of identifying bad inputs in the environments (which the authors rename "natural observations") that cause the controller under attack to leave its stable region and end up in bad states for itself, unable to recover sometimes. This paper is an empirical exploration of this phenomenon, using as examples trained agents in a Gym environment and involving two humanoid robot simulations interacting with each other in various ways. The original policies are taken from previous training, so the main experimental contribution here is to solve an RL problem on the attacker's side to try to find the bad regions in the victim's control space, going via the means of the observation-based coupling.
I think these are timely issues. I agree that, to the extent that DRL policies are being seriously considered as candidates for various practical problems, we should seriously ask questions about the (lack of) stability of solutions.
I also think that the paper would be stronger if the authors considered the rich history in control design, formal methods and other communities where this approach to counter-example based thinking is fairly standard and in particular where the use of optimization methods to find 'bugs' has been studied in some depth.
So, for instance, the authors give us useful quantification to support the observation that the process of attacker RL finds ways to put the victim outside its stable region where it is naturally unstable and ineffective. However, they do not observe that there is a literature on systematically performing 'controller testing' via optimization using various techniques. Here are just a few examples to show the diversity:
Ghosh, S., Berkenkamp, F., Ranade, G., Qadeer, S., & Kapoor, A. (2018, May). Verifying controllers against adversarial examples with Bayesian optimization. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7306-7313). IEEE.
Ravanbakhsh, H., & Sankaranarayanan, S. (2016, October). Robust controller synthesis of switched systems using counterexample guided framework. In 2016 international conference on embedded software (EMSOFT) (pp. 1-10). IEEE.
The authors might find it interesting to note that this approach of viewing the control problem 'backwards' to generate new instances has a deeper history in control theory (of course without discussion of NNs etc), e.g.,
Doyle, J., Primbs, J. A., Shapiro, B., & Nevistic, V. (1996, December). Nonlinear games: examples and counterexamples. In Proceedings of 35th IEEE Conference on Decision and Control (Vol. 4, pp. 3915-3920). IEEE.
All that said, I find the results presented here to be plausible and pointing in the desired direction for exploring how to robustify DRL. The authors defer training using these examples to future work but I think the paper would be more self-contained if that were actually demonstrated here already. For reasons I mention above, the existence of this kind of weakness in controllers is not really surprising to people who have thought deeply about control, and it is only to be expected that NN based parameterisations of control would only be even more vulnerable. So, the more satisfying result would be the positive one that shows how to train DRL to be (more) robust under such attacks.


Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data. However, in practice, this has been mostly an afterthought, with privacypreserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio, explicitly for privacypreserving training. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the fundamental learning procedures or differential-privacy analysis.

This paper presents experimental evidence that learning with privacy requires approaches that are not identical to those used when learning without privacy. These approaches include re-considering different model choices (i.e., its structure and activation functions), its initialization, and its optimization procedure. With these changes, they show that it is possible to obtain state-of-the-art results for some canonical learning tasks.
Strengths:
This paper questions nearly every component in the training pipeline, including choices about the model structure, initialization strategies, and optimization procedures. For each component, they show that judiciously choosing the components (which go against the standard choices in the non-private learning setting) enables training higher-utility models than in previous works without sacrificing privacy. Moreover, in addition to the experimental evidence alone, most of the components considered in the paper were accompanied by reasonable justification/hypotheses for why the choices enable such improvements.
This paper helps push differentially private learning to a more practically-useful realm. First, the suggested changes here are easy for a practitioner to understand and easy to implement. With only these simple changes, the concrete results then show that it is possible to achieve utility close to the analogous non-private model while still maintaining reasonable utility (\epsilon less than 3 with \delta of 10^-5).
Weaknesses:
My major concern with this paper lies in the experimental methodology. Specifically, most experiments are based on varying a single component while leaving all other components the same. While this is certainly the scientifically-valid way to demonstrate the component’s influence on the entire system given the other fixed components, it doesn’t convincingly demonstrate that the component has this influence across all (or at least most) reasonable configurations of the other components.
This can be made concrete using many experiments in the paper, but let’s take the activation functions experiment of 3.2 as an example. Here, it is shown that after fixing the privacy guarantee, model structure, training procedure, and hyperparameters -- the tanh activation performs better than the ReLU activation. However, suppose instead that we fix all of these components except the hyperparameters; it may then be the case that the ReLU activation is capable of outperforming the tanh activation when its hyperparameters are chosen carefully. In other words, to validly compare the two activations and reach a convincing conclusion, they should be compared against each other in their own individually-best settings (e.g., the results induced by the optimal hyperparameters for ReLU versus the results induced by the optimal hyperparameters for tanh).
This is similar to the problem addressed in Avent et al.’s “Automatic Discovery of Privacy–Utility Pareto Fronts” paper (https://arxiv.org/abs/1905.10862).
The specific technical details on some experiments were either difficult to find or were lacking. Given that this is fundamentally an experimental paper, having these details clearly listed somewhere for reference is important, even if relegated to an Appendix. Although this applies more broadly to most of the experiments, we can use Section 3 as an example again: the details on the experiment in 3.1 were found in the caption of Figure 1, whereas I would have expected them either in the main body or clearly listed in their own table; the details on the experiment in 3.2 specify that everything is identical between the tests of the two activation functions, however it is never specified exactly what is being altered (and by how much) to vary the \epsilon value.
4.2 Initialization by Weight Scaling proposes that judiciously scaling initial weights can improve model privacy/utility. This scaling is done by “transfer from one differentially private model to another”, where “DP-SGD can be applied to train a model with high utility, but less than ideal privacy” and then extracting the relevant information from there in order to initialize a new differentially private model that will be trained with strong privacy guarantees. It is claimed that “this extraction can be done in a differentially-private manner, e.g., as in Papernot et al. (2018), although the privacy
risk of summary statistics that drive random initialization should be vanishing”. It is unclear to me how this extraction of summary statistics should be done in such a way that doesn’t consume a significant portion of the privacy budget. If there is such a way, it should be clearly stated and its effect on the privacy budget should be explicitly incorporated into this paper’s results.
Minor: The statement that “Such accuracy loss may sometimes be inevitable” on page 1 should include a reference; e.g., Feldman’s “Does Learning Require Memorization? A Short Tale about a Long Tail” paper (https://arxiv.org/abs/1906.05271).
Overall, this work provides good practical guidance to practitioners and researchers who wish to do differentially private machine learning. However, given the lack of theoretical novelty, the experimental methodology needs to be improved in order to significantly strengthen the results (assuming they continue to hold).
----------------------------------------------------
Update: Due to the authors' writing clarifications and experimental additions, in conjunction with the concrete and realistically-applicable insights from the paper, I've modified my rating to a Weak Accept.


In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning NNs are convex in the canonical model space. We further elucidate that the gradients between the original NN model space and the canonical space are related by a pointwise linear transformation, which is represented by the so-called disparity matrix. Furthermore, we have proved that gradient descent methods surely converge to a global minimum of zero loss provided that the disparity matrices maintain full rank. If this full-rank condition holds, the learning of NNs behaves in the same way as normal convex optimization. At last, we have shown that the chance to have singular disparity matrices is extremely slim in large NNs. In particular, when over-parameterized NNs are randomly initialized, the gradient decent algorithms converge to a global minimum of zero loss in probability.

Summary:
This paper presents an argument that massively overparameterized neural networks trained with gradient descent on a supervised learning problem with convex loss function will converge. It argues that by mapping each model to a truncated Fourier series we can recover a canonical representation for the functions in the model space. Viewing this as a linear map at each point in parameter space and assuming that this matrix is always full rank, they claim convergence of gradient descent.
Decision:
This paper should be rejected because it lacks originality, the assumptions are often too strong, and the rigor of some claims is questionable.
Main argument:
Originality:
This is the main issue with the paper. Several papers, for example the ones from Allen-Zhu et al. and Du et al. cited in the paper, have shown that gradient descent on supervised learning provably converges for massively overparameterized neural networks. Not only that, but those works give rigorous proofs of their claims and even handle issues like step size, convergence rates, and precise conditions on the size and architecture of the neural network. This paper does not attempt to handle any of these details, claiming to simplify the proof. To me the proof is only simplified since the details are all omitted.
Assumptions:
The main way that details are avoided is by adding a strong assumption that the “disparity matrix” remains full rank over the course of training. This assumption is not supported in an empirical or formal way, but rather by appeal to some vague argument in section 3 that since the matrix is likely to be full rank at initialization, it is likely to remain so for large enough networks.
Rigor:
As explained above, many details are omitted in favor of strong assumptions, but there are also some technical details that I think may be wrong. On page 2, it is claimed that the mapping from Lambda_M to L^1 is surjective, this is not true. Every function in L^1 can be approximated by some function in Lambda^M, but for finite M this map cannot be surjective. Another similar issue arises in the proof of Theorem 1 when it is claimed that there is a unique theta_epsilon corresponding to each f. This is again false since by truncating the Fourier series, infinitely many functions will get mapped to the same truncation (when they only differ in the higher coefficients). These issues make me question the validity as well as the originality of the arguments presented.
Additional feedback:
- I saw a few spelling and grammatical errors. For example: in the abstract the tenses alternate between present and past almost every sentence (“we have proved” should be “we prove”),  page 2 “parametarize”, page 3 “serie”, page 5 “distint”, page 7 “Liptschitz”, section 3.3 title should be “when does” not “when do”.


Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations in a knowledge graph. A vast number of KGE techniques for multi-relational link prediction has been proposed in the recent literature, often with state-of-the-art performance. These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization. In this paper, we take a step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance. We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings. We found that when trained appropriately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results. For example, RESCAL (Nickel et al., 2011), one of the first KGE models, showed strong performance when trained with state-of-the-art techniques; it was competitive to or outperformed more recent architectures. We also found that good (and often superior to prior studies) model configurations can be found by exploring relatively few random samples from a large hyperparameter space. Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual benefits.

Summary
========
The paper conducts a thorough analysis of existing models for constructing knowledge graph embeddings. It focuses on attempting to remove confounding aspects of model features and training regime, in order to better assess the merits of KGE models. The paper describes the reimplementation of five different KGE models, re-trained with a common training framework which conducts hyperparameter exploration. The results show surprising insights, e.g., demonstrating that a system from 2011, despite being the earliest of the KGE models analyzed, demonstrates competitive results over a more recent (2017) published model.
Overall Comments
===============
The paper, and the described software release specifically, represent a solid contribution to the area of knowledge graph embeddings. I agree with the basic premise of this paper’s analysis: in order to accelerate research in a maturing field (like knowledge graphs), it is important to be able to properly compare with older systems, removing artifacts that are due to general improvements in training and optimization techniques, from modeling specific changes. The report of the strong results from the RESCAL system, along with others, drive the point through. Furthermore, the paper is well-written and easy to follow, and should become a good reference for future works on KGEs.
Detailed comments
===============
Below are some detailed comments about specific parts of the paper, in order of importance:
1. The paper mentions disregarding “monolithic” models in the current analysis, primarily due to the expensive training of these models. It may, however, be the case that the future state-of-the-art models will be larger and slower to train (and, perhaps, of the monolithic type). Are there any limitations to the proposed experimental framework that would prevent running monolithic/large models?
2. Regarding the item above, if one were to look at the training curves for the exploration of the current 5 KGE models, is it possible that verify winning hyperparameter configurations earlier than the full training is complete. In my experience, it is often the case that with fewer than 1/10th steps of full training (well before convergence), it is possible to compare model configurations (relatively). For example, “Population-base training” (https://arxiv.org/abs/1711.09846, https://arxiv.org/abs/1902.01894) is one framework where fewer training steps are used to quickly learn good hyperparameter configurations. I’m wondering whether the KGE hyperparameter exploration training curves display similar early trends. Could a shortened training procedure produce sufficient information for learning good parameters, and potentially deal with larger/slower models?  In addition: would adopting population-based training be applicable to the proposed framework?
3. In Section 3.2, “Limitations”, there is a surprising comment that performance can be improved with further hyperparameter tuning. It is not clear how the authors found the configurations that produced the improved results. It would be helpful to clarify why the hyperparameter exploration proposed in the paper did not discover these improved configurations. Were the improved configurations outside of the range of considered values? Or would the exploration require more points to find the improved configuration?
4. In Section 3.3 “Best configurations (quasi-random search)”, specifically Table 3, the paper presents an ablation of independent hyperparameters, over the best configuration for each of the 5 models. This is a very interesting section. One further suggestion, however, is whether the paper could include the performance of each of the models on the _average_ best configuration. Although the paper describes losses for switching individual parameters to their second best values, it is unlikely that the losses are cumulative. So, for example, if we can take the average/majority best value for each parameter (embedding size = 512, batch size = 1024, training type = 1vsall, loss = CE, etc.), and collect results for that configuration. I think it would be interesting to know the difference between a model trained on a “collectively known good” set of parameters vs. a model and task specific tuned set of parameters.
5. In Section 2, “Evaluation”, HITS@k is not formally defined. Unfortunately, I have encountered slight variants of this metrics (e.g: (1) given a SINGLE correct label, HITS@k is the average rate of the label being present in the top k scored results, or (2) given ALL possible correct labels, HITS@k is the percentage of correct labels present within the top k scored results, etc.). It would be nice to precisely describe HITS@k in this work.
6. Caption for Table 2 does not contain a description for the “Recent” super-column.
7. In Section 3.3 “Best configuration (quasi-random search)” Space missing at “... Tables 6 and 7(in …”, between 7 and (.


We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, i.e. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

This paper conducts a series of experiments on the multilingual BERT model of Devlin et al., aiming to inject stronger bilingual knowledge into the model for improved 'Aligned BERT'. The knowledge originating from parallel (Europarl) data improves the model significantly as shown on tasks such as contextual and non-contextual word retrieval as well as in zero-shot XNLI task. The paper continues the line of work on cross-lingual contextualised word embeddings, and it brings several minor contributions, but overall I do not see it as a very inspiring piece of work, and it leaves open several very important questions, in particular its relationship to prior work and some potentially stronger baselines than the ones reported in the paper, plus more experiments with more distant language pairs.
I am not exactly sure that the comparison between 'Aligned BERT' and the main baseline 'Aligned fastText + sentence' is completely fair. 'Aligned BERT' uses more than 2M Europarl sentences to learn the alignment, while the standard alignment methods for learning cross-lingual word embeddings (see e.g. Ruder et al.'s survey) typically rely only on 5k translation pairs or even less pairs. There is a huge difference in the strength of the bilingual signal between 2M parallel sentences and, say, 2k, word translation pairs.
The main goal of the paper is to improve alignment of the starting multilingual BERT model, but I wonder why the authors have not compared to a more suitable XLM baseline of Lample and Conneau (NeurIPS 2019; the paper has been on arXiv since January 2019) - the XLM model uses exactly the same resources as 'Aligned BERT': parallel sentences from Europarl, while the main baseline here uses only seed dictionaries to learn the mapping. Regarding the baselines, it is also not clear to me why the authors have not compared to previous work of Schuster et al. (2019) and Aldarmaki and Diab (2019) at least in tasks where the models can be directly compared (XNLI or non-contextual word retrieval). Also, another non-contextual model which is worth trying is a joint model which relies on parallel sentences (similar to Ormazabal et al., ACL-19).
For the 'Aligned fastText + sentence' baseline, it would be interesting to report numbers with another (hybrid) baseline model that combines aligned fastText vectors with sentence encodings produced by multilingual BERT or some other multilingual sentence encoder (such as LASER, see Schwenk et al., 2019). Simply taking min, max, and avg vectors over all the sentence words might not be the best way to encode the sentence, and I would like to see more experiments here.
The paper makes some claims on novelty which 1) partially overlap with prior work, or 2) it does not cite related work while it leans on its findings. For instance on Page 4, the authors claim that their "(...) alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices." However, there is at least one previous paper (Heyman et al., NAACL 2019) which did the same thing as the authors and showed that departing from learning projections only to English leads to more robust multilingual embeddings. Further, also on Page 4, the authors discuss that the assumption on learning good rotation matrices relies on the assumption of rough/approximate isomorphism without citing a body of related work that actually investigated this assumption such as the work of Sogaard et al. (ACL 2018). Also, the paper should do a better job in Section 2 and cover "word vector alignment" in more detail (e.g., a good starting point might be Ruder et al.'s survey paper on cross-lingual word embeddings).
The assumption of rough/approximate isomorphism is problematic also for non-contextual cross-lingual embeddings in settings with more distant language pairs. The authors mention that it may not hold for 'contextual pre-trained models given their increased complexity'. This is very imprecise writing taking place imho: 1) it is not clear why it should not hold in the case of contextual pre-trained models (at least for similar languages). Are there any properties of the contextual models that invalidate that assumption? It is also not exactly shown why contextual pre-trained models have increased complexity compared to e.g. fastText. How does one measure that 'model complexity' in objective terms? In fact, the paper would contribute immensely from more precise writing: e.g., on Page 3 contextual alignment of the model f is defined as accuracy in contextual word retrieval. This reads as defining a critical concept or a task as an evaluation measure (that measures the success of that task). In Introduction, the paper aims to "better understand BERT’s multilingualism", but I do not see how it contributes to our better understanding of BERT's multilingualism besides a pretty straightforward claim that it shows less multilingual potential when doing experiments with Greek and Bulgarian that use different scripts. Figure 2 and Figure 3 also do not bring anything new - the paper seems to just state known facts without proposing new solutions on how to e.g. learn better alignments for Greek or Bulgarian.
One important analysis aspect is missing from the paper: there are no experiments with more distant language pairs (the most distant language pair is English-Greek). I would like to see more experiments in this space. Another experiment which would contribute to the paper is the analysis of the importance of parallel corpora size. How much does the model lose in its performance by shrinking the parallel corpus? We cannot expect having 2M sentences for so many language pairs, and, even if we do have the data, the paper does not convince me that I should use 'Aligned BERT' instead of e.g. the XLM model of Lample and Conneau.
Minor remarks:
As a variant of the contextual word retrieval, have the authors tested if a correct target language sentence can be retrieved only looking at the context of the source language word? This would provide some insight on the importance of modeling context via BERT versus via simple context averaging.
Regarding the analysis between closed-class and open-class words performance, the difference in performance can be due to mere frequency: closed-class word types are very scarce, but their corpus frequency is quite high which also leads to learning better representations in the first place, as well as better alignments later on.


Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However, it is an open question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work, we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. Therefore, we propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks, we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data-efficient.

This paper proposes a simple yet effective method to adapt large-scale pre-trained language models, which have been shown to substantially improve performance on broadly classification-based NLU tasks, to NLG. The approach is explored in the encoder-agnostic {X}-to-text setup, where the source encoding {X} could represent arbitrary modalities, such as text or images.
More concretely, the paper leverages a pre-trained, large-scale language model (in this case a GPT-2), and examines how to best cast such unconditional language model into a decoder that generates text conditional on the source information {X}. As self-attention inherently works with sequences of any length, the proposed pseudo self-attention approach simply injects the encoder representation as additional conditioning context (using some additional projection matrices that are learned from scratch) into the pre-trained self-attention layers of the decoder. Extensive experiments and analysis on four diverse tasks demonstrate that pseudo self-attention generally outperforms two other ways of pre-training the decoder, improves NLG data efficiency, and produces texts that are judged more favourably by human evaluators.
Overall, this paper presents a simple, general, and effective method for adapting large-scale pre-trained language models to conditional text generation. Based on the pros and cons that I have listed below, I am giving a rating of "Accept". I hope that some of my concerns will be addressed in the authors' response.
Pros:
1. The paper is well-written and the methodology is explained very clearly. Figure 1 is particularly helpful in illustrating the differences between pseudo self-attention and the baselines.
2. The paper addresses a very important problem, and helps make sure that the advances that have been made in language modelling (which can leverage large amounts of unlabelled data), would transfer well to conditional text generation tasks, which hold immediate practical value yet often require expensive annotations.
3. The approach is simple and easy-to-implement, but has been shown to be effective across a broad range of problems, multiple modalities, and various evaluation metric.
4. The paper features extensive reference to relevant prior work, and clearly highlights the key similarities and differences with prior approaches.
Cons:
1. It is still unclear how using language model pre-training affects adequacy (as opposed to fluency). The paper shows that using pseudo self-attention results in a decoder that diverges less from its language model initialisation. One potential risk is that the decoder may prefer fluent, "safe" outputs (which is arguably what a language model would prefer since it is an unconditional model) that are nevertheless less faithful to the source information. Since none of the evaluation metric specifically assesses for adequacy on its own, it would be good to isolate the effect of pseudo self-attention on adequacy, and compare it with the baselines, in addition to a Transformer trained from scratch on each downstream task. How to measure adequacy is naturally still an open question, but there are a few things that can be done (e.g. recall of salient information, reverse perplexity to see how much of the source information can be "reconstructed" given the predicted target text, etc.).
2. It would be interesting to further examine the interaction between encoder pre-training and the decoder pre-training that is explored in this work. Another interesting experiment to run is whether end-to-end training (including fine-tuning the encoder) would help, since prior work has shown the benefits of end-to-end learning (at least when large amounts of data are available).
Questions:
1. Why is the context-attn model performance not included in Table 6? Is it because of the optimisation issue associated with that model in Table 3?
2. In page 7, it is mentioned that "Both models have similar attention distributions ... at the first layer, which precedes the introduction of new parameters in both models". Does the first layer here refer to the token + position embedding layer?


Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new stateof-the-art performance. Source code is available for reproducibility.

This paper revisits the way RL algorithms are typically evaluated on the ALE benchmark, advocating for several key changes that contribute to more robust and reliable comparisons between algorithms. It also brings the following additional contributions: (1) a new measure of comparison to human performance based on actual human world records (which shows that RL algorithms are not as « super-human » as is generally believed), and (2) an evaluation (based on the proposed guidelines) of Rainbow as well as a Rainbow-IQN variant (replacing the C51 component of Rainbow with Implicit Quantile Networks), showing that the latter brings a significant improvement upon the original Rainbow algorithm.
Overall I am leaning towards acceptance as I believe that such papers encouraging better benchmarking practice on Atari are definitely needed. Even if the technical contribution is limited, this paper could have a positive impact on the field by providing a clearer picture of the current state of deep RL algorithms on Atari (assuming that other researchers start following these recommendations -- and if that is not the case at least it will highlight issues with the way evaluation is currently done).
I do have a few concerns / questions though:
1.	I am not convinced by the recommendation to use performance during training for evaluation purpose. In Machado et al. (2018) it is argued that « this better aligns the performance metric with the goal of continual learning », but most deep RL algorithms trained on Atari games have not been intended to be used in a continual learning setting. It definitely has the advantage of being simple, but it seems to me that it can cause some issues, like making it difficult to compare different exploration techniques for off-policy learning (the exploration may cause poor behavior during training even if it helps the agent learn a better greedy policy), and more generally not being representative of the common practical use case where the goal is to obtain the best agent possible to use in production (with no further learning). Finally, it could make results even harder to reproduce due to the potential high variance of an agent’s performance at a fixed # of timesteps (vs. considering the max performance it can reach over the whole period). As a result, I am currently reluctant to see the proposed performance measure become the standard evaluation metric on ALE, and I would appreciate some additional justification from the authors on this point.
2.	Why not suggest to remove reward clipping in the recommendations? As mentioned in Section 6, reward clipping can prevent RL algorithms from properly playing some games, and thus in my opinion should be removed if the goal is to reach the highest score possible on all games. It seems to me that the choice of clipping the reward should be part of the algorithm (if it is not able to handle the high variety of « raw » rewards) and not of the benchmark environment, thus enabling further advancements towards algorithms that are robust to a wide range of rewards.
3.	Why bother to keep the mean performance when, as mentioned, it is highly sensitive to outliers compared to the median?
Additional remarks:
•	I might have missed it but I do not see the link to the source code. Am I correct to assume it will be released, to help with reproducibility?
•	It is not clear, when reading the paper, that the distributed version of Rainbow is actually constrained to mimic a single agent sequential algorithm in the experiments. I would suggest to remove mentions of the distributed version in the main text to avoid confusion, and mention it only in the Appendix section where it is used.
•	The « infinite reward loop » point at the end of Section 6 does not seem relevant in the list of reasons why Deep RL algorithms are far from the best human performance, since with infinite playtime and an infinite reward loop, the algorithm should be guaranteed to outperform humans.
•	I would have appreciated an evaluation of Rainbow-IQN with the current most commonly used evaluation schemes (e.g. the one used in the original Rainbow paper), for comparison purpose (even if such an evaluation has flaws, it is often the only performance measure available for existing deep RL algorithms)
Review update: thank you for the response, I am currently keeping my "Weak accept" rating because I agree it is important to highlight and (try to) fix the problems with the way algorithms are currently evaluated on ALE, in spite of the limited technical contributions (and the fact that I remained unconvinced regarding #1)


Experimental reproducibility and replicability is a critical topic in machine learning. Authors have often raised concerns about such scholarship issues, which are aimed at improving the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to reevaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.

********** Post-Rebuttal Update **********
I appreciate that authors provided comments on all the raised concerns and updated the paper accordingly. I think the revised paper is of a higher quality.
Although the paper's experimental setup could be done in a more proper way -- nested CV; lack of which degrades the conclusiveness of the comparisons, I still think the paper is better to be accepted than not, given that 1) as pointed out by the authors the computational cost of the current experiments was quite high already, 2) the issues of replicability and reproducibility are important and GNNs are quite popular for various applications and 3) the results are interesting and important to be considered for future GNN works. So, I am happy to change my rating to weak accept. However, for the record, I would like to mention the following point:
The authors removed the word nested from the paper which is good since I still believe it does not correctly reflect what’s been done in the work. However, in the reply, they imply that several works including Varma & Simon, 2006 [1] would consider their method a nested CV. They attribute this different viewpoint to the reviewer’s misunderstanding of CV for *k-fold* CV. That is not true. CV requires “cross” testing which is absent from the inner loop of this work. I refer the interested readers to actually the mentioned paper by the authors: Varma & Simon, 2006 [1], sectio:background. Page 2 clearly defines cross validation requiring a for-loop for several evaluation across different training/test sets and page 3 defines nested CV as requiring an inner loop and  outer loop of this kind. So, as far as I understand, based on [1], the proposed method is not nested CV since the inner (model selection) part uses the classic single train/test split without any loop.
********** Summary **********

The paper conducts an empirical study of 5 recently-proposed graph neural networks (GNN). Three questions are studied:
1) For some selected set of classification tasks (chemistry and social networks) and datasets (9 different datasets), how do the considered GNN models perform, relative to each other, given the same hyperparameter search strategy and a “nested” cross validation scenario.
2) How much does the structure (graph edges) bring on top of a multi-layer perceptron (MLP) operating on merely the node features?
3) Do the considered GNN models exploit structural information of the input graphs beyond that of the node degrees?

********** Strengths and Weaknesses **********

+ it is shown that the inclusion of nodes’ degree in its feature representation brings a very large performance boost. This is very interesting.
+ it is shown that on some datasets, the results of a simple baseline -- only operating on the node features, can achieve similar results to the elaborate GNNs. This result is very informative and suggests that this baseline should always be included in GNN works.
+ some of the results in table 4 contradicts the corresponding papers which can be informative for the practitioners of the field.
+ the reproducibility and replicability problems, that is the motivation of this work, are important concerns of the field.

- reproducibility and replicability problems is not specific to research done on graph neural network and is a caveat for the general machine learning research as discussed by Lipton and Steinhardt 2018., and in fact for the current scientific practice at large as pointed in the recent NSA report [Reproducibility and Replicability in Science, 2019]. So, it may be important to raise and investigate this in different subfields of machine learning, such as GNN, individually. However, the current abstract and introduction of the paper (before the last sentence of intro’s second paragraph), associates this problem specifically to the research conducted in GNN. I strongly believe the better formulation is to refer to Lipton and Steinhardt 2018 as the troubling trend in ML as well as the NSA report in general and then mention that in the current work the authors focus on the subfield of GNN *classification*.
- page 1: “For instance, it is often unclear how hyper-parameters have been selected or which validation splits have been used.”. These being *often* the case in GNN research is not backed up by statistics and is not shown to be specific to GNN. Even if it’s assumed the 5 GNN methods , under this paper’s scrutiny, are representative of GNN classification, GNNs are used well beyond graph classification.
- it should be clearly discussed what are the additional information that this work brings on top of Shchur et al. 2018. Is it only the shift of focus from node classification to graph classification?
- the two notions of “fairness in comparison” and “ablation studies” are sometimes conflated. For instance, the argument against “fairness in comparison” of using one-hot encoding in GIN is not a matter of fairness (since it’s a novel proposal for node features of GNNs) but rather a matter of “ablation studies” (to show the effectiveness of individual components of a new GNN method).

- many concerns regarding the nested cross validation:
a) it is important to note that the paper (as described in algorithm 2) does not do “nested cross validation” despite the claim in various parts of the work. Algorithm 2 conducts model selection based on a fixed train-val split. As such, there is no “cross” validation even with the minimum of 2 folds.
b) the standard deviations reported in table 3 and 4 are unreasonably high. This important observation is not properly discussed. This high std is in contrast to the standard deviation reported in the baseline papers and makes most methods fall into one-std interval of each other. I strongly suspect that this is due to the non-nested cross validation that is done in this work. That is, the hyperparameters are found using a single val set consisting of 9% of all data. A proper nested cross validation would test each set of hyper parameters against k_in validation folds to make a robust inference of best hyperparameters.
c) there are 3 runs used “to smooth the effect of unfavourable random weight initialization on test performances”, however the 10-fold outer CV should take care of this to some extent. I would argue it would have been much more helpful to use this budget to do a proper CV for model selection using 3 folds.
d) The datasets are quite small (600-5000 samples). That might make the 10-fold cross validation problematic and overly optimistic in its performance report (and high variance). It might have been better to do say 5-fold cross validation but repeat the whole process twice. Of course, this is an old dilemma in the pratice regarding the bias and variance of the estimated CV error but as far as I am aware there is a general agreement that getting “closer” to leave-one-out setup is not a good idea, Furthermore, in my anecdotal experience 10-fold CV for a dataset with only 600 samples can really be problematic.

- would the ranking of GNN methods change with different types of the node features being used?  Some GNN setups might be better at representation learning and thus should work better on raw features (as opposed to hand engineered ones).

- how is the number of parameters (or capacity) across different GNN models and the baseline? Has this been taken into account for a fair comparison? For instance, could it be that higher/lower number of parameters explain the differences across GNNs and baseline due to over/under fitting?

********** Disclaimer. I did not thoroughly check the paper’s report of the 5 GNN methods (summarized in table 1).

********** Final Decision **********
I believe the paper has merits as well as interesting findings. It is also true that this is an important concern in machine learning research. However, there are many issues that degrades the quality of the paper. Of all those critiques, I suggest “Weak Reject” mainly due to the ones raised regarding a proper nested cross validation; which is the main proposal of the paper.

********** Minor Points **********
- Page 1: “Our results put on a fair and unique reference scale many published results which, as we document, were obtained under unclear experimental settings.“: please rephrase; at least one preposition is missing.
- Page 3: Nested Cross Validation: the brief textual explanation makes the simple idea more complicated than it is. Instead, I suggest to use citation as well as a small figure or short algorithm describing it since it’s the focus of the paper. It will also serve pedagogical purposes.
- Table 1: precisely define the two mark “A” (ambiguous) and “-” (lack of information).
- page 7: “higher or equal than” → “higher than or equal to”
- in page 6, it’s mentioned that some experiments took more than 72 hours. This sounds excessive for such small datasets. My guess is that this is due to the fact that large number of epochs are allowed (e.g. 5000 in table A.3) in conjunction with extremely small learning rate (e.g. 1e-6). The question is if those extreme hyperparameter settings are actually important for this study?
- the first paragraph of 6.1 argues that “GNNs are still unable to exploit the structure on such datasets [D&D, PROTEINS, ENZYMES]”. While this might be true, the conclusion is only based on 5 methods and limited by the design choices such as the architecture. It should be toned down.
- page 5, “Features” paragraph is not very clearly written. For instance, from the sentence: “More in detail, in the former nodes ...”, it’s hard to understand what “former” refers to.
- mean and standard deviation should have the same number of decimals (table 3,4)
- page 4: “Moreover, the authors applied early stopping, which entails the use of a validation set“. Early stopping, in a less common setup, can be used by only looking at the training set and stopping with the same n-patience strategy as used in this work.
- page 4: “we conform to the available code and do not use sampled neighborhood
Aggregation.“: needs more explanation.

********** Points of improvements **********
The main point of improvement would be a proper nested cross validation setup.
The authors can also consider the ReScience journal: https://rescience.github.io/