Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019)  showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that the phenomenon also occurs in practice, corroborating our analysis. Our code is publicly available 1 .

The paper claims that 1. multi-head self-attention(MHSA) is at least as powerful as convolutions by showing that a CONV can be cast as a special case of MHSA and 2. that in practice, MHSA often mimic convolutional layers.
These claims are interesting and timely, given that there has been a fair amount of recent work that have explored the use of self-attention(SA) on image tasks, either by composing SA with convolutions or replacing convolutions altogether with self-attention (examples of each are referenced in the paper). So should these claims be true, they would give theoretical evidence that SA can completely replace convolutions.
However, I think that the claims are exaggerated and misleading.
1. The theory shows an arguably contrived link between self-attention and convolution. Theorem 1 says that a convolution can be seen as a special case of MHSA, and the constructive proof (that chooses SA parameters to derive a convolution) shows a correspondence between the output of each head of MHSA and a D_out by D_in linear transform applied to the D_in features of a single pixel, with attention weight given entirely to this pixel (i.e. hard attention). The derivation relies heavily on the use of a relative encoding scheme that sets W_qry=W_key = 0 (usually referred to as W^Q, W^K in the self-attention literature, the linear maps applied to the queries and keys) i.e. the attention weights do not depend on the key/query values, but only their relative positions. Moreover, the softmax temperature (an interpretation of 1/alpha) is set arbitrarily close to 0 to make the softmax saturate and attain hard-attention. With these two constraints, I am sceptical as to whether you can really say that you are implementing self-attention. In standard practice when MHSA is used, W^Q and W^K are never set to zero, and the scale of the logits for the self-attention weights are controlled by normalising them with sqrt(D_k) (or sqrt(D_k/N_h), depnding on how you choose to deal with multiple heads). Furthermore, the derivation only holds for when stride=1 and padding=“SAME”, such that the spatial dimensions of the input (H & W) remain unchanged. In fact the padding is not really dealt with in the derivation, and it is unclear whether the result can generalise to convolutions with stride > 1, making the claim “MHSA layer … is at least as powerful as any convolutional layer” problematic. Hence although I think the derivation is mathematically correct, I think that the link that the derivation makes between convolutions and MHSA is somewhat contrived and not a useful observation in practice. I expect MHSA with learned W_qry and W_key will behave differently to when they are set to 0, and it would be much more interesting/relevant to see how their behaviour compares with convolutions in this more realistic setting.
2. The heavy dependence of the experiments on the quadratic encoding, the aforementioned contrived form of MHSA that was used to derive the link between convolutions and MHSA, makes the results not very relevant and the claim that "MHSA often mimic convolutional layers" rather misleading. It could be more relevant if quadratic encoding can replace standard MHSA parametererisations with learned W_qry and W_key, but I’m not convinced that this is the case. Although Figure 4 suggests that this SA with quadratic encoding gives similar test performance to ResNet18, I think that CIFAR-10 classification is too simple a task to claim that quadratic encoding can replace standard SA with learned W_qry and W_key, and I think results can look very different for harder problems e.g. ImageNet, MSCOCO - explored in Ramachandran et al - made possible because they use local SA as opposed to full SA. Experiments on these problems would be much more interesting and relevant. Note that the experiments using the learned relative positional encoding have “attention scores (are) computed using only the relative positions of the pixels and not the data” (I’m guessing this means W_qry=W_key=0 again). Hence the qualitative similarities between MHSA and convolutions only hold for the rather restricted case where I get the impression that self-attention has been unrealistically constrained only to increase its chance of behaving similarly to convolutions. Also the comparison in Figure 4 and Table 1 is being used to support the claim that self-attention can be as “powerful” as convolutions, but I think this is misleading because both quadratic and learned SA uses full SA, where each pixel attends to all pixels - this means the time & memory complexity of the algorithm is O((HW)^2), whereas for convolutions it is O(HW). So the expressiveness of SA that matches convolutions for this particular problem comes at a significant cost, to the extent that for bigger problems (ImageNet, MSCOCO) full SA is not feasible due to its quadratic memory requirement, whereas convolutions don’t face this problem. I think this should be pointed out more explicitly in the text, and think the claim that “self-attention is at least as powerful as convolutions” should be replaced with a more moderate statement such as “self-attention defines a family of functions that contains convolutions (of stride 1)”
Summary: Although the writing of the paper is clear and the derivation is mathematically correct as far as I can see, the link between self-attention and convolutions in the paper are fairly contrived, hence the contribution of the paper to the field is not so significant in my honest opinion.
********************
I appreciate the authors' response, and understand that the maths suggests a single head of MHA (in the original form) cannot exactly emulate a general convolution. But empirically, the localised attention patterns do seem to suggest that each head can behave similarly to a restricted form of convolution, where similar weights are given to the receptive field (the local patch) in the neighbourhood each input pixel. Perhaps an analysis of what special case of convolution each head can emulate would be interesting, given the empirically observed similarities in the qualitative behaviour.
With the more justified nuance of the findings of the paper, and together with the authors' significant efforts to make the evaluation more relevant and thorough, I will increase my score to "weak accept".


Generative adversarial networks, or GANs, commonly display unstable behavior during training. In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs. In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisfied by the divergence that is minimized by the GAN and the generator's architecture. We find that existing GAN variants satisfy some, but not all, of these conditions. Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulfills these conditions simultaneously. In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.

This paper provides a unified theoretical framework for regularizing GAN losses. It accounts for most regularization technics especially spectral normalization and gradient penalty and explains how those two methods are in fact complementary. So far this was only observed experimentally but without any theoretical insight. The result goes beyond that as the criterion could be applied to general convex cost functional.
The main general theorem is Theorem 1 which states 3 conditions on the optimal critic and 2 others on the generator. The paper is mainly concerned by the conditions on the optimal critic and show that the first 2 conditions can be achieved by the Spectral normalization, while the last one can be achieved by some gradient penalty.
The paper is clearly written, well structured and pleasant to read.
I have the following two remarks:
	- Proposition 8 provides a way to ensure condition 2 holds (beta-smoothness). It requires spectral normalization and smooth activation functions. In practice, while the spectral normalization is important, the choice of the activation is not in general 1-smooth (Leaky-relu for instance). Does it really matter in practice?
	Some illustrative experiments could be beneficial to better understand what's happening.
	- Is it that hard to obtain generators that satisfy condition G1 and G2, it seems to be a natural consequence on the regularity of the mapping f? If that is the case, it might be worth better explaining how this is challenging.

Limitations: The paper considers only the setting where the optimal critic is reached and therefore it is still unclear if the analysis carries on to the training procedures used in practice (non-optimal critic). The authors recognize this limitation and leave it for future work.
Overall, I feel that the paper provides good insights on what regularization is important for training gans and why. For that reason, I think this paper should be accepted.
------------------------------------------------------------------------------------------------------------
Revision:
I think the paper provides a good theoretical contribution in terms of interpreting many of the tricks used for improving GAN training. In fact the paper also suggests some new regularization methods (prop 13 for conditions D3)  which would constrain the RKHS norm of the critic. The authors show how it is related to gradient penalty, in a particular case, but the result also suggests something more general. For instance [1], consider an abstract RKHS space containing deep networks and provide an upper-bound on the rkhs norm of such networks in terms of the spectral norm of their weights and a lower-bound in terms of its Lipschitz constant.
I do agree with reviewer 1 that a better discussion of the connection to [2] should be included since that paper was interested in  ensuring weak continuity of the loss, which can be thought of as  a first requirement to get more regularity of the cost functional.
I still think the paper is worth being accepted and raised my score to 8 as I think the authors addressed the major concerns that were raised.
[1] A. Bietti, G. Mialon, D. Chen, and J. Mairal. A Kernel Perspective for Regularizing Deep Neural Networks.
[2] Michael Arbel, Dougal Sutherland, Mikołaj Binkowski, and Arthur Gretton. On gradient regularizers for MMD GANs. In Advances in Neural Information Processing Systems, pp. 6700–6710, 2018.


Deep learning based approaches have been widely used in various urban spatiotemporal forecasting problems, but most of them fail to account for the unsmoothness issue of urban data in their architecture design, which significantly deteriorates their prediction performance. The aim of this paper is to develop a novel clustered graph transformer framework that integrates both graph attention network and transformer under an encoder-decoder architecture to address such unsmoothness issue. Specifically, we propose two novel structural components to refine the architectures of those existing deep learning models. In spatial domain, we propose a gradient-based clustering method to distribute different feature extractors to regions in different contexts. In temporal domain, we propose to use multi-view position encoding to address the periodicity and closeness of urban time series data. Experiments on real datasets obtained from a ride-hailing business show that our method can achieve 10%-25% improvement than many stateof-the-art baselines.

Summary: This paper proposes a clustering attention-based approach to handle the problem of unsmoothness while modeling spatio-temporal data, which may be divided into several regions with unsmooth boundaries. With the help of a graph attention mechanism between vertices (which correspond to different regions), the CGT model is able to model the (originally unsmooth) cross-region interactions just like how Transformers are applied in NLP tasks (where words are discrete). Experiments seem to suggest a big improvement when compared to baselines.
Pros:
+This should be one of the first works that apply a graph transformer alike method in this domain, and specifically on the unsmoothness problem.
+ Since the dataset is not publically available, there aren't many prior works to compare the CGT to. However, at least compared to the one prior work [1] that the authors point to in Section 4, the RMSE results achieved CGT does seem to be significantly better.
========================================
However, I still have some questions/concerns on the paper, detailed below.
1) The current organization of the paper, as well as its clarity, can (and should) be significantly improved. I didn't completely understand the approach on my first two passes, and I **had** to read the code published by the authors. Here are some issues that I found:
  - For one, Figure 2 is not quite helpful as it's too messy with font size too small. A similar problem is with Figure 4 which, without further clarification (e.g., of what "atrous aggregation" exactly mean), is very hard to interpret.
  - The notations are very inconsistent and messy:
      i) In Eq. (1), you should use a symbol different from $\mathbf{X}$ to refer to the "predictions". Since you are applying $f(\cdot)$ on $\mathbf{X}_{t-T_x+1:t}$, you should not get the "exact same" target sequence. That's your target. Maybe use $\hat{\mathbf{y}}$, which you used in Eq. (8).
      ii) In Figure 3, what is the orange line? In addition, I only saw two blue lines in the figure, but the legend seems to suggest there are four of them...
      iii) The notations used in Figure 4 are somewhat confusing. For example, what does "f->1" mean? (I later found through Eq. (2) that it means transform to 1 dimension; but the small plots in Figure 4 suggest f is a "magnitude" of the feature.) In addition, there are two $H_1$ in Figure 4 with clearly different definitions.
      iv) The authors used $\mathcal{G}_{\theta_k}(x_i)$ in Eq. (3) without defining it. The definition actually came much later in the text in Eq. (6). I suggest moving the usage of the clustering assignment (i.e., Eq. (3)) to after Eq. (6).
      v) What does $[\cdot || \cdot]$ mean (cf. Eq. (4))? (The code seems to suggest it's concatenation?)
      vi) The authors first used $h_{x_i}$ in Eq. (3) to denote the output of the CAB module. Then letter $h$ is then re-used in Eq. (4) and (5) with completely different meanings. For instance, the $W_kh_i$ in Eq. (4) correspond to line 48 of the code "model.py". (By the way, nowhere around Eq. (4) did the authors explain how $h_i$ is produced, such as taking the mean over the batch dimension, etc.).
      vii) In Section 2.6, you denote the "optimal vertex cluster scheme" with letter $C$, which is used in Eq. (2). Similar for parameter $a_k$ and atrous offset $a$.

  - This not a very big problem (as it seems somewhat inevitable), but I think there are too many acronyms in the paper.

  I think it'd be great if the authors can take care of these issues, as clarity in math and descriptions are critical to the presentation of such an involuted method. It would also be useful to clearly define the dimensionality of all the variables (e.g., you defined $V$ in Section 2.1, but never used it again in later subsections).
2) Regarding the usage of the multi-view position encoding, the authors claimed that it "provides unique identifiers for all time-steps in temporal sequences". However, if you consider $x=7$ and $x=14$, then $PE_i(7)=PE_i(14)$ for all $i=1,2,3,4$ with $PE_5(7) \approx PE_5(14)$. Doesn't this invalidate the authors' claim? Also, doesn't this mean that the proposed MVPE only works on sequences with length <= 7? (In comparison, the design of positional encoding in the original Transformer doesn't have this problem.)
(You didn't show how you implemented and initialized the position encoding in the uploaded code, so I may be missing some assumptions here.)
3) In line 48 of the code (https://github.com/CGT-ICLR2020/CGT-ICLR2020/blob/master/model.py#L48), why did you take the mean over the batch dimension? Shouldn't different samples in a minibatch be very different? Does a (potentially completely independent) sample in a batch affect another sample? A similar problem occurs for Eq. (9): Why do you require clusterings of two different samples $b_1, b_2$ to be similar? (Where these samples can come from quite different times and years of the data?)
4) In the experiments, you "sampled 10 input time-steps" due to computational resources. Typically, in Transformer-based NLP tasks the sequence lengths can be over 500, with much higher dimensionality (e.g., 512); but you are only using sequence length 10 and dimensions <= 16 (in your code, you used "self.dec_io_list = [[5,8,16,16],[16,16,16,16],[16,8,8,8]]"). What is the bottleneck for the computation of your approach? (I noticed there are more than 1K vertices in city A, which may be a costly factor indeed.) How much memory/compute does the CGT method consume? How does using a longer sequence affect the performance of CGT?
5) You performed an ablation study on MVPE. Did you simply remove MVPE, or did you use the conventional PE from the original Transformer paper (Vaswani et al. 2017)? (If the latter, I'm very surprised that MVPE is so much better than PE. In that case, you may want to try MVPE on NLP tasks to see if it also improves SOTA.)
6) How did you measure unsmoothness in Figure 6? It doesn't seem like a quantifiable property to me. You should discuss this in the experiment section.
-----------------------------------
Minor questions/issues that did not affect the score:
7) There are some strange phrases/sentences in the paper. For example, the first sentence of the 2nd paragraph of Section 1: "we will show **throughout the paper** that urban spatiotemporal prediction task suffers from..."
8) Why use an encoder-decoder architecture at all? Why can't we train the model like in language modeling tasks, where we want to predict the next token? In other words, you can simply use a decoder-side CGT, and mask the temporal self-attention as in the Transformers.
-----------------------------------
In general, I think this paper proposed a valuable approach that seems to work very well on the spatio-temporal dataset they used (which unfortunately is private). However, as I pointed out above, I still have numerous issues with the paper's organization and clarity, as well as some doubts over the methodology and the experiment. I'm happy to consider adjusting my score if the authors can resolve my concerns satisfactorily.
[1] http://www-scf.usc.edu/~yaguang/papers/aaai19_multi_graph_convolution.pdf


Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate systems, and hence may lose the invariance properties. We analyze the invariance properties of the Kronecker-Factored Approximate Curvature (K-FAC) algorithm by constructing the algorithm in a coordinate-free way. We explicitly construct a Riemannian metric under which the natural gradient matches the K-FAC update; invariance to affine transformations of the activations follows immediately. We extend our framework to analyze the invariance properties of K-FAC appied to convolutional networks and recurrent neural networks, as well as metrics other than the usual Fisher metric.

This paper is concerned with tractable (approximate) forms of natural gradient updates for neural networks, in particular with the recent K-FAC approximation, which applies a set of approximation (layer-wise independence, Kronecker structure for affine maps) in order to obtain a Hessian that can be computed and inverted efficiently. K-FAC has been introduced for MLPs, and has previously been generalized to convolutional and certain recurrent NNs.
The stated goal of this paper is to provide a mathematical re-formulation of K-FAC in terms of Riemannian metrics. While K-FAC has been developed as approximation to the exact natural gradient update, they come up with a different Riemannian metric, definition of space, etc., such that in the end, K-FAC is the exact natural gradient for that. The authors here also obtain a more precise answer to invariance properties and, given some heavy maths, what they claim to be more elegant proofs of previously known properties of K-FAC.
The paper uses very heavy math, well "over my head" and likely most ICLR attendees. Along with me, they'll ask the obvious question of what this is good for. As far as I can see, there is nothing really new being proposed here in terms of practical consequences. The authors also do not make much effort to explain why their viewpoint is useful, say to obtain practically relevant insights in future work. So, as far as I am concerned, I do not see why this work should be of much relevance to ICLR, which is not an abstract maths conference.
A final comment is that people have for a very long time tried to use second-order optimization for MLPs. The aspect that always was tricky there, is that SGD is *stochastic*, and the second-order info is hard to estimate from a mini-batch. The sets of approximations of K-FAC are pretty extreme, but they may just be needed to make things work in the end, because they may stabilize that "stochastic inverse Fisher info matrix" enough to not make the optimization process fail altogether. Now, all theoretical arguments, like "invariance to this and that", always ignore the crucial fact that you are talking about a stochastic estimate over a mini-batch, and your theory is always for E_x[...] "being the truth". It is not, it is just over a small mini-batch. I am not saying the additional theoretical insight from this work here (over previous K-FAC work), whatever it may be in the end, is not useful. I am just saying I'd be a lot more confident if the authors would specifically address the stochastic property.


Few-shot classification is a challenging task due to the scarcity of training examples for each class. The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classifier to novel classes. In this paper, we introduce a two-stage framework. In the first stage, we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss. The Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training. Besides, the Metric-Softmax classifier can be applied to base and novel classes in a consistent manner, which is critical for the generalizability of the learned feature. In the second stage, we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within a few tuning epochs. Compared with existing fine-tuning scheme, the scarce examples of novel classes are exploited more effectively. Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks.

# Summary
This paper deals with few-shot learning from a metric-learning perspective. The authors propose replacing the softmax loss, i.e. softmax + cross-entropy loss, with a so-called "metric-softmax" loss which imitates a Guassian kernel RBF over class templates/weights. This loss is used in both stages of training on base and on novel classes and the authors argue that it helps learning more discriminative feature while preserving consistency between train and test time.
Secondly the authors advance a task-adaptive transformation for stage 2 that maps the features from the previously learned feature extractor to a space which is easier to learn. The contributions are evaluated on the standard mini-ImageNet benchmark and on CUB-200-2011 individually and in domain shift mode.
# Rating
Although some of the results in the paper might look impressive, my rating for this work is reject for the following reasons (which will be detailed below):
1) the main contribution, metric-softmax loss, is not novel. It has been used and described in multiple works in the past 1-2 years.
2) a part of the evaluations and comparisons do not follow the usual protocol and are not fair
3) the second contribution, Fast Task Adaptation (FTA), is not well described and it's unclear in what it actually consists, how does it work and how was it trained exactly.
# Strong points
- This paper deals with a highly interesting and relevant topic for ICLR.
# Weak points
## Contributions
- This work ignores a large body of research in few-shot learning and metric learning aiming to improve the efficiency per training sample and feature discrimination.
The proposed loss can be traced back to Goldberger et al. [i] in  NCA (Neighborhood Component Analysis). Prototypical Networks are derived from this work and hence similar with the metric-softmax loss.
Qi et al. [ii] point out that when h and W are l2-normalized (using the notations from this submission, eq. 7) maximizing their inner-product or cosine-similarity is equivalent to the minimization of the squared Euclidean distance between them. This leads to the loss from [ii], [iii], also known as Cosine Classifier and which also is accompanied by a scaling factor or temperature as here.
Other related works on improving softmax and selecting the representative weights for a class include: center loss [iv], ring loss[v], L-GM loss[vi].
In this light, the metric-softmax is actually not novel and can be found in several other contributions from last year.
- In my opinion, it is difficult from the paper to understand what is actually the fast adatpation module. The authors describe $g$ as "simply a zero-offset affine transformation" $g(h)= M^T h$. In the implementation details we do not find out more about this module and we don't have more insights on what is it doing inside, other than a toy hand-drawn example in Figure 3. I find it difficult to assess.
## Experiments
- The authors evaluate 3 backbone architectures, Conv-4, ResNet-10 and ResNet-12. For the former they use 84 x 84 images, while for the latter they use 224 x 224 images.  The larger images are not standard in the few-shot ImageNet evaluation protocol. Data augmentation (jittering, flipping, etc.) is used here, while in most works it is not. Chen et al. are the first ones to introduce larger images and data augmentation and acknowledge that the large scores are due to this.
Testing out new configuration is not a problem as long as the baselines are evaluated in the same conditions. However, in this case they are not and this is not visible in the captions of the tables and descriptions in the paper. Training a network with data augmented images and/or higher resolution images and comparing to baselines without data augmentation and images with 7 times less pixels, for sure does not allow seeing the true impact of the proposed method. I would advise to either evaluate in the usual mini-ImageNet settings, either implement a few representative and easy to train baselines, e.g. ProtoNets, Cosine Classifier[iii] in the same conditions as here and compare against. This should provide a better idea on the effectiveness of the proposed methods.
## Other comments
- the scores for baseline methods are seemingly taken from the paper of Chen et al. who trained them themselves. This should be mentioned in the paper and in the caption
# Suggestions for improving the paper:
1) Review the experimental section and make sure at least some of the baselines are trained in similar conditions as the proposed method or alternatively evaluate the proposed methods in standard mini-ImageNet settings
2) Provide additional insights, experiments and implementation details for FTA to make it easier to understand, there are some examples in the references below.
# References
[i] J. Goldberger et al., Neighbourhood components analysis, NIPS 2005
[ii] H. Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018
[iii] S. Gidaris and N. Komodakis, Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018
[iv] W. Wen et al., A Discriminative Feature Learning Approach
for Deep Face Recognition, ECCV 2016
[v] Y. Zeng et al., Ring loss: Convex Feature Normalization for Face Recognition, CVPR 2018
[wi] W. Wan et al., Rethinking Feature Distribution for Loss Functions in Image Classification, CVPR 2018


Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

This paper presents a replication study of BERT pretraining and carefully measures the impact of many key hyperparameters and training data size. It shows that BERT was significantly undertrained and propose an improved training recipe called RoBERTa. The key ideas are: (i) training longer with bigger batches over more data, (ii) removing NSP, (iii) training over long sequences, and (iv) dynamically changing the masking pattern. The proposed RoBERTa achieves/matches state-of-the-art performance on many standard NLU downstream tasks.
The in-depth experimental analysis of the BERT pretraining process in this paper answers many open questions (e.g., the usefulness of NSP objective) and also provide some guidance in how to effectively tweak the performance of pretrained model (e.g., large batch size). It also further demonstrates that the BERT model, once fully tuned, could achieve SOTA/competitive performance compared to the recent new models (e.g., XLNet). The main weakness of the paper is that it is mainly based on further tuning the existing BERT model and lacks novel contribution in model architecture. However, the BERT analysis results provided in this paper should also be valuable to the community.
Questions & Comments:
•	It is stated that the performance is sensitive to epsilon in AdamW. This reminds us of the sensitivity of BERT pretraining to the optimizers. Since one of the main contributions of this paper is the analysis of the BERT pretraining process, more experimental analysis on the optimizer should also be included.
•	It is stated that (page 7) the submission to GLUE leaderboard uses only single-task finetuning. Is there any special reason for restraining it to single-task finetuning if earlier results demonstrates multi-task finetuning is better? Of course, it is valuable to see the great performance achieved by single-task finetuning for RoBERTa. But there should be no reason that it is restricted to be so. An additional experimental results with multi-task finetuning should also be added.


This work examines the convergence of stochastic gradient algorithms that use early stopping based on a validation function, wherein optimization ends when the magnitude of a validation function gradient drops below a threshold. We derive conditions that guarantee this stopping rule is well-defined and analyze the expected number of iterations and gradient evaluations needed to meet this criteria. The guarantee accounts for the distance between the training and validation sets, measured with the Wasserstein distance. We develop the approach for stochastic gradient descent (SGD), allowing for biased update directions subject to a Lyapunov condition. We apply the approach to obtain new bounds on the expected running time of several algorithms, including Decentralized SGD (DSGD), a variant of decentralized SGD, known as Stacked SGD, and the stochastic variance reduced gradient (SVRG) algorithm. Finally, we consider the generalization properties of the iterate returned by early stopping.

The paper studies the problem of the number of first-order-oracle calls for the SGD type of algorithms to find a stationary point of the objective function. The main results in the paper are built upon a new, general framework to analyze the SGD type of algorithms.
The main framework can be summarized as follows: At each iteration, the algorithm receives h_t, a (potentially biased) estimator of the gradient at a given point x_t, and performs a simple update x_{t + 1} = x_t - \eta * h_t. The framework says that as long as the norm (V_t) of \Delta_t = h_t - v_t (where v_t is an unbiased estimator of the true gradient with bounded variance) satisfies a particular Lyapunov-type inequality, then the algorithm can find an epsilon-stationary point as long as epsilon is not too small.
The analysis of the framework is quite standard, one only needs to write the decrement in function value at each iteration into the following three terms: the norm of the true gradient of the function, \delta_t: the difference between v_t and the true gradient (so E[\delta_t] = 0) and \Delta_t: the difference between the received gradient h_t and v_t.
The authors showed some application of this framework in Stacked SGD and decentralized SGD. The main intuitions of these applications are (1). \Delta_t comes from the synchronization difference of the nodes when computing the gradient. (2). The shrinking of V_t is due to the  (better) synchronization at each iteration. (3). The increment of V_t is due to the gradient update.
Overall, I find the general framework quite interesting and potentially useful for future research and could be used as a guide for choosing the proper algorithm in distributed computation.  The bounds in this paper are also in principle tight. The only question I have about this result is the dependency of m (the number of iterations between each evaluation of the gradient norm of the underlying function). (1). How can this (the evaluation of the gradient norm of the underlying function)) be done in a decentralized environment? What is the computation overhead?  (For example in DSGD, how can we compute \bar{x}_t?) (2). It seems that the computation cost (number of IFO) scales quadratically with respect to m. What is the intuition for this scaling? It appears to me that the scaling should be linear or better (the worst case is that within the "m" iterations, only one iteration has gradient >= epsilon). The authors should elaborate more on this point.


Deep neural networks represent data as projections on trained weights in a high dimensional manifold. This is a first-order based absolute representation that is widely used due to its interpretable nature and simple mathematical functionality. However, in the application of visual recognition, first-order representations trained on pristine images have shown a vulnerability to distortions. Visual distortions including imaging acquisition errors and challenging environmental conditions like blur, exposure, snow and frost cause incorrect classification in first-order neural nets. To eliminate vulnerabilities under such distortions, we propose representing data points by their relative positioning in a high dimensional manifold instead of their absolute positions. Such a positioning scheme is based on a data point's second-order property. We obtain a data point's second-order representation by creating adversarial examples to all possible decision boundaries and tracking the movement of corresponding boundaries. We compare our representation against first-order methods and show that there is an increase of more than 14% under severe distortions for . We test the generalizability of the proposed representation on larger networks and on 19 complex and real-world distortions from CIFAR-10-C. Furthermore, we show how our proposed representation can be used as a plug-in approach on top of any network. We also provide methodologies to scale our proposed representation to larger datasets.

The authors propose an approach to reduce the sensitivity of neural networks to visual distortions. To do so, they modify the representation of a data point within a network, using its relative position (to other points) in representation space rather than its absolute position  (which is measured as distance of a point to the decision boundary of each class). They then evaluate their approach on common corruptions from the CIFAR-C dataset.
The paper addresses an important problem, however I do not find the high-level motivation behind the proposed approach or the experimental results sufficiently convincing.
In particular, conceptually, it is completely unclear why the second order representations should be more resilient to visual distortions. Even in the sample illustration provided in Figure 2, it is evident that the clusters in this new representation space are not invariant to visual distortion, or even are significantly more invariant than the first-order representation. At a more fundamental level, given that the accuracy of the original network does drop, it is tautological that the distance of a point to the decision boundaries is decreasing under distortion (and thus is sensitive to it).
Experimentally, my chief concerns are:
1. When the authors evaluate the proposed second-order representations, they use networks with additional layers which do not seem to be present in the original baseline. Prior work [Hendrycks and Dietterich, 2019] has shown that model capacity has a marked influence on the corruption robustness of a network. Thus, it is unclear where the improvement here is coming solely from the additional capacity compared to the original model. The baseline network that the authors compare to should also include the additional layers.
2. The results reported for some of the baselines seem inconsistent with prior work.
a) For example, the authors state that the results for the Hossain et al. [2018] baseline are similar to NR1, which is worse than the original network. However, Hossain et al. report an improvement for the same corruptions and the same dataset in their paper. Where is this inconsistency coming from? Do the authors train with the DCT filtering or is it only applied at test time? If it is the latter, it could explain why the authors fail to reproduce the baseline correctly.
b) For the adversarially robust network baseline, why did the authors choose an FGSM adversary? FGSM is not typically used to train state-of-the-art robust models because of the existence of much stronger attacks (such as PGD). How was the eps used to train the network chosen? In prior work, Kang et al. [2019; arxiv:1908.0801] evaluate both L2 and Linf robust models and show improvements over the baseline for several common corruptions. This seems to suggest that the robust model baseline reported in this paper is not accurate/representative.
3. Moreover, given that the representation size scales with number of classes, the proposed method should be evaluated on datasets with more classes such as CIFAR-100 or ImageNet. Improvements demonstrated in these settings with dimensionality reduction would be more convincing.
Overall, my main reservations are: a) the lack of a conceptual justification for the proposed approach, and b) issues with the experimental evaluation, particularly in the reported baselines and how they seem to contradict prior work. Thus, I recommend rejection.
Other comments:
- The right-side of Figure 1 is essentially the same map flipped.
- Why was the MSE loss chosen for J? Given that the network was presumably trained with cross-entropy, this choice seems somewhat arbitrary. Are the results consistent for cross entropy loss as well? The authors should include these results in the appendix.
- Figure 4 is very hard to read---the authors should change the plotting style to make the results more legible.
- For the results in Figure 5, the authors should once again compare to adding extra layers to the baseline networks as well.
- In Table 1, why does the performance of NR improve when the features are based only on gradients for the last layer?


Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that selftraining is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a "noisy" version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

This paper presents a self-training approach for improving sequence-to-sequence tasks. As a preliminary experiment, this study randomly sampled 100k sentences from WMT 2014 English-German dataset (WMT100K, hereafter), trained a baseline (Transformer) model on WMT100K, and applied self-training methods on the remaining English sentences as the unlabeled monolingual data. After exploring different procedures for self-training, this study uses the fine-tuning strategy: train a model on the supervision data; build pseudo parallel data by predicting translations for all unlabeled data using the trained model; train a new model on the pseudo parallel data; and fine-tune the new model on the supervision data. This strategy alone gave a 3 points improvement of BLEU.
This paper hypothesized the reasons behind the performance improvements: beam-search decoding (+0.5 BLEU) and dropout (+1.2 BLEU). The authors argue that the beam-search decoding contributes partially to the performance gains, while the dropout accounts for the most. The authors infer that the dropout causes an implicit perturbation. Exploring different perturbation strategies, synthetic noise (e.g., input tokens are randomly dropped, masked, and shuffled) and paraphrase (round-trip translation, e.g., English-German-English), the authors reported no significant difference between these two strategies. Finally, this paper reports empirical results (on WMT 2014 English-German, FloRes English-Nepali, and English Gigaword (summarization task)) of self-training strategies presented in this paper. This paper concluded that self-training can be an effective method to improve generalization and that the noise injected during self-training plays a critical role for its success.
This paper is well structured and well written. It is interesting to see the improvements of the sequence-to-sequence tasks by using the self-training approach. This paper is interesting because it has a close connection with the back-translation approach that has been popular in recent years.
Although the hypotheses presented in this paper are interesting, they were not fully validated after all. The analyses on the loss functions, ablation tests, and experiments on the toy task can only bring indirect explanations about why we could observe the performance improvements. This impression is also demonstrated by the fact that this paper uses 'might' five times when explaining interpretations of the experimental results. Having said, I tend to agree that identifying the exact reason is difficult.
However, I have two other questions before recommending this paper: (1) whether the baseline model was trained sufficiently, and (2) whether this paper is about self-training strategies or regularization methods.
(1) In order to accept the experimental results that the self-training approach can improve the performance, we need to make sure that the baseline model was trained sufficiently. However, the appendix explains, "we basically use the same learning rate schedule and label smoothing as in fairseq examples." I'm not sure whether this training procedure was fair among different models because the baseline model and self-supervised model received totally different number of training instances (100k vs 3.9M). This claim would be stronger if this paper could show an evidence that the baseline model was trained properly by, for example, explaining the stopping criteria for iterations, tuning hyper-parameters (e.g., learning rate) individually for the baseline and self-trained models, showing the mean and variance of BLEU scores with different initializations, and showing the training curve of the baseline model.
(2) We can view the self-training strategies presented in this paper as regularization methods. For this reason, I am wondering whether the self-training strategies presented in this paper are only for self-training or general to pure supervised setting. We can easily guess that the performance would drop if we removed the dropout from the baseline method. In contrast, I would like to see whether the synthetic noise could improve the performance of the baseline method alone, behaving as a regularization method. It would be useful to see the performance of the baseline method without the dropout and with the synthetic noise to highlight the effect of the presented strategies under the self-training scenario.
Minor comments:
It would be useful to see the number of unlabeled instances in Section 3.1.
Sectoin 3.2: "This is different from back-translation where new knowledge may originate from an additional backward translation model."
I'm not sure whether a backward translation model can introduce new knowledge because the supervision data are usually the same between forward and backward directions.
Section 3.3: "at test/decoding time the model does not use dropout"
To be precise, the weights are scaled by the dropout rate (p) in the decoding time.
Reference: Lample et al. 2018 should be replaced with another paper:
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato. Unsupervised Machine Translation Using Monolingual Corpora Only. ICLR 2018.


We propose Adversarial Inductive Transfer Learning (AITL), a method for addressing discrepancies in input and output spaces between source and target domains. AITL utilizes adversarial domain adaptation and multi-task learning to address these discrepancies. Our motivating application is pharmacogenomics where the goal is to predict drug response in patients using their genomic information. The challenge is that clinical data (i.e. patients) with drug response outcome is very limited, creating a need for transfer learning to bridge the gap between large pre-clinical pharmacogenomics datasets (e.g. cancer cell lines) and clinical datasets. Discrepancies exist between 1) the genomic data of pre-clinical and clinical datasets (the input space), and 2) the different measures of the drug response (the output space). To the best of our knowledge, AITL is the first adversarial inductive transfer learning method to address both input and output discrepancies. Experimental results indicate that AITL outperforms state-of-the-art pharmacogenomics and transfer learning baselines and may guide precision oncology more accurately.

The paper proposes an adversarial transfer learning network that can handle the adaptation of both the input space and the output space. The paper is motivated by the application of drug response prediction where the source domain is cell line data and the target domain is patient data. Patient data are usually scarce, hence motivating transferring the knowledge learned from the more widely available cell line data to improve the predictive performance based on the patient data.  The idea of making use of adversarial networks is to learn a representation of the data points that is invariant to whether the data points come from the source domain or the target domain. Experiments on real-world data over four drugs demonstrate the effectiveness of the proposed methods compared to other methods that are not specifically designed for this scenario.
The major utility of the proposed method demonstrated by the paper is its empirical effectiveness on real-world data with four drugs. The reviewer has the following concerns:
1. While the proposed method is capable of handling adaption of the output space between the source domain and the target domain, it makes use of a multi-task subnetwork component, which is a sensible modeling choice but it seems that the idea of leveraging a global discriminator and a class-wise discriminator has been exploited in previous works, as pointed out by the authors in Section 4.2. Therefore, the reviewer is concerned about the modeling contribution made in this paper is somewhat incremental given existing literature.
2. The multi-task subnetwork component itself also seems to be a straightforward and naive way to deal with the different tasks addressed in the source domain and the target domain. For one thing, only the data of the source domain are used to optimize the loss in the source domain and only the data of the target domain are used to optimize the loss in the target domain. While it is understandable that such a decision is due to the lack of ground truth of binary label in the source domain and numeric response in the target domain, ideally, the discriminative process might gain further benefits from borrowing information across the two domains.
3. It is also unclear from the paper how each component of the architecture contributes to the final performance. An ablation study that gets rid of the multi-task subnetwork, global discriminator, and class-wise discriminators (and potential combinations of these components) could help to provide better insights in the importance of each component and determine whether empirically these components do play a role coinciding with the description of the paper.
Other issues/questions:
1. Intuitively, what is the purpose of the shared layer g() in section 3.2.2?
2. In the spirit of the class-wise discriminators, will it be helpful to also add a discriminator based on the value of IC50?
3. Concepts and terminology are not well explained. e.g. what is a cell line? The authors should also provide further descriptions of what do the cell line and patient datasets look like early on in the paper. It was not until section 3 it is clear to the readers that in the dataseets consider in this paper, the source domain and the target domain have the same raw feature representation. Furthermore, in section 2, the authors could consider providing a taxonomy in table format to better explain different types of transfer learning and the three approaches to inductive transfer learning.


Microsoft Word - Effective Mechanism to Mitigate Injuries During NFL Plays
Abstract-NFL(American football),which is regarded as the premier sports icon of America, has been severely accused in the recent years of being exposed to dangerous injuries that prove to be a bigger crisis as the players' lives have been increasingly at risk. Concussions, which refer to the serious brain traumas experienced during the passage of NFL play, have displayed a dramatic rise in the recent seasons concluding in an alarming rate in 2017/18. Acknowledging the potential risk, the NFL has been trying to fight via NeuroIntel AI mechanism as well as modifying existing game rules and risky play practices to reduce the rate of concussions. As a remedy, we are suggesting an effective mechanism to extensively analyse the potential concussion risks by adopting predictive analysis to project injury risk percentage per each play and positional impact analysis to suggest safer team formation pairs to lessen injuries to offer a comprehensive study on NFL injury analysis. The proposed data analytical approach differentiates itself from the other similar approaches that were focused only on the descriptive analysis rather than going for a bigger context with predictive modelling and formation pairs mining that would assist in modifying existing rules to tackle injury concerns. The predictive model that works with Kafka-stream processor real-time inputs and risky formation pairs identification by designing FP-Matrix, makes this far-reaching solution to analyse injury data on various grounds wherever applicable.

This paper introduces a machine learning pipeline for injury prediction in NFL events. The paper discusses several system settings on data streaming and processing, along with model selection and other hyper-parameter tuning details. The problem itself is very important. However, there are several disadvantages of the current status of the paper.
The writing of this paper needs to be largely improved. The description is very redundant and the texts are very hard to read. Therefore, it makes the paper much less clear to the reader.
Second, the proposed system is not related to the focus of ICLR and it lacks the novelty. Both the data processing and model selection methods are already well-known practicies. Experiments are also not established well enough to demonstrate the advantage of the proposed solution to this problem.
Due to the above issues, I think the paper is not ready for publication.


Reinforcement learning (RL) combines a control problem with statistical estimation: the system dynamics are not known to the agent, but can be learned through experience. A recent line of research casts 'RL as inference' and suggests a particular framework to generalize the RL problem as probabilistic inference. Our paper surfaces a key shortcoming in that approach, and clarifies the sense in which RL can be coherently cast as an inference problem. In particular, an RL agent must consider the effects of its actions upon future rewards and observations: the exploration-exploitation tradeoff. In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation. We demonstrate that the popular 'RL as inference' approximation can perform poorly in even very basic problems. However, we show that with a small modification the framework does yield algorithms that can provably perform well, and we show that the resulting algorithm is equivalent to the recently proposed K-learning, which we further connect with Thompson sampling.

The paper at hand presents an alternative view on reinforcement learning as probabilistic inference (or equivalently maximum entropy reinforcement learning). With respect to other formulations of this view (e.g. Levine, 2018; I am referring to the references of the paper here), the paper identifies a shortcoming in the disregard of the agent’s epistemic uncertainty (which seems to refer to the uncertainty with respect to the underlying MDP). It is argued, that algorithms based on the prevailing probabilistic formulation (e.g. soft Q-learning) suffer from suboptimal exploration.
The paper thus compares maximum entropy RL to K-learning (O’Donoghue, 2018), which is taken to address the issue of suboptimal exploration due to its temperature scheduling and its inclusion of state-action pair counts in the reward signal.
As its technical contribution, the paper re-interprets K-learning via the latent variable denoting optimality employed in Levine (2018) and introduces a theorem bounding the distance between the policies of Thompson sampling and K-learning. Empirical validation of the claims is provided via experiments on an engineered bandit problem and the tabular MDP (i.e. DeepSea from Osband et al., 2017), as well as via soft Q-learning results on the recently suggested bsuite (Osband et al., 2019).
I consider this paper a weak reject. This is in light of me finding it very hard to follow the papers main claims and arguments, even though it positions itself as communicating connections (“making sense”) in prior work, rather than presenting a novel algorithm. While this is in part due to the complicated issue and math being discussed (and the paper probably catering to a very narrow audience), the paper in its current state does seem to hinder understanding as well.
On the positive side, I do appreciate the intention of the paper, namely to connect RL as probabilistic inference, Thompson sampling and K-learning. In my opinion, this can be taken as a valuable addition to the current understanding of these approaches. Also, I like the experiments as they are specifically constructed to support the claims of the paper.
On the negative side, vague language, missing assumptions and lax notation seem to hinder the understanding of the paper to a considerable extend: e.g. it is stated, that “we connect the resulting algorithm with […] K-learning”. However, I do not recognize a new algorithm being provided. Instead the paper argues in favor of K-learning. The assumptions that come with K-learning are not mentioned. The restriction of K-learning to tabular RL is taken to be understood implicitly (whereas RL as probabilistic inference seems applicable with function approximation also, which is not mentioned in the comparison). The paper always talks of shortcomings (plural) of RL as probabilistic inference, but only provides one argument (suboptimal exploration) with respect to this. RL as probabilistic inference is introduced in a different form as in prior literature (i.e. Equation 6), while the derivation in the Appendix spanning the differences in notation being hard to follow due to (maybe minor?) notational issues (e.g. x and y seem to have replaced s’ and a; further down there is a reference to Equation 7, however probably it is meant to be 8 and even that with some leap in notation).
The paper would benefit from better proof-reading, where mistakes in a very dense argumentation make it hard to follow (e.g. I do not understand the sentence “The K-learning expectation (7) is with respect to the posterior over Q[…] to give a parametric approximation to the probability of optimality.”)
Literature wise, the paper draws heavily from two unpublished papers (Levine, 2018; O’Donoghue, 2018). While this makes it harder to arrive at a high confidence level with respect to the paper’s claims, I would not argue this to be critical.
I would consider raising my score, if the authors would improve the accessibility of the paper by polishing the argumentation and notation.
Confidence: low. It is very likely, that I have misunderstood key arguments and derivations. Also, I did not attempt to follow all of the technical derivations.
======
post rebuttal comment:
I changed the score of my review in light of the rebuttal.
The changes made to the paper overall address my concerns.
I do consider the additional explanations and re-phrasings as well as the improved notation a nice improvement of the paper.
While I did not read all of the appendix, Section 5.1 is much more readable and understandable in the new version.
In light of this paper probably being published, I share some typos/inconsistencies I still noticed:
p. 4: the solving for -> solving for the
p. 7: s_{h+1} -> s' (in Table 3) ?
p. 7: table -> Table; tables -> Tables
p. 7: (Fix position of K:) \pi_h(s)^K -> \pi_h^K(s) ((also in Appendix))
p. 9: (2x) soft-Q learning -> soft Q-learning; Q Networks -> Q-Networks; Soft Q -> soft Q-learning


Program comprehension is a fundamental task in software development and maintenance processes. Software developers often need to understand a large amount of existing code before they can develop new features or fix bugs in existing programs. Being able to process programming language code automatically and provide summaries of code functionality accurately can significantly help developers to reduce time spent in code navigation and understanding, and thus increase productivity. Different from natural language articles, source code in programming languages often follows rigid syntactical structures and there can exist dependencies among code elements that are located far away from each other through complex control flows and data flows. Existing studies on tree-based convolutional neural networks (TBCNN) and gated graph neural networks (GGNN) are not able to capture essential semantic dependencies among code elements accurately. In this paper, we propose novel tree-based capsule networks (TreeCaps) and relevant techniques for processing program code in an automated way that encodes code syntactical structures and captures code dependencies more accurately. Based on evaluation on programs written in different programming languages, we show that our TreeCaps-based approach can outperform other approaches in classifying the functionalities of many programs.

The paper proposes a neural architecture for summarizing trees inspired by capsule networks from computer vision. The authors re-use a tree convolution from previous work for the bottommost layer, and then propose adaptations to the dynamic routing from capsule networks so that it can be applied to variable-sized trees. The paper applies the proposed architecture to three different program classification datasets, which are in three different languages. The paper reports empirical gains compared to two architectures proposed by previous work.
I think that it's interesting to apply the capsule network architecture to tree classification, but unfortunately it doesn't appear that some of the motivation for capsule networks on images didn't seem to transfer neatly to this setting; for example, there is no equivalent of inverse graphics as there is no reconstruction loss (as pointed out by the authors in Section 6.4).
Also, the variable-to-static capsule routing indeed appears novel, but I was a bit confused by its internal details. It appears that the outputs of the previous layer which occur most often will get routed (considering lines 6-8 of Algorithm 1 which up-weights each of the $\hat{u}_i$ based on its similarity to $v_j$; the $v_j$ are initially a re-numbered subset of $\hat{u}_i$), without any prior transformation of the previous layer first. It seems to me that this doesn't allow for the prior layer to predict more complex features about the input that the subsequent layer is expected to capture. In fact, for certain code classification tasks, it may be that rare capsule outputs from the initial layer are the most important to preserve.
My biggest concern has to do with the empirical results. The source of Dataset C (Mou et al 2016, https://arxiv.org/pdf/1409.5718.pdf) reports 94.0% accuracy in Table 3 on their TBCNN method on the same dataset, whereas this paper reports 79.40% accuracy for TBCNN. I understand that the later result comes from a reimplementation, but it seems fairer to compare against (or additionally report) the results from the original authors of the method.
Also, the paper cites ASTNN (Zhang et al 2019, https://dl.acm.org/citation.cfm?id=3339604) in the introduction, and even though that paper reports (in table 2) 98.2% accuracy on Dataset C, the results table of the paper under review does not mention this in the evaluation section. I don't think that a paper necessarily has to achieve empirical results beating all previous ones in order to merit acceptance, but the way that the comparison is currently set up doesn't seem to facilitate a clear comparison of the pros and cons of this method versus other ones in the literature.
For the above reasons, I vote to reject the paper. For future submissions, it would be good to see a more comprehensive empirical comparison of the proposed method compared to others, and also to have more explanations about the design of the network.


In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems [10]. In this work, we suggest a simple approach for improving stability in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance across most domains.

[Summary]
This paper proposes an approach called conservative policy gradients to stabilize the training of deep policy gradient methods. At fixed intervals, the current policy and a separate target policy are evaluated with a number of rollouts. The target policy is then updated to match the current policy only if the current policy is better than the target policy. Experiments show that the proposed method, when applied to TD3, reduces the variance in performance through the training.
[Decision]
I am not convinced that the proposed method is sound and indeed useful and I vote for rejecting this paper. Experiments show stable performance. However, this stability comes at the cost of extra computation and interaction with the environment (to evaluate the policies). Claims about the method's stability guarantees and overall performance are not supported by theory and experiments. The submitted paper also needs major improvements in presentation.
[Explanation]
While Proposition 1 provides insights on how the policy evolves, it is too limited to serve as a guarantee. First, performance does not improve or degrade by a constant number. Second, the time it takes for the policy to improve is not captured by the theory. In reality, this time can depend on the hyperparameters or the policy's current performance and might even be unbounded.
I do not understand why a characterization of the performance in the limit of time in Proposition 1 is called a stability guarantee while in the rest of the paper stability refers to consistent improvements in the interim performance. Does stability in this proposition mean that the performance will reach a stationary distribution with bounded support? This property is merely a result of the assumptions that the performance evolves by a constant number at bounded times and that it does not exceed [v_min,v_max].
The theory studies the stationary distribution of the target policy's performance but the algorithm uses the online policy to interact with the environment. In Algorithm 5, line 8 (Section D in the Appendix) the target policy is only used for bootstrapping. How can a stable target policy result in more stable performance if it is not used to take actions?
The paper claims that the proposed method results in improvements in stability and overall performance. In Figure 3, the proposed method is more stable than the baseline but the overall performance is not better.
The proposed method requires more computation and interaction with the environment than the baseline. The experiments do not seem to compare these two methods with the same number of samples or with the same amount of computation. Perhaps the extra computation and samples are better spent on training TD3 for a longer time.
I find Section 2 hard to follow. This section describes Value Iteration, Policy Iteration, DQN, and DDPG in detail (with pseudocode) along with their convergence rates. The message that deep RL algorithms generally lack theoretical guarantees can be conveyed by just describing the linear and deep variants of one method. In fact, the algorithm whose stability is analyzed in the next sections, TD3, is not described in Section 2 or anywhere else in the paper.
Later in Section 3, DDPG and DQN are described as off-policy Deep RL variants of Value Iteration and Policy Iteration. DQN and DDPG are actually built on Q-learning and Deterministic Policy Gradient (DPG).
[Minor comments]
In the learning curves in Figure 1, what is the measure of performance and how is it estimated? A description of the plotted measure is necessary to show that the drops in the estimated performance are indeed due to policy degradation rather than poor estimation.
--------------------
After rebuttal: I have read the authors' response, the other reviews, and the revision. The revised version has improved presentation, but the proposed method is still introduced as a method with stability guarantees while the proposition in the paper cannot serve as a stability guarantee, and can only provide intuition on the asymptotic performance.


Deep graph generation models have achieved great successes recently, among which however, are typically unconditioned generative models that have no control over the target graphs given an input graph. In this paper, we propose a novel GraphTranslation-Generative-Adversarial-Networks (GT-GAN) that transforms the input graphs into their target output graphs. GT-GAN consists of a graph translator equipped with innovative graph convolution and deconvolution layers to learn the translation mapping considering both global and local features. A new conditional graph discriminator is proposed to classify the target graphs by conditioning on input graphs while training. Extensive experiments on multiple synthetic and realworld datasets demonstrate that our proposed GT-GAN significantly outperforms other baseline methods in terms of both effectiveness and scalability. For instance, GT-GAN performs at least 10 and 15 time faster than GraphRNN and RandomVAE, respectively, when the size of the graph is around 50.

This paper studies a problem of graph translation, which aims at learning a graph translator to translate an input graph to a target graph. The authors propose an adversarial training framework to learn the graph translator, where a discriminator is trained to discriminate between the true target graph and the translated graph, and the translator is optimized by fooling the discriminator. The authors conduct experiments on both synthetic and real-world datasets. The results prove the effectiveness and the efficiency of the proposed approach over many baselines.
Strengths:
1. The problem is new and well-motivated.
Data translation is an important problem and has been widely studied in many research domains such as computer vision and natural language processing. Despite the importance, the problem has not been thoroughly explored in the graph domain, and most existing studies only focus on standard graph generation problems. In this sense, this paper studies a very new problem, which is quite novel. Moreover, the problem is important, which can have many potential downstream applications on graph data. Overall, the problem is new and well-motivated.
2. The proposed approach is quite intuitive.
The paper proposes an adversarial training approach to the problem, where a graph translator is learned based on a graph discriminator. During training, the graph discriminator aims at discriminating between the true target graph and the translated graph conditioned on an input graph, and the graph translator is trained by fooling the discriminator. The graph translator is built on top of an encoder-decoder framework, where the encoder and decoder are parameterized by graph neural networks. Overall, the proposed method is quite reasonable, which is easy to follow.
3. The results are promising.
The authors conduct extensive experiments on both synthetic and real-world datasets, and compare the proposed approach against many strong baseline methods for graph generation. The results are quite promising, which prove both the effectiveness and the efficiency of the approach.
Weaknesses:
1. The novelty of the proposed approach is limited.
The proposed approach is mainly built on top of the adversarial training framework, where a graph neural network is used to parameterize the graph translator. For adversarial training, although it is very intuitive, such a framework has been widely explored in the data translation problem in other domains, such as image style transfer in CV and text style transfer in NLP. Compared with these works, although the proposed approach studies a new problem, the major idea is the same as the existing studies. For the graph encoder and graph decoder, they are designed based on the idea of graph neural networks, where some propagation layers are designed to propagate information across different nodes. Although the propagation layers are specifically designed for the graph translation problem, I feel like they are not so different from existing studies (e.g, message passing neural network, graph U-net). Therefore, from the model-wise, this paper combines several existing ideas, but does not provide new insights or techniques, so the contribution is quite limited.
2. The writing can be further improved.
The paper is not very well-written. Some parts of the paper are quite hard to follow, and the intuition behind the approach is not well explained. In section 3.2, it is said that "the approach learns global information by looking for more virtual neighbors regarding the latent relations". Here, it is unclear to me what is a virtual neighbor, and what is a latent relation. The authors try to illustrate their idea in figure 2, but the figure is also quite hard to understand. It would be better if the authors could explain the idea of the encoder in a more intuitive way, or give more concrete examples for illustration. Besides, equation (4) (5) and (7) are also hard to understand. The notations in these equations are quite messy, where multiple indices are used (e.g., i, j, k, l, m, n), and the intuition underlying the equations is not well explained, making it hard to understand how the encoder and the decoder work.
Also, there are many typos in the paper. For example:
In the directed graph, each node have incoming edge(s) and out-going edge(s) -> In the directed graph, each node has incoming edge(s) and out-going edge(s)
in the "node comvolution" layer -> in the "node convolution" layer
First, the “node deconvolution” layer are used to generates -> First, the “node deconvolution” layer is used to generate
The caption of table 1 says that the table shows the node degree distribution distance, but from the main body of texts, only four metrics are about the distribution distance, which is inconsistent to the caption.
Overall, the intuition of the proposed approach is not well explained, and there are many typos to be fixed, so I feel like the writing of the paper should be further improved.


Robustness of convolutional neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed perturbations which are imperceptible to humans but can cause the network to give incorrect outputs. Recent research suggests that the noises in adversarial examples break the textural structure, which eventually leads to wrong predictions by convolutional neural networks. To help a convolutional neural network make predictions relying less on textural information, we propose defective convolutional layers which contain defective neurons whose activations are set to be a constant function. As the defective neurons contain no information and are far different from the standard neurons in its spatial neighborhood, the textural features cannot be accurately extracted and the model has to seek for other features for classification, such as the shape. We first show that predictions made by the defective CNN are less dependent on textural information, but more on shape information, and further find that adversarial examples generated by the defective CNN appear to have semantic shapes. Experimental results demonstrate the defective CNN has higher defense ability than the standard CNN against various types of attack. In particular, it achieves state-of-the-art performance against transfer-based attacks without applying any adversarial training.

The paper deals with robustness against adversarial attacks. It proposes to blank out large parts of the early convolution layers in a CNN, in an attempt to shift the focus from "texture" to "shape" features. This does seem to improve robustness against adversarial examples, with only a small decrease in general classification performance. The explanation for this, on the other hand, is not really convincing.
The idea is simple: adversarial noise introduces high-frequency texture patterns, so destroy those by blanking out a large portion of the neurons in a layer. Quite obviously, this can have an influence - when blanking out 90% of the pixels (as suggested in the paper), the effective sampling resolution goes down by a factor of 3 in each axis, and high-frequency patterns are a lot less likely to be picked up. It does, however, remain unclear why this approach is particularly useful, or why it even works at all. On the one hand, an easier way to surely get rid of those patterns is simply to blur the images accordingly before feeding them to the network. That baseline is missing. On the other hand it is quite an outrageous claim that one can throw away 90% of the responses in the early conv layers with hardly any performance loss. I don't doubt the experiments results, but if one discovers something like that, it needs to be explained. The network has  a lot lessless capacity, effectively loses a factor 3 in resolution, but performance seemingly stays almost the same!
This brings me to another point. It is never really defined what is meant by "texture" respectively "shape". By reverse-engineering from the paper text I gather that "shape" is simply texture at a significantly lower resolution. But then how does destroying "texture" affect objects with significantly lower size/scale in the image?
A few technical questions remain unclear. First, the adversarial noise in the paper looks a lot stronger than normal. It is easily visible, and in that sense not "adversarial". In fact the paper openly states that they "even fool humans". So since the labels are human-annotated, these are in fact not adversarial examples of class A, but examples of a different class wrongly labeled as class A...
Another question is how much "texture" is lost, since the paper finds it important to use a different random mask for each filter in a layer. So does that really suppress so much texture? Almost all pixels will be seen by some non-defective filters, so it would seem that the hi-res information is implicitly still there. To really suppress texture it would seem more effective to always use the same mask, but that apparently does not work. Why?
What completely confused me was which networks were actually used to generate the adversarial examples. Are these adversarial against the standard CNN or against the defective one? If defective convolutions indeed become popular, then an attacker would obviously know about that and also use a defective network to generate his adversarial examples.
One thing I did not understand is the incoherent mixture of datasets. The first experiment with the reshuffled tiles is done on ImageNet. But then the actual experiments regarding robustness against attacks are done only with Cifar-10. Why suddenly switch? And then, many of the visual examples are from TinyImageNet. Why switch again? And on that account, since apparently all of them were processed, is the behaviour consistent across datasets?
A note aside, I am not sure it is a good idea to treat additive Gaussian noise in the same way as adversarial patterns. Some level of noise that is at least approximately Gaussian is present in almost all images. So it is actually a good thing if a network learns the magnitude of that noise, so as to separate it from the signal, i.e., the brightness variations that are informative and not Gaussian noise. In that view it is a good thing, not a weakness, if adding noise of the wrong magnitude misleads the network (although, ideally, it should of course flag the image as being out-of-distribution).
In summary, I find the results interesting - in particular also the one on tiled and reshuffled images. But I am at this point not convinced by the explanations. If one can indeed throw away 90% of all responses in the low layers then that would be a rather big thing that needs an explanation. Unless the task is easy enough to be solved with 3x lower resolution - but in that case I would expect that simply reducing the resolution would also destroy the adversarial pattern. I am on the fence, but  in its current state the paper leaves too many open questions.


In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions. Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice. However, contrary to conventional wisdom that largely assigns blame for this phenomena on the undue influence of KL-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks. In particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances. Regardless, the underlying message here is not meant to undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.

Summary:
This paper is clearly written and well structured. After categorizing difference causes of posterior collapse, the authors present a theoretical analysis of one such cause extending beyond the linear case covered in existing work. The authors then extended further to the deep VAE setting and showed that issues with the VAE may be accounted for by issues in the network architecture itself which would present when training an autoencoder.
Overall:
1) I felt that Section 3 which introduces categorizations of posterior collapse is a valuable contribution and I expect that these difference forms of posterior collapse are currently under appreciated by the ML community. I am not certain that the categorization is entirely complete but is nonetheless an excellent step in the right direction. One source of confusion for me was the difference between sections (ii) and (v) --- in particular I believe that (ii) and (v) are not mutually exclusive.
Additionally, the authors wrote "while category (ii) is undesirable, it can be avoided by learning $\gamma$". While this is certainly true in the affine decoder case it is not obvious that this is true in the non-linear case.
2) Section 4 provides a brief overview of existing results in the affine case and introduces a non-linear counter-example showing that local minima may exist which encourage complete posterior collapse.
3) On the proof of Proposition 4.1. In A.2.1 you prove that there exists a VAE whose ELBO grows infinitely (exceeding the local maxima of (7)). While I have been unable to spot errors in the proof, something feels odd here. In particular, the negative ELBO should not be able to exceed the entropy of the data which in this case should be finite. I've been unable to resolve this discrepancy myself and would appreciate comments from the authors (or others). The rest of the proof looks correct to me.
4) I felt that section 5 was significantly weaker than the rest of the paper. This stemmed mostly from the fact that many of the arguments were far less precise and less rigorous than those preceding. I think the presentation of this section could be significantly improved by focusing around Proposition 5.1.
a) Section 5 depends on the decoder architecture being weak, though this is not clearly defined formally. I believe this is a sensible restriction which enables analysis beyond the setting of primary concern in Alemi et al. (and other related work).
b) In the third paragraph, you write "deep AE models can have bad local solutions with high reconstruction [...]". I feel that this doesn't align well with the discussion in this section. In particular, I believe it would be more accurate to say that IF the autoencoder has bad local minima then the VAE is also likely to have category (v) posterior collapse.
c) Equation (8) feels a little too imprecise. Perhaps this could be formalized through a bias-variance decomposition of the right hand side similar to Rolinek et al.?
d) The discussion of optimization trajectories was particularly difficult to follow. It is inherently difficult to reason about the optimization trajectories of deep auto-encoding models and is potentially dangerous to do so. For example, perhaps the KL divergence term encourages a smoother loss landscape and encourages the VAE to avoid the local stationary points that the auto-encoder falls victim to.
e) It is written, "it becomes clear that the potential for category (v) posterior collapse arises when $\epsilon$ is large". This is not clear to me and in fact the analysis seems more indicative of collapse presented in category (ii) (though as mentioned above, I am not convinced these are entirely separate). Similarly, later in this section it is written, "this is more-or-less tantamount to category (v) posterior collapse". I was also unable to follow this reasoning.
f) "it is actually the AE base architecture that is effectively the guilty party when it comes to posterior collapse". If the conclusions are to be believed, this only applies to category (v) collapse.
g) Unfortunately, I did not buy the arguments surrounding KL annealing at the end of section 5. In particular, KL warm start will change the optimization trajectory of the VAE. It is possible that the VAE has a significantly worse loss landscape than the autoencoder initially and so warm-start may enable the VAE to escape this difficult initial region.
Minor:
- The term "VAE energy" used throughout is not typical within the literature and seems less explicit than the ELBO (e.g. it overlaps with energy based models).
- Equation (4) is missing a factor of (1/2).
- Section 3, in (ii), typo: "assumI adding $\gamma$ is fixed", and "like-likelihood". In (v), typo: "The previous fifth categories"
- Section 4, end of para 3, citep used instead of citet for Lucas et al.
- Section 4, eqn 6 is missing a factor of 1/2 and a log(2pi) term.
- Section 5, "AE model formed by concatenating" I believe this should be "by composing".
- Section 5, eqn 10, the without $\gamma$ notation is confusing and looks as though the argmin does not depend on gamma. Presumably, it would make more sense to consider $\gamma^*$ as a function of $\theta$ and $\phi$.
- Section 5 "this is exactly analogous". I do not think this is _exactly_ analogous and would recommend removing this word.


Learning with noisy labels is a common problem in supervised learning. Existing approaches require practitioners to specify noise rates, i.e., a set of parameters controlling the severity of label noises in the problem. The specifications are either assumed to be given or estimated using additional approaches. In this work, we introduce a technique to learn from noisy labels that does not require a priori specification of the noise rates. In particular, we introduce a new family of loss functions that we name as peer loss functions. Our approach then uses a standard empirical risk minimization (ERM) framework with peer loss functions. Peer loss functions associate each training sample with a certain form of "peer" samples, which evaluate a classifier' predictions jointly. We show that, under mild conditions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near optimal classifier as if performing ERM over the clean training data, which we do not have access to. To our best knowledge, this is the first result on "learning with noisy labels without knowing noise rates" with theoretical guarantees. We pair our results with an extensive set of experiments, where we compare with state-of-the-art techniques of learning with noisy labels. Our results show that peer loss functions based method consistently outperforms the baseline benchmarks, as well as some recent new results. Peer loss provides a way to simplify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function in such situations.

This paper proposed peer loss function for learning with noisy labels, combining two areas learning with noisy labels and peer prediction together. The novelty and the significance are both borderline (or below). There are 4 major issues I have found so far.
References: Looking at section 1.1 the related work, the references are a bit too old. While I am not sure about the area of peer prediction, in the area of learning with noisy labels (in a general sense), there were often 10 to 15 papers from every NeurIPS, ICML, ICLR and CVPR in recent years. The authors didn't survey the literature after 2016 at all... Nowadays most papers focus on sample selection/reweighting and label correction rather than loss correction in this area, but there are still many recent papers on designing more robust losses, see https://arxiv.org/abs/1805.07836 (NeurIPS 2018 spotlight), https://openreview.net/forum?id=rklB76EKPr and references therein. Note also that some label-noise related papers may not have the term label noise or noisy labels in the title, for example, https://openreview.net/forum?id=B1xWcj0qYm (ICLR 2019).
Motivation: The motivating claim "existing approaches require practitioners to specify noise rates" is wrong... Many loss correction methods can estimate the transition matrix T (which is indispensable in any loss correction) without knowing the noise rate, when there are anchor points or even no anchor points in the noisy training data. See https://arxiv.org/abs/1906.00189 (NeurIPS 2019) and references therein. See also the public comment posted by Nontawat when a special symmetric condition is assumed on the surrogate loss function.
Novelty: The paper introduced peer prediction, an area in computational economics and algorithmic game theory, to learning with noisy labels. This should be novel (to the best of my knowledge) and I like it! However, the obtained loss is very similar to the general loss correction approach, see https://arxiv.org/abs/1609.03683 (CVPR 2017 oral). This fact undermines the novelty of the paper, significantly. The authors should clarity the connection to and the difference from the loss correction approach.
Significance: The proposed method focuses on binary classification, otherwise the paper will be much more significant! Note that the backward and forward corrections can both be applied to multi-class classification. Moreover, similar to many theory papers, the experiments are too simple, where single-hidden-layer neural networks were trained on 10 UCI benchmark datasets. I have to say this may not be enough for ICLR that should be a more deep learning conference.