Few-shot classification aims to recognize novel categories with only few labeled images in each class. Existing metric-based few-shot classification algorithms predict categories by comparing the feature embeddings of query images with those from a few labeled images (support examples) using a learned metric function. While promising performance has been demonstrated, these methods often fail to generalize to unseen domains due to large discrepancy of the feature distribution across domains. In this work, we address the problem of few-shot classification under domain shifts for metric-based methods. Our core idea is to use feature-wise transformation layers for augmenting the image features using affine transforms to simulate various feature distributions under different domains in the training stage. To capture variations of the feature distributions under different domains, we further apply a learning-to-learn approach to search for the hyper-parameters of the feature-wise transformation layers. We conduct extensive experiments and ablation study under the domain generalization setting using five few-shot classification datasets: mini-ImageNet, CUB, Cars, Places, and Plantae. Experimental results demonstrate that the proposed feature-wise transformation layer is applicable to various metric-based models, and provides consistent improvements on the few-shot classification performance under domain shift.

In this paper, the authors propose a feature-wise transformation layer for the cross-domain few-shot classification task. Besides, they apply the learning-to-learn procedure to tune the hyperparameters automatically. The primary motivation behind this method does make a lot of sense for me. The reduction of significant shifts in the feature norm could be crucial for successfully transferring the source domain model to the target domain. Also, such a method is method-agnostic, which makes the proposed layer more accessible for all kinds of metric-based query&support pipeline. The extensive experiments for three baselines, MatchingNet, RelationNet, and GNN, indicate the generalizable effectiveness of the proposed layer. Specifically, the visualization and analysis of learned feature space and layer parameters look interesting to me. Remember that the proposed method is designed to minimize the large discrepancy of the feature distribution. Such a study could be beneficial for the reader to understand this paper in depth.
Meanwhile, I have some questions and suggestions for the authors:
1. I am suggesting the authors include more recent state-of-the-art as baselines and comparisons, which could make such submission much stronger.
2. This paper delivers an extensive study of the classification problem, how about the other tasks, which heavily rely on classification head, like detection or segmentation. I think it could be more attractive if the author could also show some improvement for these tasks in either a qualitative or quantitive way.
3. This paper mainly focuses on metric-based few-shot frameworks. At the same time, there are also other two groups, recurrent-based and optimization-based schemes. It could be much better if the authors could also mention the potential of the proposed layer.
4. This paper leverages the learning-to-learn mechanism to tune the hyper-parameters. How about considering the adaptive or dynamic approach. In other words, build the connection between the theta parameter and image feature.


This paper presents the ballistic graph neural network. Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline. The filters propagate exponentially faster(
2 ) comparing to traditional graph neural network( 2 ⇠ T ). We use a perturbed coin operator to perturb and optimize the diffusion rate. Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters. We also show the perturbed filters act as better representations comparing to pure ballistic ones. We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network's performance can be improved.

This paper was extremely hard to read or comprehend. It’s riddled with typos, inaccurate notations and undefined variables (see below for a sampling). The authors will need to significantly polish and improve the presentation of the paper.
After a few forward and backward passes through the paper, I was able to gather the following high level ideas about the paper:
(1) This paper is somewhat related to the Defferard et. al, 2016 in that the authors want to define a propagation filter for graph neural networks.
2) This proposed filter known as “ballistic filter” should have the property of allowing fast diffusion through the network.
(3) The authors claim that the ballistic kernel diffuses @ O(k) as compared to O(\sqrt k) when compared to traditional GCNs, where k is the number of propagation steps.
(4) The authors additionally claim that their approach needs one-third the number of parameters.
(5) The authors provide some plots to visualize the linear diffusion rate of their proposed filter.
--- Issues and clarifications ---
- Sec 3, Eq 1 seems to have been taken from Eq 1 in Defferard et. al, however there’s no reference to it and the terms g, U, etc. are not defined.
 - Sec 4, Algo 1 contains the main core of the proposed algorithm, but it’s only defined for the 2D grid case. The notation therein is extremely unclear. What is H_space, H_c? How does one sample \hat{O}_coin.? The net result is that algorithm is undefined. Without a clear definition of the algorithm, it’s completely unclear what the proposed method does.
 - Sec 4.2 is completely unparseable. What is problem setting? What is the metric? How have the baselines been implemented? How has data been split for training/testing?
 - Section 5 mentions that one-third params are used to get 97% but no details are provided as to how less params are consumed.
 - How is figure 7 generated?
 - Sec 8, feel totally unrelated to the paper. There are a whole bunch of random, unmotivated diffusion equations  Eq 6, mentions “.. \hat{g}(f) decreases as f increases and thus can be seen as a low pass filter…” . This is not true from the formula.

 --- A sampling of typos ---
Sec 4.1, .. consisits …
Sec 5 “REVISIT” -> “REVISITING”
Figure 6, text, “cassical”
Sec 6.2 title, “SUMMAY”
Sec 8  “aggreated”
Sec 8  t=\-tau to -\tau
Several typos with Hardmard, Hadmard instead of Hadamard.
Overall, the major criticisms of this paper:
 - The proposed algorithm is not clear.
 - The authors need much more experimentation to bolster their claims in the paper. It’s completely unclear if fast diffusion even if it were possible will help GNNs perform better on a diverse set of tasks.
 - The paper needs a lot more polish and proof reading to make this paper presentable.


We study the stochastic contextual bandit problem, where the reward is generated from an unknown bounded function with additive noise. We propose the NeuralUCB algorithm, which leverages the representation power of deep neural networks and uses a neural network-based random feature mapping to construct an upper confidence bound (UCB) of reward for efficient exploration. We prove that, under mild assumptions, NeuralUCB achieves O( √ T ) regret, where T is the number of rounds. To the best of our knowledge, our algorithm is the first neural network-based contextual bandit algorithm with near-optimal regret guarantee. Preliminary experiment results on synthetic data corroborate our theory, and shed light on potential applications of our algorithm to real-world problems.

The authors proposed a neural network based UCB algorithm for bounded reward contextual bandit problems with theoretical guarantee thanks to the recent development of Neural Tangent Kernel (NTK).
Throughout the paper, the authors do not use much of the specific property of neural networks and NTK. They only use the gradient of neural networks as the feature and use the NTK as the kernel in the kernelized contextual bandits. This can be beneficial, for example, it can enriching the class of kernels. However, I feel the whole paper lacks novelty, and have some technical flaws.
Detailed Comments:
1. In Sec 3, the authors argued that kernelized contextual bandits suffers from the unknown RKHS problem and RKHS realizability problem. However, with universal kernel, RKHS is dense in L^2 space, thus can in principle approximate any function in L^2 space within any precision. So generally, this is not a problem. Moreover, bounded function does not necessarily contain the linear function, generalized linear function and bounded RKHS norm function. At least if we do not add some assumption on the input, linear function can be unbounded. On the other hand, the proposed methods also need p>TK to guarantee the realizability, and we can also design some kernel with feature map dimension larger than TK with some good property to guarantee realizability, so I think this claim is not fair.
2. In Assumption 5.2, the authors assume that the norm of contexts is smaller than 1. However, as far as I know, most of the existing work assumed the context have norm 1, and can be only relaxed to the norm upper lower bounded by two positive constant c1 and c2 (see [1]). Otherwise, there can be some issue on the positive definiteness of the NTK. Can the authors carefully check this? Meanwhile, I think it is not suitable to directly assume the NTK is positive definite. It is better to follow and refer the readers to the existing work.
3. It is better to introduce \theta^* before Sec 6, like for example the parameter that can perfectly predict the mean reward.
4. I am confusing on the proof of Lemma 6.1 in Page 12. When the authors calculate the norm of \theta^* - \theta_0, how to transform Q^\top A^{-2} Q to G^\top G? If we use the singular value decomposition, we only have that G^\top G=QA^2 Q^\top. If I understand correctly, here we should do an inverse, and we cannot simply get the desired results, as the minimum singular value of G can be small under current assumption. However, it is still possible to upper bound this distance to derive the remaining proof.
5. In the first line of Equation (B.3), there is a typo that omits the \phi(x)^\top.
6. How does the second inequality of (B.4) derives? I can understand that the authors may use the Cauchy-Schwartz inequality, but Frobenius norm cannot be directly upper bounded by spectral norm (though they are equivalent, but we need to add an additional constant like \sqrt{TK}). If the authors use the spectral norm, then the second term should be nuclear norm, not Frobenius norm. Probably I do not understand it correctly and it is not the core issue, but I think it is better to clarify it.
7. There is a typo in the fourth line of (B.4), it should be \lambda / \lambda_0,
8. The last derivation of Appendix B.3 have several typos omitting det(\lambda I).
9. The authors should better include the kernelized contextual bandits for a fair comparison, as LinUCB and Neural \epsilon-greedy both have theoretical issue that can be solved by kernel methods. I doubt that kernelized contextual bandits can solve these cases well.
Overall, I feel that most of the proof can be derived similarly from [2][3]. And Lemma C.1 is also from [4]. The authors only verify some conditions that when use NTK as the kernel in kernelized contextual bandits to adjust the main result from [2][3]. Thus, I think the technique used in this paper is not novel as well.
In my opinion, the communities are interested in solving contextual bandits with ``''gradient based'' neural network methods that use the neural network to predict the rewards given some contexts as input. But just as the Equation (6.1) shows, the prediction is not based on the neural network, but with a linear model taken \phi(x_i) as input. Also, throughout the paper, the authors never use the network output f. To this end, I feel this paper is over-claimed on ''neural''. On the other hand, I think it can be interesting to think about how kernel methods can benefit from NTK. Directly use the gradient as the feature map seems not an interesting and meaningful method, I think.
[1] Cao, Yuan, and Quanquan Gu. "A generalization theory of gradient descent for learning over-parameterized deep relu networks." arXiv preprint arXiv:1902.01384 (2019).
[2] Michal Valko, Nathan Korda, Rémi Munos, Ilias Flaounas, and Nello Cristianini. 2013. Finite-time analysis of kernelised contextual bandits. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI'13), Ann Nicholson and Padhraic Smyth (Eds.). AUAI Press, Arlington, Virginia, United States, 654-663.
[3] Abbasi-Yadkori, Yasin, Dávid Pál, and Csaba Szepesvári. "Improved algorithms for linear stochastic bandits." Advances in Neural Information Processing Systems. 2011.
[4] Arora, Sanjeev, et al. "On exact computation with an infinitely wide neural net." arXiv preprint arXiv:1904.11955 (2019).


Robust Reinforcement Learning aims to find the optimal policy with some extent of robustness to environmental dynamics. Existing learning algorithms usually enable the robustness though disturbing the current state or simulated environmental parameters in a heuristic way, which lack quantified robustness to the system dynamics (i.e. transition probability). To overcome this issue, we leverage Wasserstein distance to measure the disturbance to the reference transition kernel. With Wasserstein distance, we are able to connect transition kernel disturbance to the state disturbance, i.e. reduce an infinite-dimensional optimization problem to a finite-dimensional risk-aware problem. Through the derived risk-aware optimal Bellman equation, we show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm-Wasserstein Robust Advantage Actor-Critic algorithm (WRAAC). The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.

This work aims to produce reinforcement learning methods that are ‘distributionally robust’. They approach this by assuming the transition function may vary between elements of the domain distribution and extend some recent results (Blanchet & Murthy, 2019) to give some theoretical results (contraction and optimal deterministic policy) and an actor-critic algorithm. This is an important research area and the work takes a very nice approach. However, the clarity of the work and the empirical results could both use some work.
There are a great many grammatical errors throughout the paper. The nature of the errors suggests a non-native speaker, which I really do not want to discourage, but this needs a proofread and edit. One exception, there is an actual typo just after assumption 1 “dfnied”, but actually better grammatically to just remove the word entirely.
In the intro, the citation of Mannor et al. 2004; 2007 for what is essentially a description of the gap between simulation to real transfer is a very strange choice. Is it possible this was a mistake?
The description leading up to the main result could be made much clearer. I went through this section, the main results, and the proofs in the appendix. Things look good, although I do feel like (as just said) it could be made much better/clearer. One snag I hit was in the proof of Lemma 1, everything is good except I don’t know why “u_1 \le u_2 + \gamma \| u_1 - u_2 \|_\infty” must hold, but I might have simply missed something obvious so please help me out.
The experimental results are very minimal and not very convincing. It is not immediately clear to me that Figures 1&2 actually show that the robustac algorithm is more robust than ac. The performance is worse than ac, which is completely understandable, but for most of the environments ac is still significantly better.
Lastly, a small point to notice that this paper bleeds into the 9th page, and considering the amount of contributions (largely down to the two theoretical results and a very small experiment) I do not think the extra space is warranted. This could be compressed and actually come out stronger as a result of being forced to be clearer and more concise.
Update:
Thank you for your response, especially the fix to the proof. I will keep my score, but do believe that further experiments and small adjustments to the writing will see a future version of this accepted.


Quantifying, enforcing and implementing fairness emerged as a major topic in machine learning. We investigate these questions in the context of deep learning. Our main algorithmic and theoretical tool is the computational estimation of similarities between probability, "à la Wasserstein", using adversarial networks. This idea is flexible enough to investigate different fairness constrained learning tasks, which we model by specifying properties of the underlying data generative process. The first setting considers bias in the generative model which should be filtered out. The second model is related to the presence of nuisance variables in the observations producing an unwanted bias for the learning task. For both models, we devise a learning algorithm based on approximation of Wasserstein distances using adversarial networks. We provide formal arguments describing the fairness enforcing properties of these algorithm in relation with the underlying fairness generative processes. Finally we perform experiments, both on synthetic and real world data, to demonstrate empirically the superiority of our approach compared to state of the art fairness algorithms as well as concurrent GAN type adversarial architectures based on Jensen divergence.

This paper proposes a variant of adversarial learning to achieve some of the popular group fairness definitions. The main novelty is the idea of minimizing Wasserstein distance between the conditional distributions of classifier predictions given different values of the protected attribute.
My main concern is the approximation of a simple 1d Wasserstein distance with a neural network. Wasserstein distance between two discrete distributions in 1d can be computed in closed form (simple function of order statistics). That is, eq. (1) is simple to evaluate for two empirical distributions. There is no need to use a neural network for approximation, and even if authors choose to do so, some discussion on how well it approximates actual Wasserstein distance is needed. I think the proposed algorithm could be more interesting if authors can work out the optimization problem with the actual Wasserstein distance.
On the theoretical/motivation side, it is not enough to say that demographic parity is achieved when the corresponding Wasserstein distance is 0. What is needed is that demographic parity difference is bounded from above by the corresponding Wasserstein distance (I don't know if it is true or not, but would like to know). Then minimizing Wasserstein distance to achieve demographic parity could be justified.
Finally, the paper is quite poorly written. The description of fairness in the introduction is very vague. Authors essentially describe demographic parity as fairness, while it is simply one of the several definitions of group fairness. There is also individual fairness (the paper by Dwork et al. is cited, but not properly discussed) and prior work emphasizing certain deficiencies of group fairness [1] along with several recent papers studying individual fairness [2,3], some also utilizing Wasserstein distance [4].
Authors also provided incorrect definition of disparate impact. Equation in the bottom of page 2 corresponds to statistical parity difference, while disparate impact is the ratio.
"Equality of opportunity" on the top of page 3 seems to be a typo
"the mathematical properties of the disparate impact measure are not favorable, in particular it lacks robustness and smoothness features which would be necessary to blend algorithmic practice and mathematical theory" - I don't think this claim makes sense. There are many prior works studying disparate impact and proposing algorithms to achieve it, e.g. the cited work of Feldman et al. Authors should be more specific regarding what mathematical properties they consider not favorable.
There are a lot of typos and grammatical mistakes, e.g.
in the 1st paragraph of section 2.2, the sentence "Hence the aim in this case is to” is unfinished.
in the 1st paragraph of section 3, the first sentence seems to be unfinished.
[1] Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores.
[2] Kearns, M., Roth, A., & Sharifi-Malvajerdi, S. (2019). Average Individual Fairness: Algorithms, Generalization and Experiments.
[3] Jung, C., Kearns, M., Neel, S., Roth, A., Stapleton, L., & Wu, Z. S. (2019). Eliciting and Enforcing Subjective Individual Fairness.
[4] Yurochkin, M., Bower, A., & Sun, Y. (2019). Learning fair predictors with Sensitive Subspace Robustness.


Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation.

This paper introduces a Self-Adversarial Learning (SAL) mechanism in GAN based text generation, aiming at tackling the problem of mode collapse and sparse rewards problem. Specifically, motivated by “self-play” mechanism in RL community, instead of using a binary classifier as discriminator in original GAN, SAL employs a comparative discriminator which is a pairwise classifier with three classes: “better”, “worse” and “indistinguishable”. The authors provide lots of experimental results and ablation study showing the efficiency of the proposed mechanism in comparison with previous GAN models.
Decision: weak reject.
This paper is well motivated: clearly sparse rewards and mode collapse are two problems need to be solved in GAN based text generation, however, the following concerns prevent me from finding this paper acceptable in ICLR:
The “self-play” idea is widely used in RL. In RL, the comparison between policies can be determined directly by the game simulation results. In text generation, such comparison is more difficult to be judged. This paper appears to assume that the generated sentences are “worse” than the real samples, which is similar to the original definition of discriminator in GAN, and the generated sentences in earlier epochs are “worse” than that in later epochs, which needs further justified.
As a simple extension to GAN, I’m not convinced that the problem of mode collapse could be solved by proposed mechanism. If the generator falls into a local minimum, a collapsed mode, the proposed mechanism will never pull the generator out of that. Moreover, what is the theoretical foundation of the proposed evaluation metric on quality-diversity trade-off, NLL_{gen} + NLL_{oracle}?
The setting of the important set of hyper-parameters, reward weights, is unclear in the paper. The reward weights directly influence the reward in training, thus should play an important role of model performance. More discussion about this should be provided.
Moreover, in the paper, only comparison between proposed mechanism with GAN based models are shown. Comparison with more recent models like RelGAN should be provided. And comparison with other state-of-the-art text generation model should be discussed.
Feedbacks:
References regarding experimental results in table are incorrect. For example, the results in synthetic data should be in “Table 2” and COCO image caption dataset should be in “Table 3”.
Some imprecise parts, for example, in Equation (5) and (6), it should be G_\theta(Y_{1:t-1} and G_\theta(y_t | Y_{1:t-1} .
I’m curious about what the performance would be like if the weakly supervision by regarding sentences generated in later training stage are “better” than the sentences generated in earlier training stage is removed in training comparative discriminator. This is different from the CAL model in the ablation study.
What is the influence of different values of rewards weight?
Also about the rewards weight, in the description of Scheduled rewarding, the rewards weight is described to be linearly changed with training iteration, while in Appendix C.3, the rewards weight is described to be fixed. This is very confusing.
A minor issue: for image captioning, a lot other metrics are widely used in measuring the model performance, e.g. CIDEr, SPICE and so on. Those metrics could be helpful for audience to understand how the model performs in comparison with other captioning models.
Another minor issue: for image captioning and WMT (conditional generation), the detailed model structures are not described in paper, which is not very friendly to audiences with relatively little knowledge in related areas.


Pruning is an efficient model compression technique to remove redundancy in the connectivity of deep neural networks (DNNs). A critical problem to represent sparse matrices after pruning is that if fewer bits are used for quantization and pruning rate is enhanced, then the amount of index becomes relatively larger. Moreover, an irregular index form leads to low parallelism for convolutions and matrix multiplications. In this paper, we propose a new network pruning technique that generates a low-rank binary index matrix to compress index data significantly. Specifically, the proposed compression method finds a particular finegrained pruning mask that can be decomposed into two binary matrices while decompressing index data is performed by simple binary matrix multiplication. We also propose a tile-based factorization technique that not only lowers memory requirements but also enhances compression ratio. Various DNN models (including conv layers and LSTM layers) can be pruned with much fewer indices compared to previous sparse matrix formats while maintaining the same pruning rate.

The paper addresses the problem of reducing the computational complexity of neural network pruning. Main idea is to compute a low-rank approximation of the binary index matrix used to represent the structure of the pruned network. In the considered setup, the binary index matrix is the (sparse) boolean matrix associated with the nonzero network's weights. As low-rank decomposition of binary matrices is a hard problem, the authors propose a method to approximate the solution by computing a more standard non-negative matrix factorization.
This is a nice problem and the proposed approach is interesting  but I would tend to reject the paper. The main problem, in my opinion, is a substantial lack of theoretical justifications.  What are the intuitive motivations for considering the proposed method? It is hard to understand why the approach should be preferred to others.  The idea of converting the binary low-rank factorization problem into a real-valued non-negative factorization problem is interesting but
could have been better justified (from an intuitive/theoretical and computational perspective). For example, why is it convenient to convert a hard matrix factorization problem (BMF) into another hard matrix factorization problem (NMF).
An empirical or theoretical analysis of the algorithm's computational complexity would help in this sense.
Also, it is not completely clear what is the relationship of the proposed approach with the neural network framework. Would the impact of the paper increase if presented as a method for binary matrix factorization without links to neural networks pruning? Perhaps a quantitative comparison between gains associated with sparse matrix storage versus other computational costs (related to training or pruning) would help to better collocate the proposed approach in the deep learning framework.
Questions:
- The general idea of approximating BMF with NMF is interesting and could be investigated independently and more deeply. Have the problem and the proposed solution appeared before in the matrix factorization literature (without connections to neural network pruning )?
- Is there any intuitive justification of why thresholding the matrix before and after the factorization leads to consistent results?
- The cost of the proposed approach seems to depend on the rank. Could such explicit dependence be estimated quantitatively and compared with the computational complexity of solving the problem (BMF) directly?
- In the experiments, the value of the pruning objective seems to decrease as the rank of the factorization increases. Is this an expected result? What is even less clear to me is the behaviour of the test accuracy. Is it normal for the pruned network accuracy to increase with sparsity? And why does the performance look independent from the rank?


Robustness is an important property to guarantee the security of machine learning models. It has recently been demonstrated that strong robustness certificates can be obtained on ensemble classifiers generated by input randomization. However, tight robustness certificates are only known for symmetric norms including 0 and 2 , while for asymmetric norms like 1 , the existing techniques do not apply. By converting the likelihood ratio into a one dimensional mixed random variable, we derive the first tight 1 robustness certificate under isotropic Laplace distributions in binary case. Empirically, the deep networks smoothed by Laplace distributions yield the state-of-the-art certified robustness in 1 norm on CIFAR-10 and ImageNet.

Summary.
The authors propose a new certified classifier in \ell_1 norm that is tight. That is to say, upon smoothing a given classifier f with Laplacian noise, a smoothed version of that classifier (probabilistic maximum majority vote) is certified with a radius measured in \ell_1 norm. The authors show that this bound is tight for binary classifiers. These results are complementary to Cohen et al. results.
Major comments.
1) The major contribution of this paper is the tightness under the \ell_1 norm for a binary classifier. I do not find this particularly significant. The question is of what value is such a result other than a mathematical exercise. For instance a good justification that the paper is lacking could be one where authors show that their radius is indeed tighter than all other works. The paper still lacks this (I will elaborate on this later), although, their bounds are indeed tighter than Lecure's et al. Since it is not clear whether or not the new certified smoothed classifier has indeed the largest radius among all other works, then at least a justification for why would one prefer a Laplacian noise of a Gaussian noise. Why is Gaussian smoothing sufficient for this purpose given that we do not know for sure that the radius is larger?  What value/advantages does this add? The authors motivate their work by saying deriving the tightest \ell_1 is difficult due to the "asymmetry"  of the norm. While I do agree on this; however, this is not enough motivation as we we are doing doing abstract maths here.
The new derived radius is not really comparable to the Gaussian radius with \ell_2 radius and this is my major concern. By norm equivalence, we have that \ell_2 \leq \ell_1 \leq \sqrt{n} \ell_2 where n is the dimension. That is to say that the radius computed with \ell_1 is larger than the \ell_2 in some cases by a square root of dimension. The authors can correct me on this if I'm wrong, but for a fair comparison in worst case sense the radius of Cohen et al. should be scaled by \sqrt{n}. In such a scenario, it is really difficult to understand when does it make sense to tackle such a smoothing technique as opposed to Gaussian smoothing.
I would not have asked the authors about such a question if the authors derived generic radius under \ell_p smoothing (which is difficult of course). To this end, I believe since the motivation is not clear nor the results are generic enough, I find the work incremental specifically after noting that the radius can be deduced from the work of Li et al. where the main contribution here is the tightness of the radius for a binary classifier.
Moreover, I believe the paper still requires some polishing in terms of writing and presentation.
Some more comments.
I believe the paper can benefit from some rewriting. Here is a list of things the authors can do to improve the paper.
1) Define what M is, page 3 "and it is easy to see that M is a mixed random variable". I believe the authors meant T(x).
2) The figures are hardly readable. For instance, authors can perhaps increase the legend's font size in figures 4. Also the chosen colors are suboptimal (perhaps the line width of the plots) should be increased.
3) The section below Theorem 3 should be moved up to before Theorem 3 as this discusses the proof of Theorem 2. Once a Theorem is presented, the proof sketch should follow.
4) Experiments on the undefended classifier has to be in Figures 6  7 and 8.
5) Lastly, why are comparison between Cohen et. al. and Lecuyer et. al. in Figure 6 inconsistent with Figure 5 of Cohen et al.


In many applications, it is desirable to extract only the relevant information from complex input data, which involves making a decision about which input features are relevant. The information bottleneck method formalizes this as an informationtheoretic optimization problem by maintaining an optimal tradeoff between compression (throwing away irrelevant input information), and predicting the target. In many problem settings, including the reinforcement learning problems we consider in this work, we might prefer to compress only part of the input. This is typically the case when we have a standard conditioning input, such as a state observation, and a "privileged" input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent. In such cases, we might prefer to compress the privileged input, either to achieve better generalization (e.g., with respect to goals) or to minimize access to costly information (e.g., in the case of communication). Practical implementations of the information bottleneck based on variational inference require access to the privileged input in order to compute the bottleneck variable, so although they perform compression, this compression operation itself needs unrestricted, lossless access. In this work, we propose the variational bandwidth bottleneck, which decides for each example on the estimated value of the privileged information before seeing it, i.e., only based on the standard input, and then accordingly chooses stochastically, whether to access the privileged input or not. We formulate a tractable approximation to this framework and demonstrate in a series of reinforcement learning experiments that it can improve generalization and reduce access to computationally costly information.

Summary - The paper proposes an approach, called the Variational Bandwidth Bottleneck (VBB), capable of compressing only a part of the input and still learn representations that are informative of the output. The approach is motivated from the following perspective -- there might be situations where a ``part’’ of the input is privileged in the sense that it may be costly / wasteful to maintain access to all the time, thereby rendering the standard IB pipeline infeasible (as it requires unrestricted access to the entire input). By breaking down the input into standard (always available) and privileged (not always available) components, the paper proposes a module that decides whether to access the privileged input during compression based on the standard input. The goal is to be able to decide when to access privileged information and not how to break the overall input into standard and privileged components. The approach tackles a narrow subset of problems compared to the standard information bottleneck. The authors show the applicability of the proposed approach in reinforcement learning setups -- specifically, (1) when to access an expensive model-based planner for goal-driven navigation; (2) when to access goal-information in goal-driven navigation and (3) treating communication in a multi-agent cooperative setting as ``privileged’’ information. The experimental results demonstrate that VBB accesses privileged information in a feasible and minimal manner and results in better generalization performance.
Strengths
- Apart from the flaws (highlighted in weaknesses), the paper is well-written and generally easy to follow.
- The problem statement is well-motivated. The authors did a good job of first identifying when the standard IB approach would be infeasible / costly -- even though representations are compressed and relevant, computing them still requires unrestricted access to input -- and then motivating the need for selective access to information while decision-making. The problem is well-grounded in the experimental settings the authors provide results for.
- Apart from the concerns (highlighted in weaknesses), the experimental results generally support the claims of the paper. Results in Sec. 7.1 (loosely) demonstrate that VBB accesses privileged information states where degree of freedom in terms of possible trajectories is higher -- this result, although not explored in it’s entirety, also correlates with notions of decision-states (as pointed out by authors) and bottleneck states in the existing literature. Results in Sec. 7.2 demonstrate improved generalization performance in terms of transfer. Additionally, results in Table.3 suggest that VBB accesses privileged information the least number of times. Similarly, results in Sec. 7.3 indicate VBB results in improvements in performance in the multi-agent setting while resulting in minimal communication among the agents. The baselines being compared with in the paper are also reasonable.
- The problem statement and the proposed approach have some degree of novelty. Most works along the lines of restricting access to relevant information still assume unrestricted access to the `entire’ information.
Weaknesses
That being said, the paper does have some flaws / clarity issues that make several associated details confusing when judged in context of prior work. These concerns (in addition to the strengths highlighted above) mostly surrounding experiments form the basis of my rating and addressing these would definitely make the paper stronger.
- One of the underlying motivations / intuitions behind restricting access to the privileged information is that fact that -- “..avoid accessing the privileged input because we want to generalize with respect to it..”. This statement in isolation is misleading and makes things unclear. From a complete generalization standpoint, preventing the encoder from overfitting to the input (‘privileged’ or ‘standard’) can be addressed by just tracking the relevance (predictive performance) based on the representations on a held-out set. Unless I am missing something, generalization of the learned representations in standard IB becomes major problem only when we restrict (costly or otherwise) access to the privileged input. The statement seems to suggest that learning representations completely and generally agnostic to the privileged input is a good idea. However, this is only true when at test-time, the privileged input may be significantly different than what was seen during training time -- a different model-based planner / entirely different goal-specifications, etc. Could the authors comment more on this and reframe the intuition wherever applicable?
- Last 3 lines of the 1st paragraph in section 3 (page 2) are incomplete -- “...constraining the channel capacity .. is permitted to differ from the prior r(Z)....”. The difference in the posterior p(Z|X) and the prior r(Z) quantify the channel capacity only under expectation over the distribution over inputs -- p(X). Therefore, in practice, it is also governed by the empirical distribution of the data (see page 4 of https://arxiv.org/pdf/1612.00410.pdf). The authors should change the lines to reflect the same.
- One of the key contributions of the paper is to develop a mechanism where one isn’t required to access the privileged input all the time -- depending on the standard input, one can decide whether to access the privileged information. This manifests in form of a mixture-distribution over a dirac-delta transformation of the deterministic encoder and a prior distribution. At inference-time, the channel capacity (d_cap, based on the standard input) can be used to decide whether to access the privileged information or not. There’s lack of clarity in terms of what happens at training time -- is it the case that (1) d_cap is computed, f(s, g) is also computed and based on the KL-term in Eq (3), B(S) is incentivized to generate d_cap that results in less frequent access the privileged information (d_cap < 0.5 for the most part) or (2) d_cap is computed, a sample is drawn from the bernoulli and z is sampled from either r(Z) or \del_(f(S, G) and KL is either 0 (if z ~ r(Z)) or a finite value (if z ~  \del_(f(S, G))? (1) involves an exact computation and always requires access to G (privileged input) during training whereas (2) is approximate and does not always require access to G. It’s unclear what the pipeline is from the paper. Can the authors clarify this? Are there specific reasons why (1) / (2) was chosen?
- Experimental Results: Highlighting these below:
     - In the experimental setting of Figure 2, do the authors notice any (1) qualitative differences in terms of where the agent accesses the output of the planner when InfoBot is used (adapted to this setting) and (2) quantitative differences (Table. 2) in terms of how often does InfoBot and the other proposed baselines access the planner output at junctions and hallways? Junctions in 2D mazes can potentially be identified as important decision states using simpler approaches. As a demonstrative experiment, the key point to be made using this experiment seems to be how often does VBB access the planner output compared to InfoBot and other baselines.
    - In Sec. 7.2, it is unclear how VBB is being used to generalize to novel environments -- as in, is the pipeline same as InfoBot, where a frozen encoder is used to provide an exploration incentive?
    - While the generalization results in Table. 2 are impressive in terms of success values, I think it lacks a few numbers (assuming the transfer pipeline to novel environments is same as InfoBot) -- given that the environments being tested on are different from InfoBot, how well do count-based exploration and goal-conditioned A2C baselines perform? This is to understand whether InfoBot is the right thing to compare with in these environments.
    - Furthermore, for the goal-driven navigation set of experiments can the authors report comparisons in terms of sample-efficiency as well -- how do success-rates and average task-returns vary with time-steps of training?
    - For results in Sec. 7.3, Table. 4, I would encourage the authors to compare with IC3Net (https://arxiv.org/pdf/1812.09755.pdf) which learns “when to communicate” in a multi-agent setting irrespective of whether it’s a cooperative situation or not.
Reasons for rating
Apart from the points mentioned above, I don’t have major weaknesses to point out. The paper is generally easy to follow and the proposed approach and problem statement is well-grounded and somewhat novel. However, the paper suffers from lack of experimental details and comparisons (and other weaknesses highlighted above) and therefore I am inclined towards my current rating. Addressing those would significantly benefit the paper and help me in  increasing my score.


Similarity measurement plays a central role in various data mining and machine learning tasks. Generally, a similarity measurement solution should, in an ideal state, possess the following three properties: high accuracy, high efficiency in terms of speed and independence from prior knowledge. Yet unfortunately, vital as similarity measurements are, no previous works have addressed all of them. In this paper, we propose X-Forest, consisting of a group of approximate Random Projection Trees, such that all three targets mentioned above are tackled simultaneously. Our key techniques are as follows. First, we introduced RP Trees into similarity measurement such that accuracy is improved. In addition, we enforce certain layers in each tree to share identical projection vectors, such that exalted speed is achieved. Last but not least, we introduce randomness into partition to eliminate its reliance on prior knowledge. We conduct experiments on three real-world datasets, whose results demonstrate that our model, X-Forest, reaches an efficiency of up to 3.5 times higher than RP Trees with negligible compromise on its accuracy, while also being able to outperform traditional Euclidean distance-based similarity metrics by as much as 20% with respect to clustering tasks.

This paper proposes a new method for measuring pairwise similarity between data points. The idea is to define the similarity between two data points to be the probability (over the randomness in constructing the trees) that they are close in an RP tree. More concretely, the proposed method constructs a collection of RP trees (albeit with some modifications), and takes the similarity to be the average over different RP trees of a strictly decreasing function of the distance between the leaf nodes containing the data points in each RP tree. The key modification to the RP tree is to limit the number of projection vectors used in an RP tree and re-use previous projection vectors.
I believe the method is more or less equivalent to Euclidean distance, for the following reason. Two data points would have the highest similarity under the proposed similarity measure if they are in the same leaf node. For this to happen, both points must be on the same side of the dividing hyperplane corresponding to each of the ancestor nodes. Because the threshold along the projection vector is chosen randomly at uniform, this means that for the two data points to have high similarity consistently, the distance between them along the projection vector must be small (so that the probability of splitting them is small). Because the projection vectors themselves are chosen randomly on the unit sphere, this implies that this must hold along most projection vectors for similarity to be high consistently, which means that the Euclidean distance between the two points must be low.
If true, this raises several questions:
1) Why does the proposed method work better than distance similarity (which I assume means Euclidean distance) in the experiments? Are there situations when the proposed method would yield a high similarity consistently whereas Euclidean distance wouldn’t? Are the results in Fig. 6 just for a single run of the proposed method? If so, many more runs need to be performed since the decisions of the RP tree should vary significantly depending on the projection vectors and thresholds. Both the mean and standard deviation should be reported.
2) In Sect. 1.2, the authors critiqued distance-based similarity because it often does not correspond to intuitive notions of similarity/perceptual similarity. However, it does not appear that the proposed method would correspond to perceptual similarity either. For example, consider a dataset where some coordinates are more perceptually important than others (this is the case for example for the wavelet coefficients of a natural image - the lower frequencies are typically more perceptually important than higher frequencies). A more perceptually meaningful distance than Euclidean distance would be a Mahalanobis distance (which can essentially weight different coordinates differently), but the random projections use standard inner products and so are unable to capture the appropriate weighting of the different coordinates. So, why would one expect the proposed method to be more perceptually meaningful?
3) In Sect. 1.2, the authors critiqued multi-partition-based similarity because it does depend on the data distribution and cited the elimination of “prior knowledge dependence” in Sect. 1.3 as one of the benefits of the proposed method. This appears to be at odds with the goal of devising a similarity measure that is perceptually aligned, because such a similarity measure must depend on the representation of the data (e.g.: if the data is represented in the wavelet domain, one needs to know which order the different dimensions are arranged, i.e. from lowest frequency to highest frequency or the other way around).
Overall, it is unclear if the desiderata makes sense, and if the proposed method achieves the objectives.
Other questions:
4) For the X-Projection tree (which re-uses projection vectors), it seems to be equivalent to a layer-by-layer RP tree with larger branching factor. If so, the presentation of the method should be changed to this, because a layer-by-layer RP tree with larger branching factor is both conceptually clearer and simpler to analyze. If not, the proposed method should be compared to a layer-by-layer RP tree with larger branching factor, to justify the increased conceptual complexity of the X-Projection tree.
5) For the experiments, comparisons should also be made to multi-partition-based similarity, like Multiple RP+EM and RF similarity.
6) In addition, the proposed method should be compared to two simpler baselines that computes the average and the minimum distance of the two points along multiple random projection vectors, in order to justify the increased conceptual complexity of RP trees.
7) In Sect. 1.3, the paper claims that “it is well known that in an RP Tree, data points that are closely distributed, indicating their high level of similarity in space, are always partitioned into the same subset”. This is not true, since hyperplane could divide a cluster down the middle for example.
8) One of the claimed contributions in the abstract that “we introduce randomness into partition to eliminate its reliance on prior knowledge”. Note that just by introducing randomness, prior knowledge is not necessarily eliminated. For example, the way in which random projection is performed (i.e. standard inner product vs. other inner products) assumes knowledge of the distance metric, which is induced from the inner product.
9) In Fig. 6, what is the distance metric used for the baseline?
Minor issues:
pg. 2: “project all data points into one random vector” -> “project all data points along one random vector”
pg. 3: “leading to unsatisfied results“ -> “leading to unsatisfactory results”
pg. 3: “nearest neighbours finding” -> “nearest neighbour search”
pg. 3: “pattern discovering” -> “pattern discovery”
pg. 4: “similarly data points” -> “similar data points”
pg. 4: “01 matrix” -> “0-1 matrix”


Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse crosslingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.  (Mikolov et al., 2013a;Pennington et al., 2014;Bojanowski et al., 2017) have become ubiquitous across a wide range of NLP tasks. In particular, methods for crosslingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for downstream tasks, such as text classification (Klementiev et al., 2012a), dependency parsing (Ahmad et al., 2019), named entity recognition (NER) (Xie et al., 2018;Chen et al., 2019), natural language inference , language modeling (Adams et al., 2017), and machine translation (MT) (Zou et al., 2013;Artetxe et al., 2018b;. The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages. There are two main paradigms for learning CLWE: cross-lingual alignment and joint training.
The most successful approach has been the cross-lingual embedding alignment method (Mikolov et al., 2013b), which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages. The underlying idea is to first independently train embeddings in different languages using monolingual corpora alone, and then learn a mapping to align them to a shared vector space. Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons (Xing et al., 2015;Smith et al., 2017;Joulin et al., 2018b;Jawanpuria et al., 2019), or even in an unsupervised 2 manner based on distribution matching (Zhang et al., 2017a;Artetxe et al., 2018a;Zhou et al., 2019). Recently, it has been shown that alignment methods can also be effectively applied to contextualized word representations (Schuster et al., 2019;Aldarmaki & Diab, 2019).
Another successful line of research for CLWE considers joint training methods, which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a 1 Code will be released on publication. 2 In this paper, "supervision" refers to that provided by a parallel corpus or bilingual dictionaries.

This paper proposes a new and simple framework for learning cross-lingual embeddings that, based on well-supported insights, unifies two standard frameworks: joint learning and alignment-based approaches. While I like and acknowledge the fact that the paper examines these frameworks critically and has also some didactic value, I still have some concerns regarding the current experiments, and would like to see some additional analyses in the paper. Honestly, this type of work would work better as a short ACL/EMNLP paper, as the core methodological contribution is a very simple combination of the existing techniques.
With joint methods, it is true that shared words can be used as implicit bilingual learning signal which gets propagated further in the model. Even in the case of alignment-based methods, it was shown that this signal can assist in learning shared cross-lingual spaces, but: 1) such spaces are of lower-quality than the spaces learned by relying on seed dictionaries (see e.g., the work of Vulic and Korhonen (ACL 2016)), 2) this is useful only for similar language pairs which use the same script. The paper fails to provide an insightful discussion on how the scores differ when we move towards more distant languages. For instance, English-Chinese is included as a more distant language pair, and the combined method seems to work fine, but the reader is left wondering why that happens. The same is true for English-Russian. The paper should definitely provide more information and insights here.
One experiment that would be quite interesting imho is to take seed dictionaries to initialise the joint training method. It will not be unsupervised any more, but it would be very useful to report the differences in results between fully unsupervised joint initialisation (which, as mentioned should work only with more similar languages) and such weakly supervised init: e.g., the method could take all one-to-one translation pairs from the seed dictionary as 'shared words'. It would also be interesting to combine such 'shared words' and words that are really shared (identically spelled words) as initialisation and measure how it affects the results, also in light of differences in language distance. Adding one such experiment would make the paper more interesting.
Some recent strong baselines are missing: e.g., I wonder how the model that does self-learning on top of seed dictionaries (similar to the work of Vulic et al., EMNLP 2019) compares to the proposed method. Also, can we use the same self-learning technique with the combined method here? Would that lead to improved results? Another work I would like to see comparisons to is the work of Zhang et al. (ACL 2019) and despite the fact that the authors explicitly stated that they decided not to compare to the work of Artetxe et al. (ACL 2019) as they see it as concurrent work, I would still like to see that comparison as the model of Artetxe et al. seems very relevant to the presented work.
I do not see the extension to contextualized representations (described in Section 3.2) as a real contribution: this is a very straightforward method to apply an alignment-based method on multilingual BERT representations, and very similar techniques have been probed in previous work (e.g., by Aldarmaki & Diab), only the bilingual signal/dictionary came from parallel data instead.
Finally, as mentioned before, one of the must-have experiments is including more (distant and diverse) language pairs and analysing the results across such language pairs, with an aim to further understand how the levels of sharing, over-sharing, and non-isomorphism affect performance. Also, while the authors state that reduced isomorphism is the key problem of alignment-based methods, I still fail to see how exactly the alignment refinement step does not suffer from that problem? More discussion is needed here.
Other comments:
- It would be useful to also point to the survey paper of Ruder et al. (JAIR 2019) where the difference between alignment-based and joint models is described in more detail and could inform an interested reader beyond the confines of the related work section in this paper. Similarly, differences between joint models and alignment-based models have also been analysed by Upadhyay et al. (ACL 2016); Vulic and Korhonen (ACL 2016).
- I like how Figure 1 clearly visualizes the main intuitions behind the proposed framework, and how mitigating the oversharing problem leads to better alignments. This was very nice.
- In light of the problems with silver standard MUSE datasets (see also the recent work of Kementchedjhieva, EMNLP 2019), I would suggest to run BLI experiments on more language pairs: e.g., there are BLI datasets of Glavas et al available for 28 language pairs.


The Neural Architecture Search (NAS) problem is typically formulated as a graph search problem where the goal is to learn the optimal operations over edges in order to maximize a graph-level global objective. Due to the large architecture parameter space, efficiency is a key bottleneck preventing NAS from its practical use. In this paper, we address the issue by framing NAS as a multi-agent problem where agents control a subset of the network and coordinate to reach optimal architectures. We provide two distinct lightweight implementations, with reduced memory requirements (1/8th of state-of-the-art), and performances above those of much more computationally expensive methods. Theoretically, we demonstrate vanishing regrets of the form O( √ T ), with T being the total number of rounds. Finally, aware that random search is an (often ignored) effective baseline we perform additional experiments on 3 alternative datasets and 2 network configurations, and achieve favorable results in comparison with this baseline and other methods.

In this paper, the authors proposed MANAS, which is based on DARTS, by approximating the problem space by factorizing them into smaller spaces, which will be solved by multiple agents. The authors claimed that this can simplified the search space so that the joint search can be more efficient to enable us to search a larger space faster. While the overall idea seems simple but the coordinating among agents can be difficult, where the authors proposed credit assignment techniques to address the issue. The final algorithm is evaluated on CV datasets as well as 3 new datasets.
Overall, I found the motivation and proposed solution by the authors convincing. However, in a search space where random searcher is competitive, it is important for us to have an in-depth understanding on the proposed techniques. Especially when the experimental results is not fully comparable (it is difficult to control #params to evaluate the Test Error, and Search Cost being a one-time cost), I think the magnitude of the improvement showed in the experimental results itself might not be enough to justify this new approach.
I am curious about the how does the number of agents affects the experimental results. It seems that it is not mentioned in the experiment section (or I might missed it?). And do we need to search for the best number of agents, which will add to the search cost?
I am also curious on if one can combine other search algorithms with the similar idea on dividing the search space, e.g., using random search on a subspace in a coordinate descend fashion. It will be great if the authors can provide more in-depth analysis on different component of the proposed algorithm so that we can fully understand the source of improvement.


Recent work by Jacot et al. (2018) has showed that training a neural network of any kind with gradient descent in parameter space is equivalent to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result to show that the output of a neural network trained using full batch gradient descent can be approximated by a linear model for wide networks. In parallel, a recent line of studies , Hayou et al. (2019)) suggested that a special initialization known as the Edge of Chaos leads to good performance. In this paper, we bridge the gap between these two concepts and show the impact of the initialization and the activation function on the NTK as the network depth becomes large. We provide experiments illustrating our theoretical results.

This paper discussed the property of the NTK with the increasing depth L with the help of the Edge of Chaos initialization. The authors show that if deep neural networks are not properly initialized, the NTK can have a large condition number, which leads to the poor performance of training and generalization. Moreover, the authors also introduce the conditions that make the neural network trainable by decreasing the convergence rate to a nearly constant kernel w.r.t the depth L by using the specific Edge of Chaos initialization as well as different activations and use residual connections. Experiment results show that the theoretical results are well aligned with the practice.
Detailed Comments:
1. At the paragraph after Lemma 1, the authors claimed that f_t(x) is entirely given by f_0(x), which means a generalization error of order O(1). This is a little inaccurate I think, as NTK can only characterize the training process and does not directly indicate generalization. Also, as t to infinity, the coefficient of initialization f_0(x) can be arbitrary small and thus the f_t(x) is entirely decided by initialization is not accurate as well. To analyze generalization with NTK, we need a little more, see [1]. This is not some core issue, but I think the authors should make the description more accurate.
2. The authors should explain what richer limiting NTK means at the end of the Sec 3. It is unclear how the convergence rate of the NTK related to the richer limiting NTK.
3. A typo in the first paragraph in Sec 4, ouerselvs to ourselves.
4. In definition 1, the second-order derivative of \phi is defined via the indicator function 1_{A_i}? It’s better to use another notation like \mathbf{1} or \mathbb{1} to make it more clear.
5. I think for clarity the authors should give a formal definition of ANTK, or if it is unnecessary, better directly use the original NTK. Also, it is better to show the invertibility of the NTK is equivalent to ANTK more formally before Proposition 2. This conclusion is not so straightforward at the first view. Meanwhile, I think the invertibility of ANTK can just indicate a bad condition number of NTK? I don’t think it directly related to the invertibility of NTK.
6. There are several typos in the appendix. Please go through the appendix and fix them.
7. In the proof of Lemma 3 and Lemma 4, I don’t get why |a_l| < l + |a_0|. It may depends on the property of q? For sufficiently large q, if a_0=0, a_1 = q + O(1) which is not necessarily smaller than l=1? But to get a_l/l bounded, it is not necessary to use this. Also, I think it’s better not omit the constant in the dynamic system of Lemma 3.
8. Better use the Stirling’s approximation rather than k to infinity in the calculation of k!/(k-\alpha-1)!
9. The lemma number of Appendix C is in a chaos. And I don’t quite get the idea behinds Appendix C. Is it relevant to the contents in the main text?
10. I also feel the content in Appendix D is unnecessary for this paper.
Overall, this paper is interesting and gives a unified perspective on the recently developed NTK and Edge of Chaos initialization. It also sheds light on the impact of different activation function on NTK by generalizing the results of [2]. I feel this paper can help the communities have more understanding on the properties of deep neural networks. However, this paper can be better if the authors can polish their paper and reorganize the appendix part.
[1] Arora, Sanjeev, et al. "Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks." International Conference on Machine Learning. 2019.
[2] Hayou, S., Doucet, A. & Rousseau, J.. (2019). On the Impact of the Activation function on Deep Neural Networks Training. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:2672-2680


Successfully training deep neural networks often requires either batch normalization, appropriate weight initialization, both of which come with their own challenges. We propose an alternative, geometrically motivated method for training. Using elementary results from linear programming, we introduce Farkas layers: a method that ensures at least one neuron is active at a given layer. Focusing on residual networks with ReLU activation, we empirically demonstrate a significant improvement in training capacity in the absence of batch normalization or methods of initialization across a broad range of network sizes on benchmark datasets.

The paper introduces a new type of layer, Farkas layers, that are designed to ameliorate the "dying ReLU problem". The idea of repurposing Farkas' lemma from linear programming to a component in NNs is very nice. However, the paper doesn't have convincing baselines and doesn't dig deep enough into what the Farkas layer is actually doing.
Comments:
1. Start of section 2.2: “Previous work on the dying ReLU problem, or vanishing gradient problem, in the case of using sigmoid-like activation functions, has heavily revolved around novel weight initializations”. Dying ReLUs and vanishing gradients are different problems. In particular, it doesn’t make sense to talk about the dying ReLU problem for sigmoid-like activation functions.
2. Batchnorm (BN) consists in two operations: shifting and scaling. Both are relevant to vanishing gradients. However, only shifting (that is, subtracting the mean and resetting to a learned value) is relevant to dying ReLUs because rescaling the input to a ReLU by a positive number doesn’t affect whether the ReLU is subsequently ON or OFF.
3. Given point #2, a natural baseline to compare the Farkas layer against is a “pared-down BN”, which shifts but does not rescale.
4. Similarly, when combining the Farkas operation with BN, it might be worth considering keeping BN’s rescaling but dropping the shift -- since Farkas is analogous to the shift operation in BN.
5. I claimed above that Farkas is analogous to the shift in BN, but haven’t thought about it deeply. Do you agree? Any comments on how they differ and why?
6. Our contributions, p2: “We empirically show an approximate 20% improvement on the first epoch over only using batch normalization.” I’m not sure what to make of this; improvements on the first epoch are only useful if they lead to overall improvements.
7. Figure 4 of “Shattered gradients”, https://arxiv.org/abs/1702.08591 looks at ReLU activations by layer, both with and without BN. It’s worth doing a similar analysis for Farkas layers. Concretely: how do Farkas layers change the pattern and frequency of activation, both at activation and during training?
8. Guaranteeing the activity of a single neuron per layer seems very weak. What is the empirical effect of Farkas on the number of live neurons? Is it really just making sure one ReLU is on, or does it do better? Is it possible to ensure more neurons are ON in each layer? The “shattered” paper above suggests BN sets close to half ReLUs as ON at initialization, and approximately controls how often ReLUs are ON or OFF during training via the shift.
9. As a broader point, the paper proposes an algorithm based on a hypothesis: that having (at least one) ReLU on helps training. It’s worth digging into the hypothesis a bit rather than just plotting training curves.
10. I would expect ResNets have much *less* of a problem with dying ReLUs than standard NNs because of the skip-connections. One would therefore expect Farkas layers to help more with standard NNs than ResNets. However, the reported results are for ResNets. What happens when there are no skip-connections?


Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feedforward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

This paper proposes a simple modification to the ubiquitous Transformer model. Noticing that the feed-forward layer of a Transformer layer looks a bit like an attention over "persistent" memory vectors, the authors propose to explicitly incorporate this notion directly into the self-attention layer. This involves concatenating the contextual representations with global, learned memory vectors, which are attended over.
The model is tested on widedly-utilized character/word-level language modeling benchmarks, where it is found to outperform, or be on par with, existing models while using fewer number of parameters.
Insofar as architectural advancements can translate to general improvements across multiple NLP tasks, this paper could be seen as important. However, I am not sure that in 2019, demonstrating arguably-marginal perplexity improvements on standard datasets is enough. I would love to see if this type of layer can result in better conditional generation models (e.g. translation, summarization), or can train GPT2/BERT/XLNet-style models whose representations better transfer to other tasks.
I had some further questions/comments:
- I found the motivation of the persistent memory vector as replacing the FF-layer somewhat tenuous. Eq(5) is definitely different from Eq(9)! In my opinion this work can be better motivated/presented as just a standalone modification to the Transformer layer.
- While it is impressive that the proposed approach performs better (or on par with, in the case of character-level language modeling) than the previous state-of-the-art models which are larger, to me it is not immediately clear if the benefit is coming from the proposed modifications, or something else (e.g. some of the things mentioned in 4.3). I understand most of this is taking from prior work, but as we all know, in deep models various architectural/hyperparameter modifications can interact in unexpected ways. Therefore, at a minimum, I would like to see the performance of a comparable model with the same exact setting, except for the persistent attention layer (i.e. N = 0 but using the feedforward sublayers). (If I understand the work correctly, this baseline should have comparable number of parameters?)
- The ablation studies are good but it would be good to see them on the word-level task as well.
- What is the performance of the FF-attn baseline with the same depth? (I understand the number of parameters would be  larger, but does this model perform as well as the Dai et al. 2019 work?)
- What if you combine this with the FF sublayer as well?
- Have you tried qualitatively analyzing the attention distributions? What are some examples in which the persistent memory vectors are attended to most? What are some examples in which the attention distribution for this model differs considerly versus regular self-attention layers?
[EDIT after author rebuttal]
Thank you very much for the rebuttal. I have updated my score to reflect the latest iteration of the paper.


Learned representations of source code enable various software developer tools, e.g., to detect bugs or to predict program properties. At the core of code representations often are word embeddings of identifier names in source code, because identifiers account for the majority of source code vocabulary and convey important semantic information. Unfortunately, there currently is no generally accepted way of evaluating the quality of word embeddings of identifiers, and current evaluations are biased toward specific downstream tasks. This paper presents IdBench, the first benchmark for evaluating to what extent word embeddings of identifiers represent semantic relatedness and similarity. The benchmark is based on thousands of ratings gathered by surveying 500 software developers. We use IdBench to evaluate state-of-the-art embedding techniques proposed for natural language, an embedding technique specifically designed for source code, and lexical string distance functions, as these are often used in current developer tools. Our results show that the effectiveness of embeddings varies significantly across different embedding techniques and that the best available embeddings successfully represent semantic relatedness. On the downside, no existing embedding provides a satisfactory representation of semantic similarities, e.g., because embeddings consider identifiers with opposing meanings as similar, which may lead to fatal mistakes in downstream developer tools. IdBench provides a gold standard to guide the development of novel embeddings that address the current limitations.

The paper introduces a new dataset that includes manually labelled identifier name pairs. The labels determine how much the corresponding embeddings are useful for determining their meaning in context of code, a setting that has sufficient differences from a natural language.
A significant part of the paper is devoted on data cleaning and relating the computed metrics to similar efforts in natural language processing. While there is not much novelty in this part of the paper, it is doing a good job at addressing many possible questions on the validity of their dataset. Another important aspect that is covered by the work is different kinds of similarities of identifier names - similarity corresponding to having the same or similar type in a precise type system or similarity corresponding to being synonyms. Having several of these dimensions would make the results applicable for a wide range of applications of identifier name embeddings.
While not introducing new concepts, this paper is important for the community, because it has the potential to change the way embedding computation is done for “Big Code” problems. Right now, most papers either introduce their own embeddings, or use non-optimal ones like Code2Vec.
The paper also has a surprising finding that even techniques designed for code are in some cases not as good as the FastText embeddings. This is an interesting result, because few other works include this kind of embedding in their experiments. Furthermore, the paper deep dives into the strong and weak sides of several solutions and shows that there is a large opportunity to improve on the existing results.


Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.

This paper proposed a local VAE based on the model-agnostic meta-learning concept. Images generated based local VAE are shown to be better than those generated bu \beta-VAE in general in terms of generation quality and disentanglement/compactness/informativeness.
This paper shows an interesting idea of viewing neighbourhood of a data point as a task to adopt the meta-learning concept to  train VAE which gives superior performance.
However, this paper falls short regarding its overall organisation of the paper and the way that different concepts being articulated which make it hard to follow and read. It is not easy to understand the linkage between a task and a neighbourhood being advocated without re-reading the paper for a few times. It seems to me that very substantial effort is needed for the revision to reach the ICLR standard.
The introduction section should be carefully rewritten to explain clearly the notions of locality and structural relationship. As far as I can see in the subsequent sections, the locality/structural relationship seems to be referring to the neighbourhood either in the input space or the latent space. Also, the way of discussing local learning should be elaborated a bit to make it clear what exact notion of local learning  is being presented.
Specific comments:
Page 2:
“since the most dataset tends to be governed by the consistent rules of the physical world”
->
“since most dataset tends to be governed by the consistent rules of the physical world”
“Such kind of dataset has a multi-scale structure from a local to a global scale.” - some grammatical issue
Page 5:
Line 4: Equation 3 seems not related to meta-learning


Recent advancements in unsupervised disentangled representation learning focus on extending the variational autoencoder (VAE) with an augmented objective function to balance the trade-off between disentanglement and reconstruction. We propose Sequential Residual Variational Autoencoder (SR-VAE) that defines a "Residual learning" mechanism as the training regime instead of the augmented objective function. Our proposed solution deploys two important ideas in a single framework: (1) learning from the residual between the input data and the accumulated reconstruction of sequentially added latent variables; (2) decomposing the reconstruction into decoder output and a residual term. This formulation encourages the disentanglement in the latent space by inducing explicit dependency structure, and reduces the bottleneck of VAE by adding the residual term to facilitate reconstruction. More importantly, SR-VAE eliminates the hyperparameter tuning, a crucial step for the prior state-of-the-art performance using the objective function augmentation approach. We demonstrate both qualitatively and quantitatively that SR-VAE improves the state-of-the-art unsupervised disentangled representation learning on a variety of complex datasets.

Overview:
Authors introduce a new VAE-based method for learning disentangled representations.
The main idea is to apply a “residual learning mechanism”, which resembles an autoregressive model, but here conditioning between sequential steps is done in both latent and input spaces. Namely, for each input, the method involves making several (one per *each* embedding dimension) sequential passes, where each pass takes the residual between the input and the current accumulated output (i.e. pixel-wise difference between two images) as well as the values of the sampled embeddings so far.
Authors report quantitative and qualitative evaluation with a set of reasonable baselines (FactorVAE, beta-VAE), and the results seem to be slightly better.
Decision:
The writing quality is not great: there are frequent typos and not-so-precise formulations, and is at times hard to follow (see list below).
The method itself is not really well-motivated: there seems to be no formal justification provided, and the informal one is not very clearly explained in the paper.
Scalability of the method is clearly an issue. Many applications might require the size of embeddings to have hundreds of dimensions, which would mean that the given method cannot really be applied for non-toy problems.
With all those considerations in mind, I cannot recommend to accept this paper as is, thus the final rating, “reject”.
Additional comments / typos:
* It would be worth seeing how this approach relates to auto-regressive models.
* p2. “encoder maps … x to a latent representation q” - this sentence is not very strict.
* p2. “.. is defined as following”: as follows?
* p2. “.. in addition of b”
* p3. “.. this reguralizer encourage”
* p7: “… it reduces the solution space and improve training stability“: is there any sort of theoretical reasoning that could support this?
* The reference format is not very readable, probably better to change to (Names, Year).
<update>
Authors addressed some of my concerns (thanks!), thus I increased my rating slightly.
There is still a large concern wrt to the computational complexity. I kind of understand the argument about the hyperparameter tuning, but it seems not really fair to compare to a bruteforce parameter search (one can probably do some sort of bayesian optimization).
</update>


The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive estimation approach is not suitable to large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement-learning methods. Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations denote that VILD outperforms stateof-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.

This paper proposes an imitation learning algorithm for the setting where the demonstration data consists of trajectories from sources of varying expertise. The authors proceed by defining a parameterized model of the (demonstration) trajectory distribution (Equation 2), which uses the MaxEnt-RL model for the optimal policy, and a distribution (p_w) to model the level of expertise. Imitation learning is then reduced to maximum-likelihood training under the provided demonstrations. Using appropriate variational distributions and model specification, the MLE objective is transformed to the VILD objective (Equation 5), which can be optimized with gradient descent. Expertise-level (p_w) is modeled as a Gaussian blur over the optimal action, wherein the variance is correlated with expertise (lower is better). Furthermore, a truncated IS approach is proposed for learning a better reward function. It samples more frequently from the experts that have a higher estimated expertise.
Overall, I really enjoyed reading the paper. The writing and the presentation of material (both background and novel solutions) is clean and concise. The Appendix, with all the derivations and the summarizations of the related approaches, is very informative. I would like the authors to comment on the following:
1.	It is claimed in Section 1 that prior approaches for imperfect imitation learning rely on auxiliary information from the expert, in the form of confidence scores or ranking, while VILD doesn’t use any. In my opinion, the fact that VILD uses “labeled” expert demonstrations (i.e. each demonstration is tagged with a number {1..K}) classifies as auxiliary information. Contrary to approaches such as InfoGAIL, which infer the latent structure of the expert demonstrations in a completely unsupervised fashion, VILD fixes the demonstrations-model and instead attempts to learn the parameters corresponding to this model – this holds exactly for the Gaussian policy, and approximately for the TSD policy setting.
2.	The difference in performance of VILD w/ and w/o IS is surprising. I understand the motivation in Section 3.4 that IS should help to improve the convergence rate, but for benchmarks like HalfCheetah, Walker, the performance seems to have saturated to a significantly lower value. I would like to know if the authors have some thoughts on this wide discrepancy w/ and w/o IS.
3.	Baselines – I’m not sure if InfoGAIL with a uniform prior on the context is a fair comparison to VILD-IS. Since VILD-IS changes the demonstrator sampling from uniform to expertise-dependent, one could do something similar for InfoGAIL – e.g. after training, report the best performing context, or sample context based on a performance-dependent distribution.
4.	Sample-efficiency in terms of expert data – Is the number of trajectories that are collected from each of 10 demonstrators reported somewhere?


In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand. Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome. Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive. In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities. Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific. Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

This paper presents a method to infer multi-task networks architecture, more specifically to determine which part of the network should be shared among different tasks. The main idea is to first train a specific, standalone encoder/decoder network for each task. Subsequently, a task affinity factor is computed by looking at the similarity (or, more likely the dissimilarity) of an holdout set of images feature representations. Knowing these dissimilarities (based on RDM), one can cluster the tasks and make similar tasks share more of their layers. Computational budget can also be taken into consideration in the form of the number of parameters. Results on Cityscapes, Taskonomy, and CelebA datasets show, to some extent, improvements against the state of the art.
The paper is well written and addresses a common problem in multi-task learning. The experiments provided are extensive and cover most of the current multi-task learning methods and interesting problems. I especially like the idea of formalizing the dissimilarity between tasks using RDM. There are, however, a few key points that would need improvement.
First, except for CelebA, the experiments provided use ResNet50 with only 4 different "anchor point" in the network. In other words, the task at hand is limited to selecting the best sharing point between 4 choices. This is not wrong per se, but in my opinion, it does not tackle the main problem: what to do when brute force / exhaustive exploration cannot be fulfilled? CelebA provides a more complex case, but it also requires to change the method from an exhaustive search to a beam search (end of Sec. 3.2). Doing so get us back to a kind of greedy approach, precisely what was advocated against the paragraph before (in the discussion about Lu et al. 2017).
Second, the fact that task affinities are computed a priori leads to the following conclusion: "this allows us to determine the task clustering offline". While I agree that this could be useful, one has to keep in mind that compared to other methods, this one has to first train a network for _each_ task independently, which can take a long time.
Out of curiosity, did you consider other correlation coefficients than Spearman? Why use a rank-based method?
Overall, this is a nice paper, with adequate methodology. It is not groundbreaking, and most of the good results arise when we consider both performance and number of parameters, but it is interesting for the community nonetheless. I am not sure the impact would be very high, since it can probably be replaced by an architecture exhaustive search if the number of tasks and branching points in the network are low, but the formalism of the approach is welcomed.


We introduce the Convolutional Conditional Neural Process (CONVCNP), a new member of the Neural Process family that models translation equivariance in the data. Translation equivariance is an important inductive bias for many learning problems including time series modelling, spatial data, and images. The model embeds data sets into an infinite-dimensional function space as opposed to a finitedimensional vector space. To formalize this notion, we extend the theory of neural representations of sets to include functional representations, and demonstrate that any translation-equivariant embedding can be represented using a convolutional deep set. We evaluate CONVCNPs in several settings, demonstrating that they achieve state-of-the-art performance compared to existing NPs. We demonstrate that building in translation equivariance enables zero-shot generalization to challenging, out-of-domain tasks.

The paper introduces ConvCNP, a new member of the neural process(NP) family that models translational equivariance in the data, which uses convolutions and stationary kernels to aggregate the context data into a functional representation.
This problem is well-motivated as there are various domains where such an inductive bias is desirable, such as spatio-temporal data and images, and will help especially with predictions for out-of-distribution tasks. This inductive bias was never built into NPs, and it remained unanswered whether the NP can learn such a behaviour. This paper shows that the answer is negative and that one needs to make modifications to create such inductive bias.
The architecture of the ConvCNP is motivated by theory that completely characterises the set of translation equivariant functions Phi that maps sets of (x,y) pairs to bounded continuous functions that map x to y (disclaimer: I haven’t read through the proof in the appendix, so will not make any claims on its correctness). Theorem 1 defines the set of such functions using rho, phi and psi, and the choices for each on on-the-grid data and off-the-grid data are listed in Section 4. There are ablation studies in Appendix D.4 that justify the choices.
Overall the paper is very well-written and clear for the most part, with helpful pseudo-code and well-laid out quantitative + qualitative results, and a very detailed appendix that allows replicating the setup. The evaluation is extensive, and the results are significant.
    - The results on 1D synthetic data show a noticeable improvement of the ConvCNP compared to the AttnCNP, with improved interpolation as well as accurate extrapolation for the weakly periodic function. I do think however that a more competitive baseline for AttnCNP would have been to parameterise the logits of the attention weights as a periodic function with learnable length scale (e.g. stationary periodic kernel), since this is another way of building in periodicity into the model. Arguably this is more explicit and restrictive than the translational equivariance built into ConvCNP, but would have made for a more interesting comparison.
    - Having said that, I like how the evaluation was performed on a variety of stochastic processes - previous literature only used GP + EQ kernel, but here more challenging non-smooth functions such as GP + Matern kernels and sawtooth functions are explored - and it’s very convincing to see the outstanding performance of ConvCNPs here.
    - It’s also nice to see results on regression tasks on real data (sections 5.2, 5.3), which was never explored in the NP literature as far as I know. 5.2 shows that ConvCNPs can be competitive against other methods that model stochastic processes, and 5.3 shows an instance of where ConvCNPs do a reasonable job whereas (Attn)CNP fails.
    - The results on images is also extensive, covering 6 different datasets (including the 2 zero shot tasks), and show convincing qualitative and quantitative results. The zero shot tasks are nice examples that explicitly show the consequences of not being able to model translation equivariance in more realistic images composed of multiple objects/faces.
I have several comments/questions regarding the disccusion & related work section:
    - One link that might be worth pointing out regarding functional representation of context is that ANP (or AttenCNP) can also be seen as giving a functional representations of the context; the ANP computes a target-specific representation of the context, which can be seen as a function of the target inputs.
    - I think it’s incorrect to say that latent-variable extensions enforce consistency. Even with the latent variable, if the encoder is seen as part of the model, then the NP isn’t consistent (pointed out in the last paragraph of section 2.1 in the ANP paper). So there still are issues regarding AR sampling. There does however seem to exist variants of NPs that satisfy consistency e.g. https://arxiv.org/abs/1906.08324
    - What is preventing the incorporation of a latent variable in the ConvCNP? Is this just something that can be easily done but you haven’t tried, or do you see any non-trivial issues that arise when doing so e.g. maintaining translation equivariance?
Other minor comments:
    - Are there any guidelines on choice of filter size of CNN in the image case? E.g. have you chosen the filter size of ConvCNP such that the receptive field is smaller than the image, whereas it’s bigger for ConvCNPXL? It’s not clear why having a bigger receptive field allows to capture non-stationarity, and it would be helpful to expand on that, perhaps in the appendix.
    - Also it’d help for the sake of clarity to explain why AttnCNP uses significantly more memory than ConvCNP, i.e. because memory for self-attention is O(N^2) where N=HW is the number of inputs, whereas for convolutions it’s O(HW).
    - I think it’d also help to state explicitly in the body that AttnCNP is ANP without the latent path when it is introduced.
    - typos: first paragraph of Section 2: Z_M <- Z_m (twice), finitely <- infinitely, Appendix D.1: separabe <- separable
Overall, I think this is a very strong submission and I vote for its acceptance.