We address the challenging problem of deep representation learning-the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the model parameters with respect to a task-specific loss given an input sample. Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network. We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insights. Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients. Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures. Strong results are obtained in all settings, and are well-aligned with our theoretical insights.

Summary: This paper proposes to use the gradients of specific layers of convolutional networks as features in a linearized model for transfer learning and fast adaptation. The method is theoretically backed by an appeal to the recently proposed neural tangent kernel and seems like it could be practically useful.
Edit: Post rebuttal, I am somewhat satisfied by the authors' response and find their ablation study somewhat compelling hence am increasing my score to a weak accept. However, I'm not completely convinced that their choice of layers to use as gradient features is reasonable. If this paper does end up getting accepted, I'd very strongly advocate for being extremely clear as to
1) why the gradient features used are used,
2) a good choice of gradient features in practice
3) inclusion of all of the relevant ablation studies in the final version.
I tend to weakly reject this paper currently despite liking the simplicitly and timeliness of the approach. Specifically, I would like further discussion of the choice of the layers to use as gradient features and an ablation study on supervised trained networks.
Originality: I believe that this is one of the first papers to explicitly use the Taylor approximation (neural tangent kernel) in a transfer learning setting, making the approach timely and potentially practically useful.
Significance: The approach is a quite nice merging of theoretical insights with a neat practical implementation. Although the main methodological advance is a straightforward application of the thinking behind Jacobian vector products, the method is well described and ought to be practically useful. However, I have a bit of a concern as to its practical necessity in comparison to simple fine-tuning of the final softmax layer (referred to in Table 1 as \theta_2) on a new dataset.
Clarity: While the approach described in Section 3 is quite generic – theoretically, the method should simply consist of training the final layer of the neural network (w_1) and weights from the Jacobian features around the Taylor expansion. However, the experimental approaches suggest that different layers were used in each experiment – see e.g. “[we] … compute our gradient based features from one, two, or all of the top-3 conv layers” (Section 4.2 for the BiGAN architectures) versus “our gradient based features are from the last 2 conv layers” (Table 2 for the AlexNet architectures). Why was the entire network (modulo the last feed forward layer) not simply used for the Jacobian features?
Similarly, why was the entirety of the model (again modulo the last feedforward layer) not used for forwards model in the AlexNet experiments?
Based on the ablation study in Section 4.1, the authors find that “it suffices to set the very top layer as \theta_2 to enjoy a reasonably large performance gain.” When this is the case, it is a bit tough to distinguish the approach from standard final layer transferring approaches (Sharif Razavian et al, 2014) and indeed Bayesian final layer approaches that simply retrain an output layer (e.g. Perrone et al, 2018). Indeed this seems to be the case in Table 1, if that is so, then why not just re-train \theta_2 as well throughout the experiments as it will typically just be another stochastic convex optimization problem?
Could the authors quickly describe an implementation of the scalable Jacobian inner products in Section 3.3? It seems like outputs will have to be cached during the forwards passes, thereby requiring a somewhat significant amount of software engineering (and memory overhead) to be have to do this process for each new layer. Is this the correct understanding of how one would implement the procedure? A quick paste of PyTorch pseudo-code would be sufficient here.
Quality: The experiments seem to be relatively convincing – it seems exciting that a linear model + the network’s features itself can typically perform as well as fine-tuning and occasionally even better than fine-tuning itself.
However, I am a bit concerned by the fact that the ablation studies themselves only utilize models trained in an un-supervised fashion.
I’d instead like the authors to run an ablation study on supervised trained models (perhaps 8 layer conv nets or VGG16) in the same manner as in Table 1 and Section 4.1. Specifically, I’d like to see this done to see whether the gradients in fact are as interesting as features when they have been trained in a supervised fashion.
Similarly, I’d like to see the features themselves used as a linear classifier (no network forwards passed) in the same two ablation studies. That is, could the authors use w_1^T J w_2 as the features for their linear classifier. If they have already done so and I’ve missed that in the tables somehow, I apologize. This experiment should help to test out how _useful_ the features defined by the Jacobian matrix are in comparison to the network’s forward pass itself.
Minor Comments:
Introduction: “After learning, the the (sic) activation of the deep network are considered as generic features.” Not only is there a small typo, but you should either include a citation here or be more specific as to what “the activation” of the deep network is here.
Introduction: “And the accuracy of the rthe (sic) …” typo + please do not start sentences with and if at all possible.
Section 3.1 (beneath eq. 2): “liner” should be linear.
Table 3: “Self-supervise” should be “Self-supervised.”
References:
Perrone et al, Scalable Hyperparameter Transfer Learning, NeurIPS, 2018. http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning.pdf
Sharif Razavian et al, CNN Features off-the-Shelf: an Astounding Baseline for Image Recognition, CVPRW, 2014. https://arxiv.org/abs/1403.6382


Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structure of proteins and other macromolecular complexes at near-atomic resolution. In single particle cryo-EM, the central problem is to reconstruct the 3D structure of a macromolecule from 10 4−7 noisy and randomly oriented 2D projection images. However, the imaged protein complexes may exhibit structural variability, which complicates reconstruction and is typically addressed using discrete clustering approaches that fail to capture the full range of protein dynamics. Here, we introduce a novel method for cryo-EM reconstruction that extends naturally to modeling continuous generative factors of structural heterogeneity. This method encodes structures in Fourier space using coordinate-based deep neural networks, and trains these networks from unlabeled 2D cryo-EM images by combining exact inference over image orientation with variational inference for structural heterogeneity. We demonstrate that the proposed method, termed cryoDRGN, can perform ab initio reconstruction of 3D protein complexes from simulated and real 2D cryo-EM image data. To our knowledge, cryoDRGN is the first neural networkbased approach for cryo-EM reconstruction and the first end-to-end method for directly reconstructing continuous ensembles of protein structures from cryo-EM images.

~The authors build a new method to recapitulate the 3D structure of a biomolecule from cryo-EM images that allows for flexibility in the reconstructed volume.~
I thought this paper is very well written and tackles a difficult project.
There is a previous work that these authors should cite:
Ullrich, K., Berg, R.V.D., Brubaker, M., Fleet, D. and Welling, M., 2019. Differentiable probabilistic models of scientific imaging with the Fourier slice theorem. arXiv preprint arXiv:1906.07582.
How does your method compare to this paper? In Ullrich et al., they report “Time until convergence, MSE [10^-3/voxel], and Resolution [Angstrom]). I think these statistics would be useful to report in your work, as they are more familiar with folks in the cryoEM field.
In Equation 3, how does one calculate Z, the normalization constant?
For the decoder, how large of the 3D space are you generating? What are the units? Are you using voxels to represent atomic density? What is the voxel size? Is it the same as on Page 11?
I think more description of the neural network architecture would be useful (more than what is reported on page 12).


CDIL_ICLR_2020 (10).pdf
We study the question of how to imitate tasks across domains with discrepancies such as embodiment and viewpoint mismatch. Many prior works require paired, aligned demonstrations and an additional RL procedure for the task. However, paired, aligned demonstrations are seldom obtainable and RL procedures are expensive. In this work, we formalize the Cross Domain Imitation Learning (CDIL) problem, which encompasses imitation learning in the presence of viewpoint and embodiment mismatch. Informally, CDIL is the process of learning how to perform a task optimally, given demonstrations of the task in a distinct domain. We propose a two step approach to CDIL: alignment followed by adaptation. In the alignment step we execute a novel unsupervised MDP alignment algorithm, Generative Adversarial MDP Alignment (GAMA), to learn state and action correspondences from unpaired, unaligned demonstrations. In the adaptation step we leverage the correspondences to zero-shot imitate tasks across domains. To describe when CDIL is feasible via alignment and adaptation, we introduce a theory of MDP alignability. We experimentally evaluate GAMA against baselines in both embodiment and viewpoint mismatch scenarios where aligned demonstrations don't exist and show the effectiveness of our approach.

Summary
This paper proposes a method to perform alignment between demonstrations in different domains in order to adapt a policy from expert domain to the test domain. They also include a formalism that studies alignments of different MDPs.
Strengths
1) The paper presents a formal framework to study many recent methods in cross-domain/cross-view imitation learning with a common lens.
2) The solution proposed by the authors might have a big practical advantage for cross-domain imitation learning settings. They claim that they do not need to learn the policy using RL in the new domain as they learn the action mapping between the two domains to retrieve the corresponding action in new domain based on the action in expert domain. This is a very interesting result.
3) The solution is quite modular with 3 important parts: i) state mapping network, ii) action mapping network, iii) dynamics model.
Weakness
The biggest weakness of the paper is in the evaluation section.
1) Issues with baselines:
i) There seems to be some problem with UMA. In Figure 4, it cannot figure out the alignment with self domain. Why is that the case? Where is the cartpole in UMA row for pen-cartpole alignment ? Given such poor performance during alignment is it fair to compare to UMA in the subsequent section?
ii) Why are the other methods not able to align well when there is no domain shift?
iii) How is dynamic time warping actually used to calculate the state correspondences for training IfO and IF? Is it calculated on top of the states?
iv) How are the policies trained for the transferability task? Which RL algorithm? "Baselines fail to learn the writing task as an inaccurate proxy reward function harms performance."
2 aspects are involved here:
1) initializing from alignment network
2) using rewards from the state representation to train the policy.
Do the algorithms still fail if the true reward is used with the alignment initialization?
Reward scaling is known to be important for RL algorithms[1]. May be features from these algorithms have to be scaled to scale the rewards for the RL algorithms to work.
2) Issues with environments:
i) The environments considered for domain transfer seem quite trivial for the alignment between them to be a big problem. For example, the authors present the task R2W: reacher2-tp that has a "third person" state space with a 180 camera angle offset. I am not sure how this rotation changes the state for the policy that uses states as input. One of the states of reacher is vector between goal and state which should still be informative enough to solve the task. Presumably the rotation creates a different view for the image based experiments. There also the transformation is quite easy for the spatial autoencoder given that it has access to coordinates through the spatial softmax layer.
ii) It is unclear what the R2W task entails or how difficult it is.
"The transfer task is writing letters as fast as possible. The transfer task differs from the alignment tasks in two key aspects: the end effector must draw a straight line from a letter’s vertex to vertex and minimally slow down at the vertices. "
It seems there is a term that penalizes the policy from doing things quickly. Some questions regarding this task:
1) How is this task specified?
2) Is agent rewarded for slowing down too?  Does the goal of the reacher and the state representation corresponding to the vector between end effector and goal get updated once it reached a particular point? Does the agent have to write one particular letter? Why do the other baselines not work at all on this task?
iii) Is it possible to run experiments on environments presented in Invariant Features paper so that the importance of CDIL can be assessed better.
4) "HIGH-LEVEL COMPARISON TO BASELINES" section is misleading. While the different baselines (TCN, IfO, TPIL) were developed in the context of view mismatch that does not mean the algorithm cannot be applied for embodiment mismatch. For IF also the same idea can be applied for viewpoint mismatch. TCN also proposed single-view version which can be used to learn representations without paired data. Hence, it is not right to say TCN can't work with unpaired alignment data. It also needs to be stated in this section that CDIL requires access to actions while some baselines like TCN do not need actions that can be used for training.
Questions
1) How were the alignment videos generated?
Decision
This paper has the potential to be an important paper in this field. But at this point needs further empirical evaluation with stronger baselines and known benchmark environments.
Minor Comments:
1)  "boltmzman machine reconstruction error" - Boltzmann

References
[1] "Deep Reinforcement Learning That Matters." Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger


Stochastic neural networks with discrete random variables are an important class of models for their expressiveness and interpretability. Since direct differentiation and backpropagation is not possible, Monte Carlo gradient estimation techniques have been widely employed for training such models. Efficient stochastic gradient estimators, such Straight-Through and Gumbel-Softmax, work well for shallow models with one or two stochastic layers. Their performance, however, suffers with increasing model complexity. In this work we focus on stochastic networks with multiple layers of Boolean latent variables. To analyze such networks, we employ the framework of harmonic analysis for Boolean functions. We use it to derive an analytic formulation for the source of bias in the biased Straight-Through estimator. Based on the analysis we propose FouST, a simple gradient estimation algorithm that relies on three simple bias reduction steps. Extensive experiments show that FouST performs favorably compared to state-of-the-art biased estimators, while being much faster than unbiased ones. To the best of our knowledge FouST is the first gradient estimator to train very deep stochastic neural networks, with up to 80 deterministic and 11 stochastic layers in our experiments.

***Score updated to weak accept after the rebuttal.***
Straight-Through is a popular, yet not theoretically well-understood, biased gradient estimator for Bernoulli random variables. The low variance of this estimator makes it a highly useful tool for training large-scale models with binary latents. However, the bias of this estimator may cause divergence in training, which is a significant practical issue. The paper develops a Fourier analysis of the Straight-Through estimator and provides an expression for the bias of the estimator in terms of the Fourier coefficients of the considered function. Motivated by this expression, the paper proposes two modifications of Straight-Through which may reduce the bias of the estimator, at the cost of the variance. The experimental results show advantage of this improved estimator over Gumbel-Softmax and DARN estimator.
While I really like the premise of the paper, I feel that it needs a significant amount of additional work. The text is currently fairly hard to read. The theoretical part of the paper does not quantify the variance of the estimator. The experiments are a bit unfinished and do not include ablations of the proposed modifications of Straight-Through. Most importantly, I think that in the current form the theoretical and the empirical parts of the papers are not well-connected. Because of this, I believe that the paper should currently be rejected, but I encourage the authors to continue this line of work.
Pros:
1. Theoretical analysis and empirical improvement of the Straight-Through estimator is an important avenue of work.
2. The paper makes a solid contribution of deriving the Fourier expansion of the Straight-Through estimator bias.
3. Based on this expansion, the paper proposes an algorithm with reduced bias. The algorithm is simple to implement, practical and appears to work slightly better than DARN.
Cons:
1. The key weakness of the theoretical part of the paper is that it focuses on the bias of the estimator, but does not quantify the variance, especially after the modifications. If reducing the bias was the only goal, one could use unbiased (but high-variance) estimators such as REINFORCE or VIMCO.
2. The final algorithm appears to be the DARN estimator combined with relaxation by uniform noise (“Bernoulli splitting uniform”) and scaling. The paper does not have an ablation showing how the uniform noise and scaling perform on their own.
3. There are a few incorrect statements that I’ve noticed.
* “As a side contribution, we show that the gradient estimator employed with DARN (Gregor et al., 2013), originally proposed for autoregressive models, is a strong baseline for gradient estimation.” - MuProp paper compared to this estimator under the name 1/2-estimator
* In Lemma 1 the “REINFORCE gradient” is just the exact gradient of the expectation, not a stochastic REINFORCE gradient.
* “ To the best of our knowledge, FouST is the first gradient estimate algorithm that can train very deep stochastic neural networks with Boolean latent variables.” This paper uses up to 11 latent variable layers, while [1] has trained models with >20 latent variable layers (although their “layers” have just one unit).
4. The derivation of “Bernoulli splitting uniform” trick is confusing and contains a lot of typos. For instance, the text before eqn. (14) implies that the distribution of u_i is U[-1, 1], which cannot be right and does not correspond to Algorithm 1. The statement that this trick does not lead to a relaxation is odd, since the function is being evaluated at non-discrete points.
5. There are generally many typos and some poor formatting in the math. For example, in eqn. (6) the coefficients are off by one: it should be c0 + c1 z1 + c2 z2^2 + … . The equations (10) and (11) are poorly formatted. The notation \partial_z1 f(u_1, u_2) in eqn. (14) is strange. In many places p^{i->½} is denoted as p^{1->½}.
5. I don’t think I understood the idea of representation scaling (Section 4.4). The eqn. (16) would suggest that the scaling should optimally be set to zero, which is just saying that the gradient is unbiased when the model does not use the latents. There is no other practical guidance on choosing this coefficient. Furthermore, one can always absorb the global scaling factor into the succeeding weights layer of the model, so this trick can probably be replaced by a modification of the weights initialization.
6. The experiments are missing a comparison to the Straight-Through Gumbel-Softmax estimator, introduced in the original Gumbel-Softmax paper. This is a popular biased estimator for Bernoulli latents, e.g. used in [1] [2]. Another interesting comparison would be [3] which proposes a lower-bias version of Gumbel-Softmax.
7. Figure 2 is missing the line for REBAR, even though this line is referred to on Page 8. Figure 2 and Figure 4 are both labeled as training ELBOs, despite the plots being different.
[1] Andreas Veit, Serge Belongie “Convolutional Networks with Adaptive Inference Graphs” ECCV 2018
[2] Patrick Chen, Si Si, Sanjiv Kumar, Yang Li, Cho-Jui Hsieh “Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks” ICLR 2019 https://openreview.net/forum?id=ByeMB3Act7
[3] Evgeny Andriyash, Arash Vahdat, Bill Macready “Improved Gradient-Based Optimization Over Discrete Distributions” https://arxiv.org/abs/1810.00116


We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain perturbations to the features. For example, the performance of a resume screening system should be invariant under changes to the name of the applicant. We formalize this intuitive notion of fairness by connecting it to the original notion of individual fairness put forth by Dwork et al and show that the proposed approach achieves this notion of fairness. We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.

This paper proposes a new definition of algorithmic fairness that is based on the idea of individual fairness. They then present an algorithm that will provably find an ML model that satisfies the fairness constraint (if such a model exists in the search space). One needed ingredient for the fairness constraint is a distance function (or "metric") in the input space that captures the fact that some features should be irrelevant to the classification task. That is, under this distance function, input that differ only in sensitive attributes like race or gender should be close-by. The idea of the fairness constraint is that by perturbing the inputs (while keeping them close with respect to the distance function), the loss of the model cannot be significantly increased. Thus, this fairness constraint is very much related to robustness.
---
Overall, I like the basic idea of the paper but I found the presentation lacking.
I do think their idea for a fairness constraint is very interesting, but it gets too bogged down in the details of the mathematical theory. They mention Dwork et al. at the beginning but don't really compare it to their idea in detail, even though I think there would be a lot of interesting things to say about this. For example, the definition by Dwork et al. seems to imply that some labels in the training set might be incorrect, whereas the definition in this paper does not seem to imply that (which I think is a good thing).
The main problem in section 2 is that the choice of distance function is barely discussed although that's what's most important to make the result fair. For all the mathematical rigor in section 2, the paragraph that is arguing that the defined constraint encourages fairness is somewhat weak. Here a comparison to other fairness definitions and an in-depth discussion of the distance function would help.
(In general I felt that this part was more trying to impress the reader than trying to explain, but I will try to not hold it against this paper.)
As it is, I feel the paper cannot be completely understood without reading the appendix.
There is also this sentence at the bottom of page 5: "A small gap implies the investigator cannot significantly increase the loss by moving samples from $P_*$ to comparable samples." This should have been at the beginning of section 2 in order to motivate the derivation.
In the experiments, I'm not sure how useful the result of the word embedding experiment really is. Either someone is interested in the sentiment associated with names, in which case your method renders the predicted sentiments useless or someone is not interested in the sentiment associated with names and your method doesn't even have any effect.
Final point: while I like the idea of the balanced TPR, I think the name is a bit misleading because, for example, in the binary case it is the average of the TPR and the TNR. Did you invent this terminology? If so, might I suggest another name like balanced accuracy?
I would change the score (upwards) if the following things are addressed:
- make it easier to understand the main point of the paper
- make more of a comparison to Dwork et al. or other fairness definitions
- fix the following minor mistakes
Minor comments:
- page 2, beginning of section 2: you use the word "regulator" here once but everywhere else you use "investigator"
- equation 2.1: as far as I can tell $M$ is not defined anywhere; you might mean $\Delta (\mathcal{Z})$
- page 3, sentence before Eq 2.3: what does the $\#$ symbol mean?
- page 3, sentence before Eq 2.3: what is $T$? is it $T_\lambda$?
- Algorithm 2: what is the difference between $\lambda^*_t$ and $\hat{\lambda}_t$?
- page 7: you used a backslash between "90%" and "10%" and "train" and "test". That would traditionally be a normal slash.
- in appendix B: the explanation for what $P_{ran(A)}$ means should be closer to the first usage
- in the references, you list one paper twice (the one by Zhang et al.)
EDIT: changed the score after looking at the revised version


In this paper, we consider the problem of learning control policies that optimize a reward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm -Projection Based Constrained Policy Optimization (PCPO), an iterative method for optimizing policies in a twostep process -the first step performs an unconstrained update while the second step reconciles the constraint violation by projection the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on reward improvement, as well as an upper bound on constraint violation for each policy update. We further characterize the convergence of PCPO with projection based on two different metrics -L 2 norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that our algorithm achieves superior performance, averaging more than 3.5 times less constraint violation and around 15% higher reward compared to state-of-the-art methods.

Summary :
This paper introduces a constrained policy optimization algorithm by introducing a two-step optimization process, where policies that do not satisfy the constraint can be projected back into the constraint set. The proposed PCPO algorithm is theoretically analyzed to provide an upper bound on the constraint violation. The proposed constrained policy optimization algorithm is shown to be useful on a range of control tasks, satisfying constraints or avoiding constraint violation significantly better than the compared baselines.
Comments and Questions :
	- The key idea is to propose an approach to avoid constraint violation in a constrained policy gradient method, where the constraint violation is avoided by first projecting the policy into the constrained set and then choosing a policy from within this set that is guaranteed to satisfy constraints.
	- Existing TRPO method already proposes a constrained optimization method (equation 2 as discussed), where the constraint is within the policy changes. This paper further introduces additional constraints (in the form of expected cost or safety measures) where the intermediary policy from TRPO is further projected into a constraint set and the overall policy improvement is based on the constraint satisfied between the intermediary policy and the improved policy. In other words, there are two levels of constraint satisfaction that is required now for the overall PCPO update.
	- The authors propose two separate distance measures for the secondary constraint update, based on the L2 and KL divergence.
	- I am not sure of the significance of theorem 3.2 in comparison of theorem 3.1? Proof of theorem 3.1 is easy to follow from the appendix, and as noted, follows from Achiam et al., 2017
	- Section 4 discusses similar approximations required as in TRPO for approximating the KL divergence constraint. Similar approximations are requires as in TRPO, with an additional second-order approximation for the KL projection step. The reward improvement step follows similar approximations as in TRPO, and the projection step requires Hessian approximations considering the KL divergence approximation.
	- This seems to be the main bottleneck of the approach? The fact it requires two approximations, mainly for the projection step seems to add further complexity to the propoed approach? The trade-off therefore is to what extent this approximation is required for safe policy improvement in a constrained PO problem versus computational efficiency?
	- Two baselines are mainly used for comparison of results, mainly CPO and PDO, both of which are constrained policy optimization approaches. The experimental results section requires more clarity and ease of presentation, as it is a bit difficult to follow what the results are trying to show exactly. However, the main conclusion is that PCPO significantly satisfies constraints in all the propoed benchmarks compared to the baselines. The authors compare to the sota baselines too for evaluating the significance of their approach.
Overall, I think the paper has useful merits - although it seems to be a computational ly challenging approach requiring second order approximations for both the KL terms (reward improvement and project step). It may be useful to see if there is a computationally simpler, PPO form of the variant that can be introduced for this proposed approach. I think it is useful to introduce such policy optimization methods satisfying constraints - and the authors in this work propose a simple approach based on projecting the policies into the constraint set, and solving the overall problem with convex optimization tools.  Experimental results are also evaluated with standard baselines, demonstrating the significance of the approach.


There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of 'ease-of-use' by defining an optimizer's tunability: How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical evaluation protocol for optimizer tunability that can form the basis for a fair optimizer benchmark. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

This paper introduces a simple measure of tunability that allows to compare optimizers under varying resource constraints. The tunability of the optimizer is a weighted sum of best performance at a given budget. The authors found that in a setting with low budget for hyperparameter tuning, tuning only Adam optimizer’s learning rate is likely to be a very good choice; it doesn’t guarantee the best possible performance, but it is evidently the easiest to find well-performing hyperparameter configurations.
Comments:
The paper is easy to follow. The motivation of defining tunability of optimizer is a very interesting question, however, the study seems to preliminary and the conclusion is not quite convencing due to several reasons:
In section 3.2, to characterize its difficulties of finding best hyperparameters or tunability, the authors seem to try to connect the concept of “sharpness” of a minima in loss surface to the tunability of an optimizer, which is similar to comparing the loss landscape of minimums. However, while the authors made intuitive explanation about the tunability in section 2.2, I did not see the actual plot of the true hyperpaparameter loss surface of each optimizer to verify these intuitions. Can the author be more specific about the x-axis in the illustration 1.a and 1.b? If I understand correctly, they are not the number of trails.
In addition, the proposed stability metric seems not quite related with the above intuitions, as the illustrations (1.a and 1b) define the tunability to be the flatness of hyperparameter space around the best configurations, but the proposed definition is a weighted sum of the incumbents in terms of the HPO budgets.
The definition of the tuning budgets is not clear, is it the number of trials or the time/computation budgets? The authors seems interchangeably using “runs” and “iterations”, which makes the concept more confusable.
The authors further proposed three weighting schemes to emphasize the tunability of different stage of HPO. My concern is that is highly dependent  on the order of hyperparameter searched, which could impact the tunability significantly. For instance, in case of grid search HPO and 0.1 is the best learning rate, different search order such as [10, 1, 0.01, 0.1] and [0.1, 0.01, 1, 10] could results in dramatic different CPE and CPL.
My major concern is the hyperparameter distributions for each optimizer highly requires prior knowledge. A good prior of one optimizer could significantly affect the HPO cost or increase the tunability, i.e., the better understanding the optimizer, the less tuning cost. My major concern is that the authors assume the hyperparameters to be independent (section 3.2), which is not necessarily true. Actually hyperparameters are highly correlated, such as momentum, batch size and learning rate are correlated in terms of effective learning rate [1,2], so as weight decay and learning rate are [3], which means using non-zero momentum is equivalent to using large learning rate as long as the effective learning rate is the same. This could significantly increase the tunability of SGDM. Another concurrent submission [4] verified this equivalence and showed one can also just tune learning rate for SGDM. The assumption of independent hyperparameters might be fine for black box optimization or with the assumption that practitioners have no knowledge of the importance of each hyperparameter, then the tunability of the optimizer could be different based on the prior knowledge of hyperparameter and their correlations. But it is not rigorous enough to make the conclusion that Adam is easier to tune than SGD.
The author states their method to determine the priors by training each task specified in the DEEPOBS with a large number of hyperparameter samplings and retain the hyperparameters which resulted in performance within 20% of the best performance obtained. Could the authors be more specific on the hyperparameters searched? Is this process counted in the tunability measurement?
[1] Smith and Le, A Bayesian Perspective on Generalization and Stochastic Gradient Descent, https://arxiv.org/abs/1710.06451
[2] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489
[3] van Laarhoven et al, L2 Regularization versus Batch and Weight Normalization, https://arxiv.org/abs/1706.05350
[4] Rethinking the Hyperparameters for Fine-tuning
 https://openreview.net/forum?id=B1g8VkHFPH


The spatial convolution layer which is widely used in the Graph Neural Networks (GNNs) aggregates the feature vector of each node with the feature vectors of its neighboring nodes. The GNN is not aware of the locations of the nodes in the global structure of the graph and when the local structures corresponding to different nodes are similar to each other, the convolution layer maps all those nodes to similar or same feature vectors in the continuous feature space. Therefore, the GNN cannot distinguish two graphs if their difference is not in their local structures. In addition, when the nodes are not labeled/attributed the convolution layers can fail to distinguish even different local structures. In this paper, we propose an effective solution to address this problem of the GNNs. The proposed approach leverages a spatial representation of the graph which makes the neural network aware of the differences between the nodes and also their locations in the graph. The spatial representation which is equivalent to a point-cloud representation of the graph is obtained by a graph embedding method. Using the proposed approach, the local feature extractor of the GNN distinguishes similar local structures in different locations of the graph and the GNN infers the topological structure of the graph from the spatial distribution of the locally extracted feature vectors. Moreover, the spatial representation is utilized to simplify the graph down-sampling problem. A new graph pooling method is proposed and it is shown that the proposed pooling method achieves competitive or better results in comparison with the state-of-the-art methods.

============ After rebuttal ============
I thank the authors for carefully discussing the points of my review. I have upgraded the score to marginal acceptance (6).
============ Original review ============
In this paper, the authors identify a shortcoming of existing GNN architectures for graph classification tasks -- specifically, the fact that, in the featureless regime, the graph convolutional layers rely on propagating very rudimentary structural information, making it hard (or impossible) to distinguish graphs with similar local structure. To fix the problem, the authors propose to augment the input feature space with graph-structural embeddings (computed by an algorithm like DeepWalk), and processing those in parallel with any other input features available.
On existing real-world datasets, as well as synthetic datasets carefully constructed to illustrate this phenomenon, the proposed pipeline matches or exceeds the version without the structural embedding inputs. Further, the authors note that the structural embeddings could be used to propose a novel graph pooling method -- one which attempts to preserve as diverse structural feature sets as possible. It is shown that this method is competitive to other differentiable pooling methods, like DiffPool and Graph U-Nets. Lastly, the authors demonstrate that the addition of pooling layers does not help baseline GNNs on the synthetically constructed tasks, as the fundamental issue of handling similar local structures is still not addressed.
I believe that the paper clearly exposes and proposes a nice idea which could hold great potential, and which can be useful to graph representation learning practitioners. I am particularly happy with the design of the synthetic experiments. However, I find that, in its current form, the manuscript is narrowly below the bar for a venue like ICLR.
Comments:
* The observation that existing GNN layers may struggle with distinguishing featureless graphs is not particularly novel. It's largely the centerpiece of the (already cited) work of Xu et al. (ICLR 2019), and I believe that its relevance and relation to the authors' work should be better stressed in the related work section.
* The usage of DeepWalk to encode structural information (and even to be used as initial features for a GNN) is, ultimately, also not necessarily a novel idea. At least, it's something that should be clear to any expert GNN practitioner already: if useful features are missing from the graph, a method like DeepWalk (if applicable; see below!) could be a go-to method for obtaining such features. In its current form, I don't see that the authors are proposing anything substantially architecturally novel, and their contribution is primarily on the data/feature engineering side.
* The above point is not necessarily problematic, but if the aim is to stress the importance of the architectural novelties of the proposed GNN-ESR model more, and not just the added features, I would recommend the authors to perform a few ablations: e.g. seeing how well would processing a concatenation of E and F in the same GNN layer perform.
* Many of the standard graph classification datasets are known to be noisy and unreliable (see e.g. Luzhnica, Day and Liò (ICML GraphReasoning Workshop 2019). This means that it is a must to report error bars of the cross-validation experiments. It's hard to say that many of the improvements depicted here are statistically significant otherwise.
* I have concerns about the computational complexity, or even feasibility, of using DeepWalk-like methods in the general case, e.g. for node classification. Namely, if such layers are to be applied in inductive settings (with nodes gradually added to graphs), one would require re-running DeepWalk every time a new node is added. The authors should comment on this adequately, and perhaps discuss the feasibility of other unsupervised embedding techniques for obtaining the e-vectors -- such as VGAE (Kipf and Welling, NIPS BDL 2016), GraphSAGE (Hamilton et al., NIPS 2017), Graph2Gauss (Bojchevski and Günnemann, ICLR 2018) or DGI (Veličković et al., ICLR 2019).
* While I find the proposed pooling method interesting (and more grounded in the graph's structural features than other proposed works), I find that there are many potential limitations to be discussed. For example, the fact that we start from a random first index means that we cannot rely on the obtained pooling to always be the same -- could this cause undesirable variance at test time? Furthermore, the downsampling from A^3 is a sure-fire way to obtain dense graphs after the first pooling -- potentially severely limiting the applicability of the method for large graphs. In my opinion, the authors should appropriately comment on these and perform ablations against pooling with A and A^2 (as was done in Graph U-Nets). It should also be interesting to note that there exist other structurally-informed pooling methods; see e.g. the Clique pooling method from Luzhnica, Day and Liò (ICLR RLGM 2019).
Given all of the above, I recommend (marginal) rejection, but am open to improving my score if the authors appropriately address the aforementioned comments.
Some minor comments and thoughts:
* The paper has several typos and grammar issues, and a typo pass is highly encouraged to aid clarity;
* The "attention mechanism" of Equation (5--6) seems to be nonparametric? If so, it might be interesting to compare with a version that features learnable queries, e.g. using the Transformer-style attention.
* In Equation (3), should the first min actually be a max? As we're maximising the overall minimum distances between topological features.
* I'm not sure that the paper is anywhere clear on what's the exact GNN layer being used. Could this be clarified and made more explicit? It's critical to reproducibility.
* I find it curious that the authors needed to resort to using batch normalisation---I usually found it to either have no meaningful effect on the results on the graph classification benchmarks, or it made results worse. Can the authors comment on this decision?
* The idea to concatenate output of all convolutional layers is heavily resembling Jumping Knowledge networks (Xu et al., ICML 2018), and I believe they should be appropriately cited.


Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global longterm dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code and data publicly available 1 .

Summary: This paper proposes a new architecture, R-Transformer, that blends the Transformer networks and the recurrent networks, so as to better capture both the long- and short-term features. By injecting a local RNN layer at every level of the network, the authors hoped to enhance the Transformer's ability to model locality structure. To demonstrate the modeling power of R-Trasnformer, the paper evaluates the effectiveness of R-Transformer on 4 different sequence tasks (seqMNIST, polyphonic music, character- and word-level PTB).
Contribution: The authors propose an architecture that combines the practices of recurrent and feed-forward sequence models. However, I have major concerns regarding the novelty this paper, the various claims it makes, as well as its experiment setting.
----------------------------------------------------------------------------
Major issues/questions:
1. The techniques proposed by this paper lack novelty. For instance, the entire section 3.2 is simply the original design of the multi-head self-attention by Vaswani et al. The major difference between R-Transformer and the original Transformer is the replacement of positional embedding with an RNN layer, but (in my opinion) the authors did not demonstrate sufficiently its effectiveness via ablative studies (see below). Moreover, some prior works have already exploited the locality structure in Transformers. For instance, [1] showed that a sparse, local Transformer can work extremely well and be very efficient (they achieved SOTA on large-scale char-level language modeling tasks).
2. The experiments do not entirely convince me.
    i) The authors use the "same hidden size for R-Transformer and Transformer." But in fact, as the R-Transformer has one extra RNN/LSTM/GRU layer at every level of the network, the tests were carried out (in effect) using a larger model than the baselines. I think the authors should instead control the # of model parameters, especially since you are running only on small tasks with small-sized models.
    ii) It is nice that the authors tested R-Transformer on a variety of tasks--- this is important. However, in no way do these number achieve the levels of the "state-of-the-art", which the authors claim at the end of Section 1 (e.g., [2] has better number on seqMNIST and character-level PTB, and the Transformer-XL actually achieves <55 perplexity on word-level PTB). Therefore, the numbers don't look particularly appealing to me.
    iii) Lack of more challenging, or large-scale experiments. Sequential MNIST is known to be a relatively simple task, and alternatives such as sequential CIFAR-10 or permuted sequential MNIST are more valuable "small sequence tasks." (e.g., prior works studying long-term sequence model dependencies [3]) I also think that benchmarking R-Transformer on large-scale tasks like WikiText-103 or 1Bword (which prior works like QRNN, TrellisNet, RMC and Transformer-XL all explored) would be much more indicative of its usefulness (and should not be left out in such a paper). For instance, it would be useful to compare on small models with controlled model size, even if larger models that require TPUs are not available.
    iv) Lack of ablative study. Table 4 seems to suggest that Transformer-XL is better than R-Transformer. Is it because of their usage of the relative positional embedding? How does the "finite window size" affect the performance of R-Transformer? Why don't you use Transformer-XL for the other 3 tasks? Lots of interesting questions are unanswered here. More details in (3) below.
3. Regarding the motivation to insert a local RNN at the start of each layer. The authors claim (by citing Al-Rfou et al.) that the positional embedding (PE) have only limited effect. But in fact, Al-Rfou only says that the PE features (which is added only at the first layer, by the design of Vaswani et al.) can get lost once the transformer gets very deep--- which is totally expected. Therefore, Al-Rfou et al. propose to use a learnable embedding for each layer. This in no way suggests that positional embedding has "limited effect" (Note that in their ablative study, when they turned off the learnable PE, they also added back the original PE). Moreover, the authors didn't use convolution to capture local data, because it "completely ignores the sequential information of positions within the local window". That is true. However, **stacking** convolutional layers does capture sequential information. Prior works like [2] showed exactly how temporal convolutions are related to finite-window RNNs. I would suggest the authors to at least compare these different options with ablative studies. I would at least expect a comparison of i) temporal convolution; 2) fixed-window RNN (LocalRNN); 3) unlimited-window RNN; 4) positional embedding; 5) relative PE (which Transformer-XL uses; is that why it's better in Table 4?).
4. The authors claim that the finite-window RNN captures local features. But doesn't that claim only applies to the first layer? Once the first layer multi-head attention mixes all input elements across the sequence, the "local" features fed into the second layer RNN will be, actually, **global** features? Doesn't that "defeat" the purpose of using a local RNN though?
============================
Minor issues that have mild or zero impact on the score:
5. Inconsistent notations. Equation (2) and (3) both describe the LocalRNN(...) function, but clearly have a different input-output signature. Instead of using italics, it's better to have well-defined notations. Another case is Equation (7): you have 'FeedForward(mt)', but 'mt' is not on the right-hand side of the equation at all. Moreover, the symbols used in Eq. (7) are currently inconsistent with Eq. (8).
6. Show the # of parameters in the model in Table 1-4.
7. According to the code released in the dropbox URL, you precompute the index in the finite window and later called 'torch.index_select' to produce a tensor that is 'ksize' larger than the original sequence (cf. Line 130-143 in models/RTransformers.py in the dropbox folder). How does R-Transformer compare to Transformers in terms of speed and memory? Since you convert the batch dimension to (batch_size * seq_len) at every level of the network, I imagine that could slow down the process, especially in high-dimensional/large-scale experiments?
8. There are some typos (e.g., Sec. 4.2, LTSM) and grammatical mistakes in the paper (Sec. 3.3).
9. In Section 4.2, the authors claim that both LSTM and TCN performed better than Transformers on the polyphonic music dataset because "these music tunes exhibit strong local structures". However, the difference between Transformer and LSTM is actually very small, and it's even better than GRU/vanilla RNN. Is that too big a claim to make? In polyphonic music datasets like JSB or Nottingham, there are still longer sequences with longer dependencies...
============================
Overall, I feel that this paper has a good motivation to combine different sequence model families (Transformers, TCNs, RNNs) to improve their modeling power. But at the same time, I feel that the experiments can be a lot stronger, and the paper has limited novelty when compared to prior works. I'm happy to consider adjusting my score if my concerns above are addressed.
[1] "Generating Long Sequences with Sparse Transformers", https://arxiv.org/abs/1904.10509
[2] "Trellis Networks for Sequence Modeling", https://arxiv.org/abs/1810.06682
[3] "Learning Longer-term Dependencies in RNNs with Auxiliary Losses", https://arxiv.org/abs/1803.00144


Off-policy training of contextual-bandit policies is attractive in online systems (e.g. search, recommendation, ad placement), since it enables the reuse of large amounts of log data. State-of-the-art methods for off-policy learning, however, are based on inverse propensity score (IPS) weighting, which requires that the logging policy chooses all actions with non-zero probability for any context (i.e., full support). In real-world systems, this condition is often violated, and we show that existing off-policy learning methods based on IPS weighting can fail catastrophically. We therefore develop new off-policy contextual-bandit methods that can controllably and robustly learn even when the logging policy has deficient support. To this effect, we explore three approaches that provide various guarantees for safe learning despite the inherent limitations of support deficient data: restricting the action space, reward extrapolation, and restricting the policy space. We analyze the statistical and computational properties of these three approaches, and empirically evaluate their effectiveness in a series of experiments. We find that controlling the policy space is both computationally efficient and that it robustly leads to accurate policies.

This paper talks about the problem of off-policy or batch learning in the contextual bandit setting without the complete support assumption. This problem setting is very realistic and encountered in most problems, especially in temporally extended settings, such as reinforcement learning. They compare three approaches for the same: restricting action selection, learning extrapolated reward models, and by restricting the policy class. They derive a SNIPS style estimator for the support constraint in the final approach. The approach with restricting the policy class demonstrates decent empirical results although the direct method is very much comparable.
I would lean towards being mostly neutral in terms of acceptance. While the problem being solved is very relevant and their approach compares three different approaches to the deficient support problem, I am not sure how this work is positioned with respect to approaches solving similar problems in the reinforcement learning land. For example, Batch constrained Q-learning ([1]) restricts the set of actions that can be used, Bootstrapping Error Accumulation ([2]) and SPIBB ([3]) restrict the policy class in batch reinforcement learning. I would appreciate some comparison/positioning to such methods in the bandit setting as well. The support estimation metric and the corresponding objective (Eqn 10, 11) should also be compared and contrast with explicit divergences designed for support matching (for example, in [4]).
References:
[1] Off-Policy Deep reinforcement learning without exploration, Fujimoto et.al. ICML 2019
[2] Stabilizing Off-policy Q-learning via Bootstrapping Error Reduction, Kumar et.al. NeuRIPS 2019
[3] Safe Policy Improvement with Baseline Bootstrapping. Laroche et.al. ICML 2019
[4] Domain adaptation with asymmetrically-relaxed distribution alignment. Wu et.al. ICML 2019


This paper proposes a novel method to detect anomalies in large datasets under a fully unsupervised setting. The key idea behind our algorithm is to learn the representation underlying normal data. To this end, we leverage the latest clustering technique suitable for handling high dimensional data. This hypothesis provides a reliable starting point for normal data selection. We train an autoencoder from the normal data subset, and iterate between hypothesizing normal candidate subset based on clustering and representation learning. The reconstruction error from the learned autoencoder serves as a scoring function to assess the normality of the data. Experimental results on several public benchmark datasets show that the proposed method outperforms state-of-the-art unsupervised techniques and is comparable to semi-supervised techniques in most cases.

In the article, the authors solve the problem of anomaly detection using a fully unsupervised approach. They try to deal with the main challenge of anomaly detection: a lack of certainty on what defines normal data and anomaly one. For this purpose, the authors iteratively use: 1) autoencoders to learn the representation of the data; 2) applying in the latent space clustering to get a new training set and retrain autoencoders. The experimental results show that the author’s method performed better results than such a baseline model as one-class SVM and one-class NN.

The proposed algorithm looks robust and well-motivated, but the text of the article and the experiments can be improved. As the proposed approach is a heuristic, the experiments should be done more persuasively, including more metrics used and more alternative algorithms considered.
The key comments are the following:
1. The formatting of the article needs to be improved e.g.:
2. there is no comma between rows in the equation (1) ;
<<to be accepted into the “training” set, .>> - there is an extra comma;
3. round brackets in the equation (6) should be bigger;
4. Table 3 is bigger than the page sizes.
5. The quality of the pictures should be improved:
- Increase the captions font size in Figure 2;
- The captions and the legends in Figure 3 are practically not visible;
6.  Is the DAGMM method SOTA in anomaly detection with deep autoencoder? There are many other methods with similar ideas. We expect that we should provide a comparison with other methods:
https://arxiv.org/pdf/1809.02728.pdf - IGMM-GAN
https://papers.nips.cc/paper/7915-generative-probabilistic-novelty-detection-with-adversarial-autoencoders.pdf - GPND AE
6. Also, DAGMM works badly according to the experiments in the article with max AUROC in Table 1 only 50.3 (so it seems that it is no better than the coin-flipping)
7. Why was the only selected digit for analysis 4? Usual for comparison anomaly detection on MNIST dataset apply the following procedure: for each figure in dataset consider corresponded class as anomaly data, and the rest of the digits are used as normal data, e.g.:
https://arxiv.org/pdf/1802.06222.pdf
https://arxiv.org/pdf/1906.11632.pdf
8.  Class imbalances can affect the value of the AUROC metric. Possibly, the other metrics like AUPRC, F1-scores will better reflect the work of the algorithms for comparison. Also, AUROC is not representative when it comes to the selection of the threshold for anomaly detection. Precision and Recall can help to get more insights.
9. In Table 3, the result of applying the proposed algorithm presented with standard deviation, but other methods are represented by one metric value. Why? The explanation is required.


We study the important and challenging problem of controllable generation of long-term sequential behaviors. Solutions to this problem would impact many applications, such as calibrating behaviors of AI agents in games or predicting player trajectories in sports. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are significant challenges that are unique to or exacerbated by generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated temporal behavior faithfully demonstrates diverse styles? In this paper, we leverage large amounts of raw behavioral data to learn policies that can be calibrated to generate a diverse range of behavior styles (e.g., aggressive versus passive play in sports). Inspired by recent work on leveraging programmatic labeling functions, we present a novel framework that combines imitation learning with data programming to learn style-calibratable policies. Our primary technical contribution is a formal notion of style-consistency as a learning objective, and its integration with conventional imitation learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the MuJoCo physics environment, and show that our learned policies can be accurately calibrated to generate interesting behavior styles in both domains.

This paper proposes a method to train style-conditional policies. The method has three components, a dynamics model, a labeling procedure and its approximation, and a policy. The model is trained on real trajectories, while the policy is trained on both real and simulated data coming from the model. The policy is trained to both imitate and be sensitive to the style labelling of states.
Such a method yields policies that can be executed with styles chosen externally, e.g. by a user.
This paper proposes a novel method that is an interesting take on imitation learning, but it is hard to judge how relevant this method is, as the paper has several inconsistencies and weaknesses that need to be resolved before it is accepted.
Inconsistencies: the reported NLL results do not match the trajectories shown in Figures. Some equations do not seem to reflect what is being optimized.
Weaknesses: Many quantities that should be reported are not: quality of models, diversity of policies, etc. The source of datasets should be clarified. (see detailed comments)
It's interesting that this method can leverage these very sparse or poor quality annotations, but it would be helpful to get a sense of how good the annotations provided here are. What's the threshold were annotation's quality is too low to be helpful? In the general case I suspect this threshold to be higher than what is hinted at in this paper, especially given Table 2. What happens in more realistic settings where e.g. users may specify hundreds of different labels?
Detailed comments:
- how dependent on having a variety of policies is this method? It's not clear how diverse the set of policies coming from the basketball dataset nor the Cheetah policies is, nor how this diversity affects learning. An experiment with explicitly different levels of diversity would strengthen understanding of this method. For Cheetah, it would be easy to report p(y) as a function of the target forward speed, that would give readers a sense of diversity for each label.
- For something like cheetah, with the labels that you propose being a very simple function of the state space, this somewhat resembles DIAYN [1] but with some grounding. Would it make sense to compare to a similar baseline?
- (Table 1) Why report (only) the median over 5 seeds? Papers usually report means. Plus, an indication of variance would be nice.
- It's not clear what the basketball dataset is. Where does it come from? Does it come with a simulator? If not, how do you evaluate style consistency or run step 11-12 of Alg. 2? Do you have a train/valid/test split to choose hyperparameters? (the appendix only suggests a train/test split)
- Why not cite MuJoCo? [2]
- For Figure 2 & 3, I suggest lowering the transparency/alpha value of the trajectories, as there is a lot of overlap.
- Table 1 is somewhat confusing. In (4) and the paragraph thereafter, you define \mathcal{L}^{style} as an error rate, i.e. when \lambda(\tau) \neq y, but in Table 1 you seem to report instead accuracy? (i.e. when \lambda(\tau) = y) If so, then you are reporting percentages, so \times 10^2 rather than 10^-2.
- You never report how well C_\psi is doing, and it's not clear to me why C_\psi is needed at all. When optimizing (8) are you directly treating (8)'s inner terms as negative rewards which is differentiated wrt \pi's parameters? If so, you are doing a form of DDPG, but there is a problem: after (4) you mention that L^style is not differentiable, meaning C_\psi doesn't provide gradient information to \pi. I assume that you acutally optimize (8) with L^label? Whether that's what you're doing or not, it should be clarified. If you are truly using L^style, then it's not clear why C_\psi is needed, as there is no differentiability anyways, and simply using \lambda directly will provide more signal.
- I'm somewhat perplexed by the values of Table 4. A log-likelihood of -190 represents a probability of 10^-83 (a _negative_ LL of -190 is, on the other hand, impossible by virtue of logs and probabilities, but I assume it is a "typo"). What is this the probability of? Entire trajectories? If so it would make more sense to report the _average_ log-likelihood, i.e. per timestep, because at this point in the paper, readers have no sense of how long trajectories are, and thus what these likelihoods represent. (for example, a NLL of 190 could be an average likelihood of 0.5 for 275 steps, or of 0.1 for 83 steps, or of .8 for 850 steps, which are all very different results! According to the appendix Table 5, the basketball trajectories are 25 steps long, which would mean that the imitation objective is not respected at all, exp(-190/25)=.0005, which would mean that pi(a|s) is .0005 on average.)
- Again Table 4, if the "-" is indeed a typo, then CTVAE-style is actually performing better than the baselines, especially for Cheetah (normally one wants negative log-likelihood to be as close to 0 as possible). Same for Table 10, if the NLL column is truly actually log-likelihood, then the "style+" objective really degrades imitation quality rather than improves it.
[1] Diversity is All You Need: Learning Skills without a Reward Function, Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine
[2] MuJoCo: A physics engine for model-based control, Emanuel Todorov, Tom Erez Yuval Tassa


In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treatment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not controlled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose Rate-Distortion Optimization guided autoencoder, in which the Jacobi matrix from real space to latent space has orthonormality. It is proved theoretically and experimentally that (i) the probability distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant; (ii) our model behaves as non-linear PCA, where energy of acquired latent space is concentrated on several principal components and the influence of each component can be evaluated quantitatively. Furthermore, to verify the usefulness on the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.

Summary of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.
Review tl;dr: weak reject, for three main reasons:
(i) While the existing literature around VAEs, beta-VAEs,  and Rate-Distortion theory is mentioned in the related work, the connections are not nearly discussed sufficiently.
(ii) On top of (i), the derivation of their loss function and architecture is not sufficiently motivated. This is in astonishing contrast to 1.5 pages of main text and 8 pages of (much appreciated!) analysis of properties.
(iii) Given the paper is clearly related to existing approaches in the literature, the experiments would require a much more careful comparison to existing models. It remains unclear why an interested user should favor your model over conceptually simpler generative models with fewer hyperparameters.
Detailed review:
Nota bene: This review is a late reassignment. While I reviewed the paper to the best of my ability, time constraints did not allow me to review parts of the paper in depth.  I am open to reassess my review during the second stage.
Connection to prior art: As a probabilistic, neural autoencoding model, the connections to the family of VAE models are obvious. The loss function (eq. (4)) still looks very much like the ELBO, where the typical conditional log-likelihood was split into two distortion terms. How is this different from e.g. a beta-VAE? Particularly, what is the connection between the rate-distortion analysis of beta-VAE by Alemi et al. and yours? These things need to be discussed explicitly, with more than a sentence or two in the related work section.
A lesser, but still important omission in your discussion of prior work: The Jacobian of the generator has also been studied, even for the VAE, cf. e.g. [1]. I believe this deserves more attention in your assessment of prior art.
Motivation: You use two distortion terms: actual sample vs. undistorted reconstruction. Why is that? What is the interpretation of the multipliers? How do I choose them? Why is a large part of your architecture (the pipeline from x to \hat(x)) actually deterministic? Why are you using the entropy of the prior over the latents, rather than the KL divergence between encoder and a prior? I think an interested reader could learn much more from your paper if you discussed your model embedded in th related work rather than in isolation.
Theory: Due to aforementioned time constraints, I was not able to review the extensive theoretical analysis in depth. Still, I would strongly recommend structuring the respective sections more clearly. Separate model and architecture description from the theoretical analysis; precisely formulate your claims. In particular, state your assumptions clearly. For instance, you assume "that each function's parameter is rich enough to fit ideally" (and similar e.g. in Appendix A). Does this only mean that the true distributions are part of the parametric family? What if this is not the case? Do your parameters need to be in the optimum for your analysis to hold true?
Given that the full 20-page manuscript spends 10 pages on theory, I think this contribution is not given appropriate space in the main text.
Experiments: There are three experiments: a simple 3D proof of concept; anomaly detection; analysis of the latent state in CelebA. As mentioned in my review of the methods section, I believe the approach to be very similar to established models. None of the experiments provides convincing evidence why I should prefer the new, arguably more complex model.
For instance, I would have much preferred that you investigate properties of your model against alternatives over the anomaly detection experiments, which did not further my understanding of the proposed model.
Summary: The paper tackles an important problem, namely the lack of control over the latent embedding in autoencoding generative models. I believe the author's contribution can be valuable, and I particularly appreciate the effort to investigate theoretical properties. As is, the case is not sufficiently convincing to be accepted, but I encourage the authors to improve the paper.
Minor comments:
1. While I appreciate a pun, I would recommend to rename the model along with the acronym to a more concise name.
2. Please revise your notation and typsetting. Examples: x1 instead of x_1, f of f(\cdot) instead of f(), \log instead of log.
3. Introduce acronyms before using them (e.g. VAE, MSE, SSIM), even when they seem obvious to you.
4. Please carefully check the manuscript for typos, missing articles, missing spaces etc.
5. Your citations are inconsistent, in that they sometimes use first names, sometimes first name initials, and sometimes no first names.
6. To my knowledge, the term scale function does not have an obvious definition. I think you are simply referring to monotonically increasing functions. Please clarify!
7. Your figures should be understandable without too much context, they need more detailed captions.
[1] http://proceedings.mlr.press/v84/chen18e.html


Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases linearly with the number of symbols and poses a critical challenge on memory and storage constraints. In this work, we propose a generic and end-to-end learnable compression framework termed differentiable product quantization (DPQ). We present two instantiations of DPQ that leverage different approximation techniques to enable differentiability in end-to-end learning. Our method can readily serve as a drop-in alternative for any existing embedding layer. Empirically, DPQ offers significant compression ratios (14-238x) at negligible or no performance cost on 10 datasets across three different language tasks.

This paper works on methods for compressed embedding layers for low memory inference, where the compressed embedding are learned together with the task-specific models in a differentiable end-to-end fashion. The methods in the paper build on a slight variant of the K-way D-dimensional discrete code representation proposed by Chen et al.. Specifically, the two methods in the paper are motivated by the idea that the K-way D-dimensional code can be viewed as a form of product quantization. The first proposed method (DPQ-SX) uses softmax-based approximation to allow for differentiable learning, while the second proposed methods uses clustering centroid-based approximation. Empirically, the authors demonstrate that the proposed methods for generating compressed embedding can achieve matching inference accuracy with the corresponding uncompressed full embedding across 3 NLP tasks; these proposed approach can outperform the pre-trained word embedding and the K-way D-dimensional code baselines in language modeling tasks.
This paper builds on an existing embedding representation approach---K-way D-dimensional code. But I think the perspective on viewing k-way D-dim approach as product quantization (which motivate the differentiable learning approach in the paper) is very interesting. Also I think the empirical performance of the proposed method is promising. I gave weak rejection because 1) the proof of theorem 1 is flawed; 2) The experiment might need additional validation to fully support the merit of the proposed methods.
I list below the major concern / questions I have. I am happy to raise the score if the following questions are properly resolved in the rebuttal:
1. Correct me if I am wrong, I think the proof of theorem 1 is wrong---if the integer based code book C is full rank , it does not necessarily imply the one-hot vector based code book B is full rank. E.g. assume K = 2 D = 2,
C = [1 1;1 2;1 1; 1 2] is full-rank (rank = 2), but the corresponding B = [1 0 1 0 ; 1 0  0 1; 1 0 1 0 ; 1 0  0 1] is not full rank (rank < 4).
2. As the proposed methods advocate training the inference-oriented compressed embedding together with the task models (such as translation models), I think the following naive baseline is necessary to fully evaluate the merit of the proposed approach: one can train the full embedding with the task model as usual, compress the task-specific full embedding using the K-way D-dim approach by Chen et al. or using the deep compositional code learning approach by Shu et al., and then use it for the inference. This provides an alternative way to use product quantization based approach for embedding with low inference memory, without training together with task models. Without this, I can not evaluate the necessity to use the train-together approach the author proposed.
3. The proposed DPQ-SX approach performs better in the two proposed approaches. However this approach uses different K and V matrix where in the original K-way D-dim approach we have K = V. This makes it hard to say if the better performance is due to the decoupling of K and V, or because of the training method inspired from the product quantization perspective. It needs ablation study here.
4. In Table 4, the authors only compare to baselines on the LM task, I am wondering how it compares to the the baselines on the other two translation and text classification models.
For improving the paper, the relatively minor comments are as the following:
1. In equation 4, the partition function Z is not explicitly defined.
2. In the second paragraph of section 3.1, it is not clear what exactly is the pre-trained embedding used as baseline.
3. For better readability, it is better to inflate the caption of figure and tables to provide useful take-away message there.


We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network.

In this paper, the authors propose a sparse momentum algorithm for doing efficient sparse training. The technique relies on identifying weights in a layer that do not have an effect on the error, pruning them, and redistributing and growing them across layers. The technique is compared against other recent algorithms on a range of models.
The paper is well written, and very easy to read. The proposed algorithm looks interesting and seems to empirically work well.
I am, however, a bit confused with how the sparse momentum algorithm is discussed in this paper. In the paper, momentum is intuitively presented as an algorithm that reduces the variance of the noise in the gradients. However, there are a number of papers that show that this is not the case (and if anything it is the opposite). Further, a few recent papers show that momentum works better than SGD only for learning rates that are not too small, and this is because it averages out the gradients in the high-curvature directions (these are the directions where the gradients switch signs) and makes them stable in these directions, thus allowing larger steps in the low curvature directions. See for example the following two papers:
Momentum Enables Large Batch Training. Samuel L Smith, Erich Elsen, Soham De. ICML Workshop on Physics for Deep Learning, 2019.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, Roger Grosse. NeurIPS 2019.
Given these papers, can the authors comment on what they think is the reason for the effectiveness of their sparse momentum algorithm? These papers seem to indicate to me that an interesting (and important) ablation study would be to compare using just the gradients vs using momentum for the sparse training algorithm for both a small batch and a large batch. Without this ablation study, it is a bit unclear to me why/when this algorithm is working well, and this primarily explains my current score.
The experimental results look impressive, although I am not very aware of other work in sparse training, so unfortunately it is harder for me to properly evaluate the significance of the empirical results in this paper, and whether the numbers reported are indeed a significant improvement over current state-of-the-art algorithms. I do have a few questions about the design and the results of the experiments presented:
1. Because the momentum of zero-valued weights are used, does that mean that the gradient over all weights are required to be taken at each step? How much does this affect training time in your experiments?
2. How were the learning rates decided, and why are they kept fixed across methods? It seems feasible that the optimal learning rate could vary highly between methods?
==========================================
Edit after rebuttal:
I thank the authors for the very detailed response and for doing the additional experiments requested. Due to the thoroughness of the response, I am increasing my score to a weak accept. I do however have a couple of comments regarding the author response:
1. I do not agree that keeping the learning rate fixed across methods is the right approach. Many different things might interact with each other to change the optimal learning rate, and it is always more interesting to see the optimal performance attained by a method through a grid search. As long as the tuning budget across all methods are somewhat similar, I think the experiments would be more informative when doing a learning rate sweep. Further, while doing the sweep, the authors should be careful that the optimal learning rate does not lie on the edge of the grid search. This is what happens in the experiments reported in the authors' response titled "Results from learning rate grid search" where the optimal learning rate for all models lie at one of the extremes of the grid search (0.09).
2. I would request the authors to slightly rewrite certain parts of their paper so as not to imply that momentum decreases the variance of the gradients in general. There are many papers (including the ones I mentioned in my review) that specifically show momentum does not have this effect. All of these papers however consider a different setting from the one considered in this paper (where there is a parameter redistribution step), so it is not immediately clear whether momentum has the same effect in this case. However, unless there is very specific evidence to show a variance reduction effect, I think putting in that intuition might be misleading.
It is interesting to see that momentum helps for small batches in this setting, and I agree that further investigation into this would be an interesting future direction. Note that there is a concurrent submission on sparse training (https://openreview.net/forum?id=ryg7vA4tPB&noteId=ryg7vA4tPB) that seems to show that higher momentum values (0.99) does better when considering large batches. So perhaps additional and more careful experiments need to be done about this before these additional experimental results can be put in the paper.


Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule. Both these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour. On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their metagradients, leading to methods that can not scale beyond few-shot task adaptation. In this work we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations. WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning. Learning (how)  to learn implies inferring a learning strategy from some set of past experiences via a meta-learner that a task-learner can leverage when learning a new task. One approach is to directly parameterise an update rule via the memory of a recurrent neural network (Andrychowicz et al.,  2016; Ravi & Larochelle, 2016; Li & Malik, 2016; Chen et al., 2017). Such memory-based methods can, in principle, represent any learning rule by virtue of being universal function approximators (Cybenko, 1989; Hornik, 1991;Schäfer & Zimmermann, 2007). They can also scale to long learning processes by using truncated backpropagation through time, but they lack an inductive bias as to what constitutes a reasonable learning rule. This renders them hard to train and brittle to generalisation as their parameter updates have no guarantees of convergence.

Summary:
The current paper deals with meta-learning and essentially proposes a generalization of MAML (a popular gradient-based meta-learning algorithm) that mostly builds upon two main recent advances in meta-learning: 1) an architectural one (see e.g. T-Nets), which consists in optimizing the parameters of additional layers during the meta-learning outer loop (as opposed to only optimizing the initial conditions of the original parameters like in MAML), and 2) a theoretical one (see e.g. Meta-SGD, Meta-curvature), which is based on the geometrical observation that one set of parameters can precondition a second set of parameters that are consequently being optimized in a "warped" geometry, possibly speeding up learning.
The authors provide a great and thorough overview of the literature, in particular for gradient-based meta-learning methods, which helps putting all this in perspective.
The way they obtain the mentioned "warped" geometry in practice is by adding additional so-called warp-layers to an architecture that is being trained with meta-learning. Such warp-layer are generic deep learning modules (such as convolutions followed by BatchNorm, or LSTM layers), which are being trained in the outer-loop of the meta-learning optimization. In this sense, WarpGrad extend T-Nets, which only allowed for linear layers.
The second main innovation of WarpGrad is the proposal of a new meta-learning objective, which incorporates a meta-learning internal loop of only one step of (preconditioned) SGD, meaning that, as the authors notes, "in contrast to MAML-based approaches (Eq. 1), [...] avoids backpropagation through learning processes".
The authors test their algorithm on several meta-learning benchmarks, including few- and multi-shot learning tasks demonstrating very competitive performance when their algorithm is combined with MAML or Leap. They then deploy WarpGrad on a maze navigation reinforcemente learning task to demonstrate training of recurrent architectures, and on a continual learning toy dataset to show that their objective can be adapted to mitigate catastrophic forgetting.
Decision:
This is a good paper which proposes an interesting generalization of previous gradient-based meta-learning methods like MAML and T-Net, with an impressive number of experiments. However, some of the statements regarding the advantages of WarpGrad over previous algorithms seem a little bit misleading, in particular in situations where WarpGrad needs to be combined with these same algorithms. For instance (and I might have completely misunderstood things here), it seems that when the WarpGrad objective is being combined with MAML (which requires backpropagation through multiple-step gradient descent trajectories), then also the resulting combined objective will necessarily need to backprop through the same multi-step trajectory, defeating the stated advantage of the WarpGrad algorithm (i.e. that its objective avoids backpropagating through the learning processes).
In general, even if one only considers the WarpGrad objective eq. (10), that comprises a meta-learning inner loop which consists of one step of (preconditioned) gradient descent. However, it seems like an arbitrary (and limiting) choice of the authors to only perform one step, as opposed to multiple ones. As a matter of fact, even very sophisticated second order gradient descent methods like natural gradient descent typically require more than one step to reach a local minimum. That is to say, that the main advantage showcased by the authors (the fact that the WarpGrad objective avoids backprop through a whole learning trajectory) seems like a limitation, rather than the result of a principled derivation.
It would be beneficial if the authors could clarify this points. In particular, whether combining WarpGrad with MAML does not indeed negate the stated advantages of WarpGrad over MAML, and whether there is a principled way of demonstrating that executing only one step in the inner loop of the WarpGrad objective is completely general (i.e., additional steps do not help the inner loop).
Minor:
- The authors use the wrong citation key when referring to the T-net paper: it should be Lee et al 2018, instead of Lee et al. 2017
- I believe that when the authors mention Fast and slow weights, they are being described in the opposite way: slow weights should be in charge of meta-learning information, while fast ones are in charge of task-specific information.
- Line 3 and 4 of Algorithm 1 and 2: shouldn't it say "mini-batch of tasks" (plural), instead of "mini-batch of task", since several tasks are being sampled? Otherwise, it might be erroneously interpreted as "mini-batch of (samples belonging to) task T".
- The comment that "learning to precondition gradients can be seen as a Markov Process of order 1" is never clearly elucidated or developed. It would help to develop this.


Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4-8.2 percent points under existence of real-world noise.

This paper proposes a training strategy for robustness against label noise. The training strategy is simple and straightforward. The neural network will first be trained on the entire dataset with all the noisy labels. After obtaining the network with lowest validation error, the network will be used to make a prediciton on the original training set and select a subset of it to construct a maximal safe set. Finally, the network will be findtuned on this maximal safe set. The training strategy is very similar to tradictional  self-training in semi-superivsed learning and co-training for domain adaptation ([Co-training for domain adaptation, NIPS 2011]), except that the proposed prestopping only iterate the procedure once.
The paper discusses two important questions for the method: (1) when to early stop the training; (2) how to constuct a maximal safe set. The authors' responses to these questions are very natual but less interesting. Using the lowest validation error to early stop the training could be suboptimal, since the small validation set can not fully capture the data distribution and could make the network empirically overfit to this validation set. The criterion to contruct a maxial safe set is also conventional, and is similar to what a number of papers are doing, for example,
[1] Co-training for domain adaptation, NIPS 2011
[2] Self-ensembling for visual domain adaptation, ICLR 2018
[3] A dirt-t approach to unsupervised domain adaptation, ICLR 2018
[4] Iterative learning with open-set noisy labels, CVPR 2018
In experiments, the results are not very surprising. There are some baselines that adopt a similar (iterative) pipeline (learning the network - selecting a subset of the training samples - re-learning the network):
[1] Iterative Learning with Open-set Noisy Labels, CVPR 2018
[2] Dimensionality-Driven Learning with Noisy Labels, ICML 2018
[3] Symmetric Cross Entropy for Robust Learning with Noisy Labels, ICCV 2019
The authors can consider to compare to some of these baselines, especially [1] and [2]. The difference between the paper and [1,2] is basically the criterion to construct the maximal safe subset.
Besides, I suggest the authors to conduct large-scale experiments on ImageNet or even a subset of ImageNet, since the difficulty of detecting label noise is much higher when the resolution of images become bigger. CIFAR-10 and CIFAR-100 only contain 32x32 images, which is far less challenging.
Overall, I think the paper is well written, the idea is clearly presented, and the experiments also seem convinceing. However, the contribution of this paper is very incremental.


The nonlocal network is designed for capturing long-range spatial-temporal dependencies in several computer vision tasks. Although having shown excellent performances, it needs an elaborate preparation for both the number and position of the building blocks. In this paper, we propose a new formulation of the nonlocal block and interpret it from the general graph signal processing perspective, where we view it as a fully-connected graph filter approximated by Chebyshev polynomials. The proposed nonlocal block is more efficient and robust, which is a generalized form of existing nonlocal blocks (e.g. nonlocal block, nonlocal stage). Moreover, we give the stable hypothesis and show that the steady-state of the deeper nonlocal structure should meet with it. Based on the stable hypothesis, a full-order approximation of the nonlocal block is derived for consecutive connections. Experimental results illustrate the clear-cut improvement and practical applicability of the generalized nonlocal block on both image and video classification tasks.

The paper proposes a spectral non-local block, which is a generalized method of the non-local block and non-local stage in the literature. The proposed spectral non-local block can be plugged into a neural network to improve its effectiveness. The paper also provides theoretical analyses of the stability of the proposed method, and also extend the method by including more  Chebyshev polynomial terms. Experiments are conducted on image classification and action recognition tasks, and they valid the effectiveness of the proposed method.
The idea is well-motivated, and it is a generalization of existing works in the literature. I do like this idea. However, I am afraid that the idea is not well explained and supported, thus I gave a weak reject to encourage the authors to further improve the paper.
The major concern I have is the reasonability of the experiments.  The experiments in the paper show relative performance gain with respect to a baseline method. It seems that there is a lack of comparison with state-of-the-art methods in the literature. For example, in Table 8, a performance gain is observed when compared with I3D. However, the recent STOA models can achieve much higher accuracy than the baseline. and also the proposed method. Since the proposed method is generic to all neural nets, it makes more sense to compare with SOTA and make improvements based on SOTA.  What is the conclusion from Table 4? Are you trying to demonstrate that the best configuration is DP3, and increasing the number of consecutive non-local blocks (from SP3 to SP5) doesn't work? It is awkward since the paper gives a stable hypothesis for deeper nonlocal structure, but experimentally the deeper structure doesn't work well. Figure 4 is abrupt without much background descriptions. Are the images randomly chosen? Ours here means SNL or gSNL? Is the colored superimposition the attention map (I believe so but the paper doesn't indicate so) and how to interpret it? What is the relation of the coverage of the critical parts on birds and the long-range dependency? More background descriptions and interpretations of the results are needed.
Another concern I have is the clarity of the writing. There are quite a number of informal use of English, mismatched descriptions, undefined acronyms, etc. For example, in the caption of Fig. 1, it is said self-attention and self-preserving are taken effect by W1 and W2, which is contradictory to what is illustrated in the figure. Also, the terms self-attention and self-preserving, and other terms such as CGNL, A2, Hadama (Hadamard?) product, are not formally defined or described.  A lot of grammar errors and informal use of English are present, such as "which lead to", "the weight means", "when using in the neural network", "fig. 4", "Figure. 2", "more liberty for the parameter learning.", etc.