Deep learning models achieve high predictive accuracy in a broad spectrum of tasks, but rigorously quantifying their predictive uncertainty remains challenging. Usable estimates of predictive uncertainty should (1) cover the true prediction target with a high probability, and (2) discriminate between high-and low-confidence prediction instances. State-of-the-art methods for uncertainty quantification are based predominantly on Bayesian neural networks. However, Bayesian methods may fall short of (1) and (2) -i.e., Bayesian credible intervals do not guarantee frequentist coverage, and approximate posterior inference may undermine discriminative accuracy. To this end, this paper tackles the following question: can we devise an alternative frequentist approach for uncertainty quantification that satisfies (1) and (2)? To address this question, we develop the discriminative jackknife (DJ), a formal inference procedure that constructs predictive confidence intervals for a wide range of regression models, is easy to implement, and provides rigorous theoretical guarantees on (1) and (2). The DJ procedure uses higher-order influence functions (HOIFs) of the trained model parameters to construct a jackknife (leave-one-out) estimator of predictive confidence intervals. DJ computes HOIFs using a recursive formula that requires only oracle access to loss gradients and Hessian-vector products, hence it can be applied in a post-hoc fashion without compromising model accuracy or interfering with model training. Experiments demonstrate that DJ performs competitively compared to existing Bayesian and non-Bayesian baselines.

This paper studies how to construct confidence intervals for deep neural networks with guaranteed coverage. The authors propose an algorithm, “discriminative jackknife”, based on the standard jackknife confidence interval estimate which they augment by a “local uncertainty estimate” based on the variability of the n leave-one-out fitted versions of the underlying algorithm (n = # data points). The whole study is concluded with toy and real-world examples showing the proposed algorithm is competitive with existing methods while also achieving the desired coverage.
I am currently leaning towards recommending rejection of the paper. The main reasons for this are: (i) A potential failure to cite and acknowledge the prior contribution of [1] which seems to have non-trivial overlap with this paper (on arxiv since end of July which is more than 30 days before the ICLR submission deadline and thus should be treated as prior work); (ii) The claims of guaranteed frequentist coverage are not backed up as, according to thm.2, they only hold when n >> 0 and the number of influence functions used goes to infinity (ideally, the authors would provide non-asymptotic bounds as in [1], but at the very least, these limitations should have been clearly pointed out and their practical implications discussed). Finally, I would like to say that I am holding the paper to a higher standard due to its 10 page length as instructed by the guidelines.
Major comments:
- Can you please explain the relation of this work to [1]? It seems that [1] already proposes use of the higher order expansions, provides an efficient implementation based on forward mode autodiff (do you plan to release code?), and moreover provides non-asymptotic bounds which are not present in your work?!
- On a related note, Giordano et al. provide a careful analysis and discussion of the assumptions in their sect.4. Can you please clarify which of the assumptions you also make, and why you don’t need the others (if any)?
- Throughout the paper (e.g., in and around eq.1), you seem to assume that there exists a unique minimiser of the objective which generally won’t be true (especially in your application to deep neural networks). In [1], the technical assumptions ensure this is true but I don’t see how this is handled in your case?! Can you please help me understand how to interpret your work in case there are multiple (possibly local) minima, and whether this has any effect on the results in thm.2?
- I think it would be beneficial to the reader if you could please provide a discussion and/or formula for how large n has to be for Theorem 2 to apply.
- On p.8, you say “To ensure a fair comparison, the hyper-parameters of the model f(x; θ) were the same for all baselines.” I am not sure I agree this is a fair comparison. Commonly, one has the opportunity to select hyperparameters for their algorithm (e.g., using a validation set, cross-validation, etc.) so it seems it would have been fair to run each algorithm with its best hyperparameters. Can you please provide a discussion of how this would affect the reported results?
Minor comments:
- On p.3, you say “We do not pose any assumptions on how the loss function in (1) is optimized.” Do you mean to say that you do not assume anything **but** that the chosen optimiser reaches a (global?!) minimum of the loss function? It seems like you would want to exclude pathological optimisers (e.g., one that always outputs zero) but also more realistically think about the known pathologies of commonly used optimisers (see, e.g., [2]).
- Can you please clarify if and how the use of the algorithm from (Agarwal et al., 2016) for approximation of the Hessian products affects accuracy of your confidence intervals?
- In fig.3, it seems like some of the methods are not properly tuned. For example, MC-dropout should not have zero uncertainty around zero (did you by any chance set bias variance to zero?!), and BNN-SGLD does not seem to have converged (can you please provide plots providing some evidence that the MCMC sampler has mixed + information about how the hyperparameters were selected?).
- I am somewhat confused by the statement “The only hyper-parameter involved in our method is the number of HOIFs m — this was tuned by optimizing the evaluation metrics ...” on p.8. Wouldn’t thm.2 suggest that you should use as high m as possible?
- Also on p.8, can you please clarify how you selected the threshold for the evaluation of “discriminative power”? In particular, thm.2 seems to suggest that what one would desire is that the ranking based on width of predictive intervals is equivalent to the ranking based on the actual prediction error. This would suggest, for example, comparing these two rankings using Kendall’s tau coefficient (perhaps a binned modification where one would count only the number of times true error puts a point into a different bin than the width of its confidence interval). Note that I am not suggesting the above method is perfect either (it completely ignores the actual sizes of the intervals), but I’m currently having trouble interpreting the results you report, so it would be very helpful to understand how you selected this particular measure of “discriminative power” and why alternatives like the example above were discarded please.
References:
[1] Ryan Giordano, Michael I. Jordan, Tamara Broderick. A Higher-Order Swiss Army Infinitesimal Jackknife. https://arxiv.org/abs/1907.12116
[2] Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. https://arxiv.org/abs/1705.08292


One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions. While this paradigm has inspired significant research on the properties of large networks, relatively little work has been devoted to the fact that these networks are often used to model large complex datasets, which may themselves contain millions or even billions of constraints. In this work, we focus on this high-dimensional regime in which both the dataset size and the number of features tend to infinity. We analyze the performance of a simple regression model trained on the random features F = f (W X + B) for a random weight matrix W and random bias vector B, obtaining an exact formula for the asymptotic training error on a noisy autoencoding task. The role of the bias can be understood as parameterizing a distribution over activation functions, and our analysis directly generalizes to such distributions, even those not expressible with a traditional additive bias. Intriguingly, we find that a mixture of nonlinearities can outperform the best single nonlinearity on the noisy autoecndoing task, suggesting that mixtures of nonlinearities might be useful for approximate kernel methods or neural network architecture design.

This paper investigates the asymptotic spectral density of a random feature model F(Wx + B).  This is an extension of existing result that analyzed a model without the bias term, i.e., F(WX). This extension requires a modification of the proof technique. In addition to that, it analyzed a mixture of linear and non-linear activation functions, and show that mixture is better than single nonlinearity in terms of expected training error for ridge regression estimators.
Pros:
- This paper investigates an interesting problem and it successfully extends the existing work. The theoretical curve well matches the simulated curve.
- The finding that mixture of nonlinearities gives better expected training error is interesting.
Cons:
- The extension to the model with bias seems a bit incremental. In practice, we may consider an input with additional constant feature, X <- [X,1], to deal with both models in a unified manner. There should be more discussion about why this kind of trivial argument cannot be applied in the analysis.
- The effect of mixture of activation functions is investigated in the "training error," but I don't see much significance on investigating the training error thoroughly. Instead, people are interested in the test error. I guess there does not appear such a trade-off for the test error and the linear activation function would be always better because the true function is the linear model. Hence, more expositions about why the training error is investigated should be provided.
More minor comment:
- I guess the definition of Etrain  (Eq.(17)) requires an expectation with respect to the training data.
- Assumptions of the activation function f should be provided; is it just assumed to be differentiable?, ReLU is included?
- The definition of G(\gamma) in page 6 had better to be consistent to that in previous pages.


Visual attention mechanisms have been widely used in image captioning models. In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TVMAX). With sparsemax, we obtain sparse attention weights, selecting relevant features. In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TVMAX. By selecting relevant groups of features, the TV-MAX transformation improves interpretability. We present results in the Microsoft COCO and Flickr30k datasets, obtaining gains in comparison to softmax. TV-MAX outperforms the other compared attention mechanisms in terms of humanrated caption quality and attention relevance.

This paper proposes two sparsifying methods of computing attention weights, dubbed sparsemax and TVmax, which appear to slightly improve objective and subjective image captioning scores.    The sparsifying projections are posed as optimization problems, and algorithms for their computation, along with formula for their gradients are given.  Proof of the optimality of these algorithms relies significantly on prior work, so could not be checked deeply without bringing in additional sources.
It is not clear that the motivation for these sparsifying objectives is sound.   The conventional softmax approach to attention weights should be capable of producing attention weights near zero, which would be effectively sparse, especially if the pre-activations, z_i, in equation (1), are allowed to have a large enough range.   It's not clear why weights should need to be zero exactly in the ignored regions, since being near zero should be sufficient to contribute almost nothing to the subsequent weighted sum.    So it is also not clear why the strict sparsity itself, as opposed to the effective sparsity of the softmax, should explain the differences in Figure 1, and in the results.  In particular it is unclear why the strict sparsity should prevent repetition; when looking at the weight distributions in the two cases, a more likely story seems to be that the weight distributions don't repeat as much from one word to the next in the second case, but there is no clear reason to attribute this to sparsity.   The pictures of the attention weights are lacking a color scale, so it is impossible to see how close to zero it comes in the unattended regions, although the gray color values chosen for these regions might be misleading.
The TVmax approach, in addition to sparsity, also constrains the non-zero region to be contiguous.   To the extent that this improves performance, this presumably introduces an inductive bias that matches the data.   It is unclear why this fails to produce better objective scores than sparsemax, while producing better human ratings.   In any case it is not clear why this should necessarily be a good inductive bias for all images, although it is plausible that it helps in some cases.
In many neural network problems, what makes a difference has more to do with the optimizability of the gradients, than the specific activations per se, and that might be the case here too, although the paper does not analyze this aspect of the proposed models.
Overall the paper is flawed by the lack of clarity in the motivation for the proposed methods, and the lack of retrospective analysis and understanding of why the proposed methods should improve results.


Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.

This paper aims to theoretically understand the reason that MaxEnt RL (RL with an entropy bonus) works so well. It suggests that MaxEnt RL works well in the setting where there is uncertainty about the reward function. It proves two main theorems to support this. First, for any instance of a class of “meta-POMDPs” where the agent only has a *belief* over the goal trajectory, there exists a reward function for which MaxEnt RL on that reward function is the optimal solution to the meta-POMDP. Second, for any reward and MDP, there exists a set of reward functions such that MaxEnt RL maximizes the worst-case return for a reward chosen from that set.
The paper tackles an important question, since entropy bonuses are commonly used in RL, but are primarily a “hack” added to incentivize exploration without any principled justification. However, I question the applicability of the theorems to the success of MaxEnt RL.
I’m recommending a weak reject, for the following reasons, in order of importance (explained in more detail later):
- Section 4.2 assumes that you must deploy a single policy across all timesteps of the meta-POMDP, but it should be possible to update your policy across timesteps.
- Section 4.3 claims to reduce goal-reaching problems to single-goal-trajectory problems, but this doesn’t work because goal-reaching problems can have multiple optimal trajectories.
- The adversary of Theorem 5.2 is quite unusual, and it is unclear why robustness to such an adversary should be useful.
- Since the theorems are about cases where the agent is uncertain about the reward, they don’t explain why MaxEnt RL is useful even in the case where we have no adversaries and care only about performance on a single reward function.
- The paper is not very clear. The quality of exposition could be improved significantly.
It seems quite likely that some of my critiques are misguided (especially the one about Section 4.2), and I encourage the authors to point this out in the rebuttal.
----
My primary complaint is that these theorems don’t apply to the case they are meant to explain: the authors say they want to explain why MaxEnt RL works in practice, but their explanations and theorems center on cases in which the reward is unknown. But currently MaxEnt RL works well on tasks where the reward is known! It’s not that MaxEnt RL finds different solutions that don’t do well on the original reward but do better by some other criterion: the solutions found by MaxEnt RL are the best solutions when evaluated by the known reward.
Nonetheless, it is still an interesting question to study the benefits of MaxEnt RL in the context of reward uncertainty. I’d recommend that the authors change the motivation and introduction to focus on that setting, without claiming to explain why MaxEnt RL works with current systems. For the rest of the review, I’ll evaluate the paper from that perspective.
----
I found the section on meta-POMDPs very confusing. First, one minor confusion: I believe you are assuming that the underlying MDP is deterministic? (Perhaps not, but the assumption that there is a policy with probability proportional to sqrt(p(tau)) is much less likely to hold in stochastic environments.)
Section 4.2 is confusing to me. First, let me quote the description of the meta-POMDP: “Each meta-step of the meta-POMDP corresponds to one episode of the original MDP. A meta-episode is a sequence of meta-steps, which ends when the agent solves the task in the original MDP. Intuitively, each meta-episode in the meta-POMDP corresponds to multiple trials in the original MDP, where the task remains the same across trials. The agent keeps interacting with the MDP until it solves the task.”
By my understanding, this means that in the meta-POMDP the agent is able to learn across episodes. In particular, in the setting where the agent has a distribution over a single goal trajectory, and the underlying MDP is deterministic, the optimal policy is obvious: try the most probable goal trajectory, then try the second most probable goal trajectory, and so on until you find the true goal trajectory, and then repeatedly execute that goal trajectory. Why is this not the result of Section 4.2?
(My guess is that you assumed that the agent is not able to learn across episodes (why?), or that the agent receives literally zero information about whether or not it has completed the goal (highly unrealistic, and certainly doesn’t apply to the physician example). Perhaps there is a different unstated assumption instead.)
I’d encourage the authors to more clearly formalize the meta-POMDP as well as the relevant assumptions that lead to the results of Section 4.2. As currently defined, I don’t agree with Section 4.1, though I do think there is a formalization which makes everything work (though that formalization does not seem realistic to me).
----
I’m also confused about Section 4.3. In Section 4 (before subsections) and Section 4.1, it seems that the model is that there is a *single* goal trajectory which the agent must replicate, but the agent is uncertain about that trajectory. However, Section 4.3 considers goal-reaching problems, in which there can be *multiple* goal trajectories. I don’t understand how the goal-reaching problem is reduced to the single-goal-trajectory problem: I believe the current reduction is *not* solving a goal-reaching problem.
For simplicity, let’s consider a goal-reaching problem where the goal is known: a 3x3 grid where the agent starts at the bottom left and the goal is to get to the top right. Assume a horizon of 4 for simplicity. Then any trajectory involving two Ups and two Rights solves the problem, i.e. there are six optimal trajectories: {UURR, URUR, URRU, RUUR, RURU, RRUU}. However, the meta-POMDP defined in the reduction assigns 1/6 probability to each such trajectory, which by the semantics of the meta-POMDP means that the agent “actually” wants to choose one of those trajectories in particular, but doesn’t know which one is appropriate. This is not an accurate representation of the goal-reaching problem: the goal-reaching problem is solved as long as *any* of these trajectories are selected.
(Lemma 4.2 is technically accurate, because it only talks about the meta-POMDP and the corresponding reward function, and doesn’t claim anything about the relation to the goal-reaching problem, but all the prose around Lemma 4.2 is misleading.)
This general problem persists even if you have a belief over the goal state, rather than being certain about the goal state. You could try fixing this by changing the meta-POMDP to put probability over *sets* of goal trajectories, and Section 4.1 would still go through (the regret would be a geometric random variable of the probability the trajectory lies within the true goal trajectory set), but Section 4.2 would no longer work.
----
Theorem 5.2 shows that there exists a set of reward functions such that MaxEnt RL maximizes the worst-case return for a reward chosen from that set. Taken literally, this is not particularly interesting, as it is also true of regular RL: the optimal regular RL policy pi optimizes the worst-case return for the set { r’(s, a) = r(s, a) + f(s, a) with f in F }, for any class of non-negative functions F that contains the zero function, simply because every other reward in the set is at least as large as the true reward r(s, a). (Another set that works is { r’(s, a) = K r(s, a) with K > 0 }.)
So really the interesting content of the second theorem is the particular set of reward functions that MaxEnt RL is robust to: the set {r’(s, a) = r(s, a) - log q(a | s) for all q in Pi}. This is an interesting adversary model: essentially, at every state, the adversary is required to provide a non-negative bonus b_a for each action a. To prevent the adversary from providing zero bonus everywhere, the bonuses must satisfy \sum_a exp(b_a) = 1.
This essentially means that at each state, the adversary wants to find actions that the policy doesn’t take, and allocate more of the bonus to that action. Given this particular adversary, it makes sense to inject a little noise into the policy so that the adversary can’t “hide” the bonus in actions that are rarely taken, and so it makes sense the MaxEnt RL could be the best response to such an adversary. However, this is quite a strange adversary -- we wouldn’t usually expect that the things we need to be robust to are going to be dividing up some bonus across actions. So I’m not very confident that this theorem will actually matter in practice.
----
Typos:
Abstract: “Probability, as this strategy is called,”: “Probability matching”
Introduction: “model for decision decision making”: repeated “decision”
Introduction: “empirical benefits of MaxEnt RL arise implicitly solving”: “arise by implicitly solving”
Preliminaries: “These approaches cast optimal control as an inference problem be defining”: “be” --> “by”
Preliminaries: In the derivation of Eq. 1, p(s_1) was dropped
Preliminaries: The last equality in the derivation of Eq. 1 would only hold if you had a log on the LHS, so you probably wanted to maximize log p(O_t) in the entirety of the derivation.
Lemma 4.2: “p˜(sT(τ), aT(τ)) = p(τ)”: Reverse the order to write “p(τ) = p˜(sT(τ), aT(τ))” to follow the convention that the thing being defined is on the LHS


Recurrent Neural Networks (RNNs) are the state-of-the-art approach to sequential learning. However, standard RNNs use the same amount of computation at each timestep, regardless of the input data. As a result, even for high-dimensional hidden states, all dimensions are updated at each timestep no matter the choice of recurrent memory cell. Reducing this rigid assumption could allow for models with large hidden states to perform inference more quickly. Intuitively, not all hidden state dimensions need to be recomputed from scratch at each timestep. Thus, recent methods have begun studying this problem by imposing mainly a priori-determined patterns for updating the states at each step. In contrast, we now design a fullylearned approach, SA-RNN, that augments any RNN by predicting discrete update patterns at the fine granularity of independent hidden state dimensions. This is achieved through the parameterization of a distribution of update-likelihoods driven by the input data. Notably, through this approach we impose no assumptions on the structure of the update pattern. Better yet, our method adapts the update patterns online, allowing different dimensions to be updated conditional to the input. To learn which dimensions to update, the model solves a multi-objective optimization problem, maximizing task performance while minimizing the number of updates based on a unified control. Using publicly-available datasets we demonstrate that our method consistently achieves higher accuracy with fewer updates compared to state-of-the-art alternatives. Additionally, our method can be directly applied to a wide variety of models containing RNN architectures.

Summary: This paper proposes selective activation RNN (SA-RNN), by using an update coordinator to determine which subset of the RNN’s hidden state dimensions should be updated at a given timestep. The proposed loss term is then a sum of the original objective (e.g. classification) and a weighted sum of the probability that each dimension will be updated for each timestep. The method is evaluated on 3 time series datasets: Seizures, TwitterBuzz, Yahoo.
Decision: Weak Reject. Although the authors tackle a challenging problem, their empirical results are lacking to provably demonstrate that their approach outperforms existing baselines.
Supporting Arguments/Feedback: The authors compare SA-RNN to 5 baselines: random updates, clockwork RNN, phased LSTM, Skip RNN, and VC-GRU. Although I appreciated the authors’ comparison across the suite of methods with respect to various metrics (e.g. # FLOPS, proportion of neurons that weren’t updated, etc.), the experiments were conducted on datasets that were relatively simple. For example, in prior work, the empirical evaluations were on much larger-scale datasets such as Wikipedia [Shen et. al 2019], real clinical data sources [Liu et. al 2018], and Charades videos [Campos et. al 2018], among others. I would be very interested to see how this training procedure fairs when evaluated on much more complex tasks, and would make the results about computational speedups at train/test time much more convincing.
Questions:
- I’m curious if you tried different types of gradient estimators to get around the non-differentiability rather than the straight-through estimator. Also how was the slope-annealing conducted (e.g. annealing schedule)?


Granger causality is a widely-used criterion for analyzing interactions in largescale networks. As most physical interactions are inherently nonlinear, we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements. Our proposed approach relies on modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units (SRUs). We make a case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes' time series measurements. We propose a variant of SRU, called economy-SRU, which, by design has considerably fewer trainable parameters, and therefore less prone to overfitting. The economy-SRU computes a low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing. Additionally, the internal weight parameters of the economy-SRU are strategically regularized in a group-wise manner to facilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events. Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP, LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality.

In this paper the authors propose using Statistical Recurrent Units to predict the network for Granger causality. They motivate this choice by the high representation power of SRUs for multivariate time series, the good performance they usually enjoy and as a way to alleviate the vanishing gradient problem. More importantly the particular form of the SRUs gives a very simple predictor and therefore explanability for Granger causality: the authors propose to simply mark serie $i$ as Granger caused by $j$ if the $j$th column of the input mixing matrix of the $g_i$ is non-zero.
In order to force whole columns of the input matrix to be negative, the authors use a group regularization on the columns of the input matrix $W_{\text{in}}$. The resulting problem is then optimized by proximal gradient descent.
The second contribution of the authors is eSRU, a smaller (in terms of parameters) variant of SRU in order to prevent overfitting.
First the authors introduce a dimensionality reduction layer before the SRU units by using fixed non-trainable random projections. Then the authors propose adding a regularization term for the output mixing matrix, which represents the bulks of the parameters of the model.
The eSRU and SRU causal models presented are then compared to the previous state of the art on several datasets, both synthetic and real, where they manage to reach a new state of the art.
Regarding the two improvements in eSRU proposed by the authors, I have several questions:
First, regarding the random projections, have the authors tried learning the projections ? While it increases the risk of overfitting it is possible, with enough regularization, that it may help learn good representations. Secondly, as I understand it a single projection is drawn at random for every component. While we know from Johnson-Lindenstrauss's Lemma that this will in average be a good strategy, how stable is the model to spurious projections ?
Regarding the scarification of the output matrix: the number of parameters is effectively reduced but the computational requirements are still the same. Did the authors try any explicit methods, maybe matrix factorization, to exploit the very specific sparse structure of $W_o$ and reduce the number of operations and parameters ?
Finally, while the results are undeniable, just observing the non-zero columns of $W_\text{in}$ is as I understand it a mostly ad hoc rule. It would have been interesting to empirically verify its validity by contrasting it with the results given by following equation 2.
Overall the paper presents an interesting model for inferring Granger causality. The authors clearly present their two main contributions and verify them using varied datasets.
The authors also made a clear and appreciated effort toward reproducibility by including all hyperparameters and implementation details as well as the code, including those of competing techniques.
The inclusion of the literature review in the appendix is also greatly appreciated.


In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress. In this work, we use a curriculum of progressively growing action spaces to accelerate learning. We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space. Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data, value estimates, and state representations from restricted action spaces to the full task. We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.

The paper presents a method of scaling up towards action spaces, that exhibit natural hierarchies (such as a controllable resolution of actions), throughout joint training of Q-functions over these. Authors notice, and exploit a few interesting properties, such as inequalities that emerge when action spaces form strict subsets that lead to nice parametrisation of policies in a differential way. The evaluation is performed in simple toy-ish tasks, and in micro-management problem in 5 scenarios in the game of SC2.
On a fundamental level, proposed method resembles that of Mix & Match, that authors discuss in the paper. In the M&M paper authors use the matching (distillation) of the policies, to ensure knowledge transfer, while in GAS, authors share information through said differential reparametrisation. Ablations provided imply that this part is indeed crucial (as with independent Qs, called "Sep-Q" learning flat-lines). The ablation testing the off-policy modification, seems a bit less conclusive, despite authors claiming that "This ablation performs slightly, or considerably, worse in each scenario. " We can see in Figure 3 that there were 5 experiments:
- in one [95z vs 50m] GAS works much better
- in two [80m v 80m, 80m v 85m] there seems to be no difference (in terms of longer term performance)
- in one [60m v 65m] GAS works slightly better
- in one [50h v 50h] GAS works slightly worse
This mixed bag of results would rather suggest that the offpolicy part is not the main contributing factor, and might require closer investigation to really understand which part of the system proposed brings the benefits. Could this ablation be also done on the toy-ish tasks from experiment 1? Given its simplicity it should be cheap enough to run these extra experiments (I am assuming the SC2 ones are quite expensive?)
Reviewer finds it hard to understand, given current description, how was M&M baseline adapted to the Q-learning setup? Was the distillation loss replaced with L2 one? Was the exact same architecture used for these experiments? How were the missing parent actions handled? Or did M&M experiments use an actor critic learning instead (which would make the comparison more about Q-learning vs Actor-Critic learning, than about methods of action space scaling). Methods section (mentioning entropy loss) looks like M&M was indeed trained with actor-critic, which would make the baseline hardly comparable. Adapting M&M strategy to Q-learning (or picking other baselines, that work on the same RL setup), in reviewer's opinion, is crucial for actual evaluation of author's contributions (given that this is the only baseline).
On a minor point - given how unique SC2 environment (and problem of unit micromanagement) is, would it be possible to provide baselines results also for the toy-ish experiment 1?
Overall, I recommend a weak rejection of the paper, and I am happy to revisit this evaluation given that authors address the above comments.


In this paper, a Gaussian conditional random field model for structured binary classification (GCRFBC) is proposed. The model is applicable to classification problems with undirected graphs, intractable for standard classification CRFs. The model representation of GCRFBC is extended by latent variables which yield some appealing properties. Thanks to the GCRF latent structure, the model becomes tractable, efficient, and open to improvements previously applied to GCRF regression. Two different forms of the algorithm are presented: GCRFBCb (GCRGBC -Bayesian) and GCRFBCnb (GCRFBC -non-Bayesian). The extended method of local variational approximation of sigmoid function is used for solving empirical Bayes in GCRFBCb variant, whereas MAP value of latent variables is the basis for learning and inference in the GCRFBCnb variant. The inference in GCRFBCb is solved by Newton-Cotes formulas for one-dimensional integration. Both models are evaluated on synthetic data and real-world data. It was shown that both models achieve better prediction performance than relevant baselines. Advantages and disadvantages of the proposed models are discussed.

The work involves modifiying gaussian conditional random fields to work for classification problems instead of regression problems. The main idea is to apply a bernoulli distribution on top of the regression values to convert them to work with binary classification problems. Two variations are discussed along with the inference and learning methodology. The inference can be done using numerical approximation and learning using variational methods and is still untracktable. Comparisons with other modeling strategies is done using experiments.
The paper is incremental and doesn't really provide improvements to learning parameters (or at least there is no theory showing this in the paper). The experiments do not seem satisfactory as discussed below.
a) Applying a bernoulli distribution on the output of the GCRF seems trivial. It is not very clear when the GCRFBCb model would be better than the GCRFBCnb. The learning procedure is untracktable and hard to follow on why this might provide better results.
b) The datasets (music classification and gene classification) don't seem to be good datasets for structured predictions i.e. the interaction needed between the nodes is not clear. Since they are multilabel problems, one could have just modeled the system with N independent nodes or design a multinomial distribution instead of only for binary classification.
c) There should be more thorough fine-tuning of other models, for e.g. in the ski lifts experiment, the CRF does much worse than logistic regression in the results. This is most likely because the parameters were not initialized properly using normal tricks like using logistic regression. Typically for truly structured problems, CRFs do better than their logistic regression counter parts. It is also not clear how the other models (CRF and SSVM) pairwise potentials were modeled.
It would really help to make this paper stronger by showing the new modeling technique does better than CRFs (that are tuned properly) on better structured datasets. It would be good to have a discussion on when this model would do worse than the other structured models and why.


Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data. However, the primary focus has been on handling simple undirected graphs. Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it. Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only. In this paper, we propose COMPGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph. COMPGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations. It also generalizes several of the existing multi-relational GCN methods. We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results. We make the source code of COMPGCN available to foster reproducible research. However, most of the existing research on GCNs (Kipf & Welling, 2016;Hamilton et al., 2017;Veličković et al., 2018) have focused on learning representations of nodes in simple undirected graphs. A more general and pervasive class of graphs are multi-relational graphs 1 . A notable example of such graphs is knowledge graphs. Most of the existing GCN based approaches for handling relational graphs (Marcheggiani & Titov, 2017;Schlichtkrull et al., 2017) suffer from overparameterization and are limited to learning only node representations. Hence, such methods are not directly applicable for tasks that require relation embedding vectors such as link prediction. Initial attempts at learning representations for relations in graphs (Monti et al., 2018;Beck et al., 2018) have shown some performance gains on tasks like node classification and neural machine translation.

In this paper, the authors developed GCN on multi-relational graphs and proposed CompGCN. In comparison with existing multi-relational GCN, CompGCN leverages insights from knowledge graph embedding and learns representations of both nodes and relations, with the aim to alleviate the problem of over-parameterization. Moreover, to improve the scalability w.r.t. the number of relations, the initial relation representations are expressed as a linear combination of a fixed number of basis vectors. In contrast to existing works, the basis vectors are only defined for initialization but not for every GCN layer. The authors also compared the proposed CompGCN with other existing GCN variants and summarized the relationships between CompGCN and other models. In the experiments, three tasks including link prediction, node classification and graph classification were performed to evaluate the performance of the proposed method. By comparing with existing methods, the effectiveness of the proposed method was demonstrated.
There are several concerns on this paper. First, this work is somewhat incremental on R-GCN ("Modeling relational data with graph convolutional networks"). Although the differences were highlighted, the overall technical contribution is limited. In this paper, relation representations are learned jointly with node representations, but it is not clear on what are the benefits by doing so, or how significant the over-parameterization problem can be alleviated in this manner. Second, the experimental setup is unclear. As for the datasets, some more descriptions about the meaning of relationships and labels should be provided. As for the three different tasks, what loss functions were used for model training, how to obtain graph-level representations from node-level representations for graph classification, should be clarified. Third, in Table 2, there are some missing values of some compared methods, but it is not clear on their absence. Finally, from the results of Figure 3, it is hard to see the scalability of the proposed method w.r.t. the increasing number of relations since the variable is the number of basis. It is suggested to provide some more results to show the insights of advantages of the proposed method, such as the accuracy change w.r.t. increasing number of relations and the corresponding running time, in comparison with other methods.


ABSTRACT (Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly trainprune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping, low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7× energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training.

This paper presents a method to speed up training of deep neural networks. The main contribution is a method to quickly identify winning lottery tickets (denoted early-bird, or EB by the authors), without running the model to convergence. The authors present interesting preliminary experiments that motivate their method, and show that it works on two image recognition datasets using two models.
This paper addresses an under-explored, but very important problem in AI: the increasing cost of training models. The authors present interesting evidence about the potential to detect EBs early on. The experiments presented in Figures 1 and 3 are convincing and will be of interest to the community. The proposed method seems to work, at least on the setups explored by the authors. I am leaning towards acceptance, but am concerned with the following:
1. The authors experiment with a limited set of datasets (CIFAR-10 is a relatively easy task), and with a set of non-competitive baselines (SOTA for CIFAR-10/100 is 99%/91.3%, see https://benchmarks.ai/cifar-10{,0}). I would have liked to see whether the proposed method translates to harder datasets and stronger models.
2. I might be missing something here, but to the best of my understanding the large learning rate part (page 4) does not demonstrate the benefits of increasing the learning rate, but the problems with *decreasing* it. The two might seem like the same thing, but in fact they're not: the authors claim the [80,120] policy is standard, and use it when training the subnetwork, so showing that [0,100] is inferior does not present a way to improve over the current approach, but evidence that the other approaches are inferior.
Other questions:
1. In Figure 1, it seems that the extracted subnetworks are doing very well even after 0 epochs. Does this mean that a trained version of a random subnetwork could reach within 1-2 points of the unpruned model? or is it pruned after training for 1 epoch?
2. If I understand correctly, Figure 5 should be illustrating the proposed method, which automatically identifies the early stopping point. In that case, I am not sure why the plot is a function of the epoch.
3. Do the authors have any intuition as to the sharp decrease in the 70% graph in Figures 1 and 2 around epoch 50?
Writing:
1. The language used by the authors is sometimes exaggerated. Expressions such as "bold guess" (section 3.2), "innovative ... scheme" (section 4) and comparisons to Winston Churchill would be better left out of the manuscript.
2. Typos and such:
-- several across the paper. For instance:
- Intro: After *bring* identified (should be "being")
- Related work: when training *it* isolation (in)
-- Missing venue for Frankle and Corbin (2019)


Many practical robot locomotion tasks require agents to use control policies that can be parameterized by goals. Popular deep reinforcement learning approaches in this direction involve learning goal-conditioned policies or value functions, or Inverse Dynamics Models (IDMs). IDMs map an agent's current state and desired goal to the required actions. We show that the key to achieving good performance with IDMs lies in learning the information shared between equivalent experiences, so that they can be generalized to unseen scenarios. We design a training process that guides the learning of latent representations to encode this shared information. Using a limited number of environment interactions, our agent is able to efficiently navigate to arbitrary points in the goal space. We demonstrate the effectiveness of our approach in high-dimensional locomotion environments such as the Mujoco Ant, PyBullet Humanoid, and PyBullet Minitaur. We provide quantitative and qualitative results to show that our method clearly outperforms competing baseline approaches.

The paper proposes a method for exploiting structure in locomotive tasks for efficiently learning low-level control policies that pass through waypoints while achieving some goal (typically 3D Cartesian position). This is in contrast to goal-conditioned RL policies that sample random goals during training and are thus sample inefficient, which are trained to execute one policy at a time. In particular, the paper proposes the notion of generalized experiences, where new trajectories are generated from existing trajectories, in such as a way that they are equivalent to each other (in this case translation and orientation invariant) with respect to actions.
The idea proposed here, of exploiting environmental structures in order better generalize previous knowledge to new, unseen situations is an interesting direction for achieving sample efficiency in practical RL, such as in robotics.
I have the following comments/questions.
1. If I understand correctly, rather than randomly sampling the environment, the paper proposes starting off with trajectories generated while learning some single-goal policy, and generate from these new ones that the agent must execute in the environment. In which case, the question is how much less interaction does the agent have with the environment compared to random goal sampling, to achieve the same performance?
2. What happens if you use a standard goal-conditioned RL with the Generalized Experiences, without training an IDM? For example, using VGCP with GE, where the goals for VGCP are teminating states of each trajectory. In other words, can a vanilla goal-conditioned RL benefit from the proposed trajectory sampling technique (GE), and how does that compare with random sampling and using the proposed latent representation technique?
3. How do you modify the LFM so that it only accepts the current state and goal, instead of a set of two e.m.o equivalent states and goals? Is the same query treated as two queries that are e.m.o equivalent?
4. In A.1.2, the paper mentions that the first half of the data generated using the RL algorithm is used for generating the second half of data for GE and LR, right? Which means GE and LR are trained from the same number of steps as the baselines, except that GE and LR make use of generalized experiences. However, the paper states in A2 that GE and LR learn from inferior samples. Can the authors please clarify this? My understanding is that GE are generated by modifying existing trajectories and letting the agent apply the same actions by interating with environment. Meaning that, given 2M samples, GE will generate 2M more samples that are e.m.o equivalent by interacting with the environment.
5. Some of the references in the text do not have year of publication, e.g., Kulkarni et al.


Producing agents that can generalize to a wide range of visually different environments is a significant challenge in reinforcement learning. One method for overcoming this issue is visual domain randomization, whereby at the start of each training episode some visual aspects of the environment are randomized so that the agent is exposed to many possible variations. However, domain randomization is highly inefficient and may lead to policies with high variance across domains. Instead, we formalize the visual domain randomization problem, and show that minimizing the policy's Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies. We propose a regularization method where the agent is only trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant. We conduct experiments that demonstrate that our technique leads to more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.

This paper proposes a regularization scheme for training vision-based control policies that are robust to variations of the visual input, which the paper classifies as "visual randomization". Visual randomization, as the paper notes, is one of the state-of-the-art techniques proposed from simulation to real robot transfer of vision-based policies trained on synthetic images. The regularization proposed in the paper aims to produce policies that rely on features that are invariant to the visual randomization, so that the resulting behaviour of the agent is consistent across different variations of its visual input for the same states.
The paper proposes that forcing a policy trained under randomization to have a Lipschitz constant of K, over the randomization parameters, causes the optimal policy to be similar across randomization parameters, with the difference in expected returns of two randomized environments being bounded by K times the maximum difference in their randomization parameters.
I recommend this paper to not be accepted until the following issues are addressed.
* There are missing details from the experimental setup, which makes the results hard to interpret (see, below).
* There are  missing details on how vanilla domain randomization was implemented. Domain randomization aims to maximize the expected performance over the environment distribution. This can be implemented properly by computing the expected gradients with data from more than one environment. From the algorithm descriptions in the appendix, it is not clear that this is how vanilla domain randomization was implemented.
* The title, introduction and conclusions do not reflect the scope of the paper. The paper only addresses situations where the same behaviour is achievable on all environments, an assumption (Mehta et al, 2019) also makes, and its proposed regularization is based on the assumption that the optimal behaviour is achievable with the same policy on all environments. But this is not true in general: for dynamics randomization, different environments may require different policies (e.g. driving a car on a road vs driving off-road). The regularization method may result in overly conservative policies i such situations.
Questions about experimental details:
What are the maximum returns  for Cartpole when trained until convergence without randomization? (175? 200? 1000?) If the maximum returns are higher than 175, how does Figure 4 look with more data? This is crucial to understand, for example, the results in Figure 11. That figure shows the proposed regularization slightly hinders the performance for the environments near l=1, g=50 (that region is a darker shade of green on the left subfigure). How do we know if the task has been successfully solved in the green vs purple regions? In all experiments, are the training curves showing the performance of the policies over the same environments (same seeds)? If not, how are the training curves comparable?
Other things to improve:
The conclusions of this paper can be made stronger by adding a comparison with EpOpt-PPO (i.e. optimizing the worst case performance over a set of trajectories sampled from multiple environments)


Intriguing empirical evidence exists that deep learning can work well with exotic schedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN (Ioffe & Szegedy, 2015), which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN (Ioffe & Szegedy, 2015; Santurkar  et al., 2018;Arora et al., 2018) • Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some (1 + α) factor in every epoch for some α > 0. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the network stays wellbehaved due to normalization. • A worked-out toy example illustrating the above linkage of hyperparameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.

UPDATE TO MY EARLIER REVIEW
===========================
The authors improved the writing of the paper substantially relative to the first version they submitted, and fixed minor issues. I am changing my rating from "Weak accept" to "Strong accept". It is now a must-read paper for anyone doing research on deep learning.
MY EARLIER REVIEW
=================
This exciting and insightful paper presents theorems (and illustrating examples and experiments) describing an equivalence of commonly used learning rate schedules and weight decay settings with an exponentially increasing learning rate schedule and no weight decay, for neural networks with scale-invariant weights. Hence, the results apply to a large set of commonly employed settings. The paper contains an interesting example of a neural network for which gradient descent converges if with batch normalization as well as with L2 regularization, but not when both are used.
From a theory viewpoint, the paper offers new insights into Batch normalization and other normalization schemes. From a practical viewpoint, the paper suggests a way to speed up hyper-parameter search, effectively allowing to consider learning rate and weight decay as one parameter.
A small gripe: this paper is a bit rough around the edges, and reads a bit like a draft (see comments on details below).
Detailed Comments / advice / questions
==================================
- It often takes a bit of searching to figure out what proof goes with what theorem / fact. I recommend to add to each occurrence of “Proof.” (before a proof) with a reference to the theorem or fact that is being proved, e.g. “Proof of Theorem 2.6”.
- The authors state that theorem B2 applies to “general deep nets”. In this theorem, the limit R_\infty could very well be zero (e.g. for networks with weights that are not scale-invariant), in which case the statement contains a division by zero. I wonder if the authors overlooked this or forgot to state an assumption in the theorem. Maybe I am missing something. Since this theorem does not appear to be critical to the main contributions of the paper, it may be easiest to remove the theorem if the division by zero is indeed a problem.
- On page 6, if c_w(x) is independent of w, would c(x) be more suitable?
- At the bottom of page 6, the authors state that “As a result, for any fixed learning rate, ||w_{t+1}|| ^4= …”. It appears that the authors get the expression for  ||w_{t+1}||^4 from the expression for ||w_{t+1}||^2 above (maybe by multiplying by ||w_{t+1}||^2?), but I don’t see how. Could the authors explain this? Maybe authors mixed up w_t and w_{t+1} by accident?
- The authors claim to “constrict better Exponential learning rate schedules”. Since the authors only perform a limited evaluation of their proposed learning rate schedule on CIFAR10, I suggest qualifying this statement.
- Theorem 1.1 does not introduce what \tilde{\gamma} is. It’s somewhat obvious, but I would state it nonetheless.
- The authors state that “…the exponent trumps the effect of initial lr very fast, which serves as another explanation of the standard wisdom that initial lr is unimportant when training with BN”. I don’t think that this constitutes a full explanation without further argument. I am also wondering if “another” is appropriate here: I am not aware of any (other) mathematically precise explanations of why initial learning rates do not matter in this set-up. If the authors are, they should cite it.
- The authors often forgot spaces before \cite commands. If you are trying to avoid a line break before the \cite command, you can use a tilde ~ , like this “Group Normalization~\cite{somepaper}”.
- Please introduce the abbreviation LR for “Learning Rate”, and always use the all-upper-case version (not “lr”).
- Definition 1.2 has a broken \cite .
- Theorem 2.7 should introduce \hat{eta}_t, but it doesn’t.
- Appendix A.1 contains a broken sentence (“as a function of…”)
- It’s odd that the proof for theorems B1 and B2 appear before theorems B1 and B2. I would restructure the appendices to improve this.
- Theorem B2 contains a stray “when”, and “exists” should be “exist”
- What the authors call “Proof of Theorem 2.4” in the appendix is really a proof of the rigorous version of Theorem 2.4 - Theorem 2.7! The proof should in my opinion be labeled “Proof of Theorem 2.7”
- The typo “eventsequation” should be replaced with something like “the events from equations”
- Replace the colloquialism “nets” with “networks”.
- Replace “BatchNorm” with “Batch Normalization”
- “COvariate” has casing issues
- “Riemmanian” should be “Riemannian”
- “BNhas” should be “BN has”
- The paper’s title states that the results are for batch-normalized networks, while the analysis appears to be more generally for networks with scale-invariant weights, which as the authors point out can arise from mechanisms other than batch normalization. Have the authors considered changing the paper’s title to better capture what their work applies to? In terms of discoverability, the authors would do the community a service by titling the paper in such a way that it captures the set-up well.


Many machine learning problems reduce to the problem of minimizing an expected risk. Incremental gradient (IG) methods, such as stochastic gradient descent and its variants, have been successfully used to train the largest of machine learning models. IG methods, however, are in general slow to converge and sensitive to stepsize choices. Therefore, much work has focused on speeding them up by reducing the variance of the estimated gradient or choosing better stepsizes. An alternative strategy would be to select a carefully chosen subset of training data, train only on that subset, and hence speed up optimization. However, it remains an open question how to achieve this, both theoretically as well as practically, while not compromising on the quality of the final model. Here we develop CRAIG, a method for selecting a weighted subset (or coreset) of training data in order to speed up IG methods. We prove that by greedily selecting a subset S of training data that minimizes the upper-bound on the estimation error of the full gradient, running IG on this subset will converge to the (near)optimal solution in the same number of epochs as running IG on the full data. But because at each epoch the gradients are computed only on the subset S, we obtain a speedup that is inversely proportional to the size of S. Our subset selection algorithm is fully general and can be applied to most IG methods. We further demonstrate practical effectiveness of our algorithm, CRAIG, through an extensive set of experiments on several applications, including logistic regression and deep neural networks. Experiments show that CRAIG, while achieving practically the same loss, speeds up IG methods by up to 10x for convex and 3x for non-convex (deep learning) problems.

This paper proposes a novel extension to SGD/incremental gradient methods called CRAIG. The algorithm selects a subset of datapoints to approximate the training loss at the beginning of each epoch in order to reduce the total amount of time necessary to solve the empirical risk minimization problem. In particular, the algorithm formulates a submodular optimization problem based on the intuition that the gradient of the problem on the selected subset approximates the gradient of the true training loss up to some tolerance. Each datapoint in the subset is a medoid and assigned a weight corresponding to the number of datapoints in the full set that are assigned to that particular datapoint. A greedy algorithm is employed to approximately solve the subproblem. Theory is proven based on based on an incremental subgradient method with errors. Experiments demonstrate significant savings in time for training both logistic regression and small neural networks.
Strengths:
The proposed idea is novel and intriguing, utilizing tools from combinatorial optimization to select an appropriate subset for approximating the training loss. Based on the experiments provided in the paper, it does appear to yield a significant speedup in training time. It is interesting to observe how the order of the datapoints matter significantly for training, and that CRAIG is also able to naturally define a good ordering of the datapoints for SG training. This is strong algorithmic work.
Weaknesses:
Some questions I had about the work:
- How well does one have to approximate $d_{ij}$ in order for the method to be effective? The authors provide an approach to approximate this for both logistic regression and neural networks. How does one guarantee that one is obtaining the maximum over $x in \mathcal{X}$ for neural networks via backpropagating only on the last layer? Does taking this maximum matter?
- How does one choose $\epsilon$? Is this related to how $d_{ij}$’s are approximated?
- If one were to consider an algorithm that samples points from this new distribution over the data given by CRAIG, if one were to include the weight $\gamma_j$ into the algorithm, would the sample gradient be unbiased? What if one were to simply use $\gamma_j$ to weight that particular sample in the new distribution?
- In machine learning, the empirical risk (finite-sum) minimization problem is an approximation to the true expected risk minimization problem. What is the effect of CRAIG on the expected risk? Is there any deterioration in generalization performance?
- In page 4, what does the $\min_{S \subseteq V}$ refer to? Should the equation be interpreted as with the set $S$ fixed or not?
- Theorems 1 and 2 are stated a bit non-rigorously. Are these theorems for fixed $k$? What does it mean for these bounds that $k \rightarrow \infty$?
- In Theorems 1 and 2, what is the bound on the steplength in order to obtain the convergence result for $\tau = 0$?
- In Theorem 1 for $0 < \tau < 1$, why does one obtain a result where $\|x_k – x_*\|^2 \leq 2 \epsilon R / \mu$, why is the distance to the solution bounded by a constant? What if one were to initialize $x_0$ to be such that $\|x_0 – x_*\|^2 > 2 \epsilon R / \mu$? (Similar for Theorem 2.)
- In the experiments, how is the steplength and other hyperparameters tuned? Are multiple trials run?
- Is $\epsilon$ used to determine the subset or is it based on a predetermined subset size?
- How do the final test losses compare between CRAIG and the original algorithms?
- How do the relative distances (rather than the absolute distance) to the solution behave?
- How does CRAIG perform over multiple epochs? How does the algorithm transition when the subset is changed (as in neural networks)?
- Why does CovType appear more stable with the shuffled version over the other datasets? Is the stability related to the distribution of the weights $\gamma_j$?
Some grammatical errors/typos/formatting issues:
- Equation (9) needs more space between the $\forall x, i, j$ and the rest of the equation.
- What is $\Delta$ on page 5? Is it supposed to be $F$?
- On page 8, And should not be capitalized.
- Page 14, prove not proof
- Page 14, subtracting not subtracking
- Page 16, cycle not cycke
Overall, although I like the ideas in the paper, the paper still needs some significant amount of refining in terms of both writing and theory, as well as some additional experiments to be convincing. If the comments I made above were addressed, I would be open to changing my decision.


We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure? This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation. While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier. These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than naïve adaptation as well as learning from scratch. Building on this intuition, we propose risk-averse domain adaptation (RADA). RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics. Then, when dropped into a new environment, it employs a pessimistic exploration policy, selecting actions that have the best worst-case performance as forecasted by the probabilistic model. We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes. We compare our approach against other approaches for adapting to new environments, including meta-reinforcement learning.

This paper proposes to adapt RL agents from some set of training environments (which, in the current instantiation, vary in some simple respect) to a new domain. They build on a framework for model-based RL called PETS.
The approach goes as follows:
2-step process
 * train probabilistic model-based RL agents in a “population of source domains”
 * dropped into new environment use “pessimistic exploration policy”
Then at test time, in order to compute estimates for the rewards for each action the authors use a “particle propagation” technique for unrolling through their dynamics model .
The action is chosen by looking at the sum of the 0 through kth percentile rewards.
This is a weird choice. Why are they looking at a sum over quantiles vs a quantile itself?
The claim is that the models from the first stage capture the epistemic uncertainty due to not knowing z.
However, the authors give a too scant a treatment of what these uncertainty estimates really mean.
For example, they appear to only be valid with respect to an assumed distribution over z.
The paper’s experiments however focus in large part on what happens when the model is evaluated
on values of z that were outside the support of the distribution over training domains.
In this case, any benefit appears to be ill explained by the underlying motivation.
The next step here is to finetune the model as data is collected on the new domain.
Authors propose heuristics for this finetuning that include
1. Drawing experiences from the past experiences (under different domains) and
2. “keeping the model close to the original model”, via some sort of regularization presumably.
>>> 	why isn’t the exact nature of how they “keep the model near the original model explained in the text?
	perhaps the authors mean that 1. and 2. are one and the same (1 as  means to achieve 2)
	if this is the case, then the exposition should be improved to make this more clear.
Some important details appear to be missing. For example, how many distinct source domains are seen during pretraining? Do they set z different z for every single episode of pretraining? Some language here is unclear, for example what precisely does an “iteration” mean in the context of the experiments?
The choice to report “average maximum reward” seems strange if what the authors care about is avoiding risk. Can they explain/justify this choice or if not, present a much more comprehensive set of experimental results?
The figures tracking catastrophic failures vs performance resembles those in
“Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear”  https://arxiv.org/abs/1611.01211
This raises some question about why they don’t if concerned with “catastrophic events” model them more explicitly.
Else, if the return accurately captures all desiderata, why to we need to count the failures?
In short this is a simpzle empirical paper that makes use of heuristic uncertainty estimates,
including in settings when the estimates have no validity. The writing is reasonably clear
and the ideas are straightforward (which is perfectly fine!). A few of the decisions are unnatural,
a few are ad hoc, and a few details are missing. Overall my sense is that this paper
has some good qualitities, including the clarity of much of the exposition,
but it’s still below the mark to be an impactful ICLR paper.
==========UPDATE=================
I read the rebuttal and am glad that the authors took time to read my review and engage with the criticism as well as try to make some small improvements to the paper, especially exploring the impact on the number of training environments on the results (in the original paper the number of environments available at train time was unlimited). The answers to some of the other questions were less convincing. E.g. the seemingly incoherent objective of summing over the quantiles falls flat. Why should we care more about being a "strict generalization" of some previous algorithm built upon than of having a coherent objective? Overall, I don't think the paper makes it over the bar to accept but I hope the authors continue to improve upon the work and get it into shape where it could be accepted at another strong conference.


In this paper we present a model for unsupervised topic discovery in texts corpora. The proposed model uses documents, words, and topics lookup table embedding as neural network model parameters to build probabilities of words given topics, and probabilities of topics given documents. These probabilities are used to recover by marginalization probabilities of words given documents. For very large corpora where the number of documents can be in the order of billions, using a neural auto-encoder based document embedding is more scalable then using a lookup table embedding as classically done. We thus extended the lookup based document embedding model to continuous auto-encoder based model. Our models are trained using probabilistic latent semantic analysis (PLSA) assumptions. We evaluated our models on six datasets with a rich variety of contents. Conducted experiments demonstrate that the proposed neural topic models are very effective in capturing relevant topics. Furthermore, considering perplexity metric, conducted evaluation benchmarks show that our topic models outperform latent Dirichlet allocation (LDA) model which is classically used to address topic discovery tasks.

This paper proposes a neural topic model that aim to discover topics by minimizing a version of the PLSA loss. According to PLSA, a document is presented as a mixture of topics, while a topic is a probability distribution over words, with documents and words assumed independent given topics. Thanks to this assumption, each of these probability distributions (word|topic, topic|document, and word|document) can essentially be expressed as a matrix multiplication of the other two, and EM is usually adopted for the optimization. This paper proposes to embed these relationships in a neural network and then optimize the model using SGD.
I believe the paper should be rejected because: 1) most aspects of this paper are a little dated 2) novelty is little 3) experimental section is very limited and unconvincing.
To elaborate on the experimental section:
- Only LDA has been presented as baseline. There's plenty of neural topic models to compare against (you mentioned some in your related work section) but no comparison with any of those is presented. If the concern is their training time on large datasets, they should be at least presented as comparison for the smaller datasets. For the large datasets there's other approaches that would scale and should be presented as baselines: 1) train on a sample of the dataset 2) co-occurrence based topic methods on sliding windows of text are extremely fast (eg see "A Biterm Topic Model", "A Practical Algorithm for Topic Modeling with Provable Guarantees", and "A Reduction for Efficient LDA Topic Reconstruction" which could fit your scenario with large datasets where topics most likely have small overlap with each other and are almost separable by anchor words.)
- Even regarding just LDA: what hyper-parameters \alpha and \beta did you set for LDA? Tuning \beta to a small value might have an impact for large datasets.
- Metrics: only perplexity is presented and metrics but it's well known that perplexity on its own is quite limited and often is not correlated to human judgment. Consider adding topic coherence measures as well.
- The section on continuous document embeddings is confusing and the explanation should be improved and the formalism tightened.
Other (did not impact the score):
- Biases: you're adding biases to your probability estimation equations. This is not in line  with the PLSA assumption. What happens if no biases are used?
The paper has several typos and grammatical errors, e.g.:
- page 2, L#1: networks -> network
- page 4, sec 3.2: set unobserved -> set of unobserved
- page 5, sec 5: pratise -> practice
- several places: it's -> its


Recently various neural networks have been proposed for irregularly structured data such as graphs and manifolds. To our knowledge, all existing graph networks have discrete depth. Inspired by neural ordinary differential equation (NODE)   for data in the Euclidean domain, we extend the idea of continuous-depth models to graphs, and propose graph ordinary differential equation (GODE). The derivative of hidden node states are parameterized with a graph neural network, and the output states are the solution to this ordinary differential equation. Noticing that NODEs are typically trained with the adjoint method, with the advantages of adaptive evaluation and a free-form continuous invertible model; however, their performance on benchmark image classification tasks is significantly inferior to discrete-layer models. We show the reason is adjoint method generates inaccurate gradient due to numerical error in reverse-mode integration. We then propose a memory-efficient framework for free-form ODEs with accurate gradient estimation, which is fundamental to deep-learning models. Furthermore, when applied to invertible blocks, our method achieves constant memory cost with GODE (NODE). We show our method for free-form ODEs generalizes to various model structures and achieves high accuracy for both NODE and GODE in benchmark tasks.

Summary: This paper proposed a deep model called graph-ODE (GODE), which extends the continuous deep model, neural ODE [1], to graph structured data. Two methods of computing the gradient are proposed, one is the adjoint method proposed in [1] which is memory-efficient but not perfectly accurate, and the other method is to discretize the ODE and compute an accurate gradient, which is also memory-efficient but relies on an invertible architecture.
In general, this paper contains many interesting thoughts, but to me, it seems these ideas are not new and have been proposed before[1,2,3,4]. I could not find a strong contribution of this paper.
- - - about the ODE stability issue
Among invertible deep models, the advantage of ODE-based continuous models [1] is: there is *no restriction* on the form of f. The drawback is the computed gradient has an error, depending on discretization step size and stability. This paper pointed out the stability problem (which is also discussed in [2,3]), but do not provide a solution in the domain of continuous deep models.
Instead, the solution they provided is to use a discretized version and compute the gradient accurately. Then it becomes a standard invertible DL model with discrete layers, where the invertible building blocks have a specific *restricted form*. The 'adjoint method with discrete-time' in Eq (10) is the same as the chain rule, which has also be pointed out in [4]. To this point, I think GODE is in the class of discrete invertible DL models trained by gradient descent. I think it less related to continuous models, except the step size can be adaptive in the forward pass.
- - - about the invertible building block
The proposed invertible building block replaces 'sum' in [5] by a function \psi. This is not novel enough to serve as a contribution.
- - - comparison to graph neural network
I think it is interesting to apply ODE techniques or other invertible architectures to graph-structured data, for which I didn't see similar works before and could be a contribution of this paper. However, for the experimental results shown in table 3 and 4, the improvement is really small. A stronger result is needed to demonstrate the advantages.
[1] Chen, Tian Qi, et al. "Neural ordinary differential equations." Advances in neural information processing systems. 2018.
[2] Chang, Bo, et al. "Reversible architectures for arbitrarily deep residual neural networks." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[3] Behrmann, Jens, et al. "Invertible Residual Networks." International Conference on Machine Learning. 2019.
[4] Li, Qianxiao, et al. "Maximum principle based algorithms for deep learning." The Journal of Machine Learning Research 18.1 (2017): 5998-6026.
[5] Gomez, Aidan N., et al. "The reversible residual network: Backpropagation without storing activations." Advances in neural information processing systems. 2017.
-------------after reading the response
I'd like to thank the authors for their explanations.
However, the authors' explanation of the benefit of transforming the invertible building block into an ODE model is still not convincing to me.
The authors explained that ODE solutions can represent highly nonlinear functions. However, discrete NN also represents highly nonlinear functions (e.g. with Relu activation, they are *piecewise* linear, but they are highly nonlinear)! From my point of view, their difference is that ODE model is more smooth. However, the benefit of using a smoother model is still unclear. For the example that the authors provided, $\sin x$, why being able to represent that kind of function is an advantage for the graph classification problem? Why is this a good model bias? I think the authors' responses are still not convincing enough, so I choose to retain the score.


Sample efficiency has been one of the major challenges for deep reinforcement learning. Non-parametric episodic control has been proposed to speed up parametric reinforcement learning by rapidly latching on previously successful policies. However, previous work on episodic reinforcement learning neglects the relationship between states and only stored the experiences as unrelated items. To improve sample efficiency of reinforcement learning, we propose a novel framework, called Episodic Reinforcement Learning with Associative Memory (ER-LAM), which associates related experience trajectories to enable reasoning effective strategies. We build a graph on top of states in memory based on state transitions and develop an efficient reverse-trajectory propagation strategy to allow rapid value propagation through the graph. We use the non-parametric associative memory as early guidance for a parametric reinforcement learning model. Results on Atari games show that our framework has significantly higher sample efficiency and outperforms state-of-the-art episodic reinforcement learning models.

The paper proposes a new method for organizing episodic memory in with deep Q-networks. It organizes the memory as a graph in which nodes are internal representations of observed states and edges link state transitions. Additionally, nodes from different episodes are merged into a single node if they represent the same state, allowing for inter-episode value propagation of stored rewards. The authors experimentally evaluate the method against reasonable baselines and show improved performance in the majority of tasks.
While the proposed method seemingly leads to better performance, the analysis of the results appears superficial. First, it lacks theoretical rigor to explain why the proposed propagation mechanism works, for instance, a proof of the optimal substructure in Eq. (3) would be helpful. While I see the authors mentioned previous work which do not have optimal substructures and mention this for the case of navigation-like tasks, the matter is not discussed and unclear in other scenarios.
This in itself would not be much of an issue if the experiments highlighted advantages and limitations of the proposed method, but that is not the case. For instance, per-task comparisons to EVA (mentioned in the paper) could indicate how useful inter-episode value propagation was in each task.  Table 1 mentions EVA, but only on aggregated results, thus not providing insight into this. Furthermore, the experiments selected to be detailed in Figure 4 are, in my opinion, suboptimal choices, as the worst-performing of the selected tasks is still better than the baselines. It would be more insightful to also compare those whose performance is close to the baselines, such as the Boxing environment, and significantly worse than baselines, such as the StarGunner environment (as shown in Figure 3). With those issues in mind, my conclusion is to recommend to reject the paper in the current state.
Minor comments:
- Thoroughly review the writing and grammar, the text in its current state needs significant improvement in this regard
- Equation references missing parentheses
- Introduction, 1st paragraph, second sentence is incomplete
- Algorithm 2: tuples taken from/appended to sets G and D are not consistent (cf. lines 11 and 13)
- Figure 3: The large amount of bars in the plot would benefit from horizontal lines across the plot for each tick on the y axis
- Table 1: caption before the table; should also state the meaning of numbers (percentage or normalized score, calculated from what?)
- Figure 4: labels (all text except plot titles) are impossible to read in print
- Section 3 could be placed before Section 2, laying the mathematical framework, and then following the discussion with related work
- There is redundant content in Sections 2 and 3
----
I am happy with the authors response and changed the score accordingly


The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains. Devising a definitive defense against such attacks is proven to be challenging, and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism. In this paper, we consider the adversarial detection problem under the robust optimization framework. We partition the input space into subspaces and train adversarial robust subspace detectors using asymmetrical adversarial training (AAT). The integration of the classifier and detectors presents a detection mechanism that provides a performance guarantee to the adversary it considered. We demonstrate that AAT promotes the learning of class-conditional distributions, which further gives rise to generative detection/classification approaches that are both robust and more interpretable. We provide comprehensive evaluations of the above methods, and demonstrate their competitive performances and compelling properties on adversarial detection and robust classification problems.

Review: The paper addresses the adversarial example detection problem. The framework proposed in the paper divides the input space into subspaces based on a classifier’s output and trains detectors on these subspaces to classify a natural sample (classified as that class) from an adversarial one fooling the network. The goal is to use a robust optimization approach to enable detection methods to withstand adaptive/dynamic attacks. Hence, an asymmetrical adversarial training (AAT) regime is employed which presents solving a min-max problem. AAT supports the detectors to learn class conditional distributions, motivating generative detection/classification approaches. There are three different attacking scenarios and evaluation shows that the combined attack turns out to be most effective (as it fools both the classifier and detectors) against integrated detection. The paper also demonstrates empirical improvements over state of the art detection techniques with higher L2 distortion of perturbed samples.
Pros:
- With the vulnerabilities associated with neural networks, the motivation behind building defense mechanisms against adversarial attacks has been well-justified.
- Most of the evaluation metrics look appropriate and well-defined.
- It was interesting to observe the perturbed samples produced by attacking generative classifier. While they exhibited visible features of the target class, these perturbations had to be different on a sematic level to be distinguished from the natural samples.
Cons:
- While the idea to partition into subspaces and learn a different detection for each of them is novel, it involves training multiple detectors one by one that can be computationally expensive. The loss function for different attacking scenarios uses the outputs from all the detectors, which can also be expensive, especially when there are a lot of classes.
- To deal with extremely unbalanced data sets when training the detector, the solutions used in the paper resamples to balanced the positive and negative class data. This would mean throwing off most of the data, I would see how it affects the training.


Discovering the underlying mathematical expressions describing a dataset is a core challenge for artificial intelligence. This is the problem of symbolic regression. Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are lacking. We propose a framework that combines deep learning with symbolic regression via a simple idea: use a large model to search the space of small models. More specifically, we use a recurrent neural network to emit a distribution over tractable mathematical expressions, and employ reinforcement learning to train the network to generate better-fitting expressions. Our algorithm significantly outperforms standard genetic programming-based symbolic regression in its ability to exactly recover symbolic expressions on a series of benchmark problems, both with and without added noise. More broadly, our contributions include a framework that can be applied to optimize hierarchical, variable-length objects under a black-box performance metric, with the ability to incorporate a priori constraints in situ.

This paper presents deep symbolic regression (DSR), which uses a recurrent neural network to learn a distribution over mathematical expressions and uses policy gradient to train the RNN for generating desired expressions given a set of points. The RNN model is used to sample expressions from the learned distribution, which are then instantiated into corresponding trees and evaluated on a dataset. The fitness on the dataset is used as the reward to train the RNN using policy gradient. In comparison to GP, the presented DSR approach recovers exact symbolic expressions in majority of the benchmarks.
Overall, this paper presents a novel technique of using an RNN to learn a distribution over mathematical expressions, and using the fitness of sampled expressions as the reward signal to train the RNN using policy gradient. The idea of using the parent and sibling nodes to predict expression nodes in an autoregressive fashion is also interesting, which exploits the fact that the operators are either binary or unary. The experimental results show that it outperforms genetic programming as well as the ablation study shows the usefulness of different extensions.
Grammar VAE (Kusner et al. 2017) learns a distribution over parse trees in a grammar and then uses Bayesian optimization to search over this space to perform symbolic regression. It would be important to empirically evaluate how GVAE performs on these symbolic regression tasks compared to DSR.
During the search for expressions using DSR, it wasn’t clear why the algorithm chose all constants to be 1 for the first 12 benchmarks. Is it because the RNN never chose the constants in the learnt distribution or the BFGS algorithm prefers constants to be 1? Also, could it be the case that GP is being unfairly penalized for such cases as it might be trying to learn real-valued constants. Would it be possible to report what expressions GP came up with for the first 12 benchmarks?
How was the Recovery metric calculated? Does it require exact syntactic match or it also allows for semantically equivalent expression (that might be different syntactically)?
What are the runtimes for the REINFORCE and GP methods? It wasn’t clear how big the total search space of expressions was without constants. How would a random search that enumerates over all expressions and uses BFGS to compute constants work?
The evaluation is only performed on 16 expressions and only 4 expressions have real-valued constants. It would be good to evaluate the technique on more benchmarks especially with real-valued constants, and larger expressions. Is Nguyen-4 already getting to maximum length of 30? Can the expressions from AI Feyman (Udrescu & Tegmark 2019) or synthetically generated expression be used?
From the experiment benchmarks, it seems only 4 expressions had real-valued constants, and for these benchmarks GP did quite well in terms of NRMSE. What expression is GP coming up with for these benchmarks?
Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobato. 2017. Grammar variational autoencoder. ICML 2017


Due to the lack of enough training data and high computational cost to train a deep neural network from scratch, transfer learning has been extensively used in many deep-neural-network-based applications. A commonly-used transfer learning approach involves taking a part of a pre-trained model, adding a few layers at the end, and re-training the new layers with a small dataset. This approach, while efficient and widely used, imposes a security vulnerability because the pre-trained model used in transfer learning are usually available publicly to everyone, including potential attackers. In this paper, we show that without any additional knowledge other than the pre-trained model, an attacker can launch an effective and efficient brute force attack that can craft instances of input to trigger each target class with high confidence. We assume that the attacker does not have access to any targetspecific information, including samples from target classes, re-trained model, and probabilities assigned by Softmax to each class, and thus called target-agnostic attack. These assumptions render all previous attacks impractical, to the best of our knowledge. To evaluate the proposed attack, we perform a set of experiments on face recognition and speech recognition tasks and show the effectiveness of the attack. Our work sheds light on a fundamental security challenge of the Softmax layer when used in transfer learning settings.

The paper proposed a method of attacking deep neural networks that were trained using transfer learning. The primary claim is that the proposed technique only requires knowledge of the base model (i.e., if the frozen parameters taken from pretrained VGG, ResNET models are known, then that is sufficient) and doesn’t require any samples of target classes of the post-transfer application to successfully attack the DNN.
Core idea is that Softmax layers used in classification can be exploited to craft a perturbation that will fool the network. The algorithm uses a brute force approach to iterate through each neuron on the final layer before the softmax. For each activated neuron, all other neurons are zeroed out and a regularized loss is computed between the activated feature and the neuron. The gradient of the loss function is then used to craft a perturbation using typical k-step gradient based method (as used in FGSM, PGD attacks). In other words, they are trying to craft a perturbation that will zero out all other class probability and put maximum weight on the target adversary class when applied to the network.
Experiments were conducted on VGG-Face and Pannous Speech dataset.
1. Paper is easy to follow and different empirical results show a few intuitive observations for a few different settings.
2. The main issue is that the algorithmic is rather simplistic and seems impractical. Most security applications that motivate the paper can easily defend themselves by simply limiting the number of attempts. Even if this core motivation issue is discarded, the algorithmic contribution is very minimal, no analytical understanding either.
3. Some comparison with Black Box models should be added. Although authors claim that Black Box models would require many queries, a specific comparison and contrasting with number of attempts and queries would help understand the critical advantages of the proposed technique.
4. Table 1 provides a good overview of the results but it should include standard deviations for experiments that were averaged.
5. Is there a notion of imperceptibility/attack budget here? Some of the attacks are clearly not imperceptible. This should be discussed.
6. Section 6 seems a little brief and can use more details. Authors claims that using EVM, “the model successfully defeat all our crafted images” but “there is 7.38% chance that the EVM-base model classifies the input as one of the target classes”. This statement is a little confusing.