This paper establishes rates of universal approximation for the shallow neural tangent kernel (NTK): network weights are only allowed microscopic changes from random initialization, which entails that activations are mostly unchanged, and the network is nearly equivalent to its linearization. Concretely, the paper has two main contributions: a generic scheme to approximate functions with the NTK by sampling from transport mappings between the initial weights and their desired values, and the construction of transport mappings via Fourier transforms. Regarding the first contribution, the proof scheme provides another perspective on how the NTK regime arises from rescaling: redundancy in the weights due to resampling allows individual weights to be scaled down. Regarding the second contribution, the most notable transport mapping asserts that roughly 1 /δ 10d nodes are sufficient to approximate continuous functions, where δ depends on the continuity properties of the target function. By contrast, nearly the same proof yields a bound of 1 /δ 2d for shallow ReLU networks; this gap suggests a tantalizing direction for future work, separating shallow ReLU networks and their linearization.

The paper studies approximation properties (in L2 over some data distribution P) of two-layer ReLU networks in the NTK setting, that is, where weights remain close to initialization and the model behaves like a kernel method given by its linearization around initialization.
The authors obtain a variety of results in order to obtain such approximation guarantees, which are obtained by sampling from a so-called 'transport mapping', which is essentially a function T:R^{d+1}->R^{d+1} with a bound on sup_w ||T(w)||, which can approximate well various classes of target functions (section 3).
In particular, they show that such a sampling leads to weights close to initialization, that the neural network function is close to its linearization in L2(P), and that the linearization is close to the target function in L2(P).
Together with a control of the norm of T required to approximate the target function, this leads to approximation bounds in Theorem 1.3.
The techniques used to obtain transport mappings are quite interesting and seem novel, and the general approach for controlling various steps from neural network function to the target function in L2(P) norm in the NTK setting is insightful and novel as far as I know.
That said, the presentation lacks a certain amount of polish in its present form, which makes me lean towards the reject side. I also have some comments related to novelty of certain aspects.
Comments:
* the paper is not well organized, with the main result appearing in the introduction with little details on the involved quantities, no clear separation or connection between intermediate lemmas in later sections, and very little motivation and explanation of some results. Further, there are many typos and inconsistent notations throughout which make the paper hard to read.
* the sampling result in Lemma 2.1 is very similar in flavor to random feature approximation results (for the NTK here), e.g., [1, Proposition 1], which could perhaps be more precise in practice as it is data-dependent, and only needs an L2 control on the function T. Can this be applied here or would the initialization term mess things up? A comparison would be helpful either way.
* the approximation rates should be discussed more (are they optimal?), and compared to prior work, both on general two-layer networks, and kernels arising from a similar setup, in particular in [2] and the cited Sun et al. (note that the NTK behaves similarly in terms of approximation, see [3])
* the section on the "natural RKHS" is largely unclear, as is the corresponding bound in Theorem 1.3 (shouldn't B be proportional to the RKHS norm?)
* how these results apply to networks obtained via optimization in the NTK regime should probably be discussed more
smaller things:
* eq (1.2): what is the mearning of the epsilon factor? is it standard?
* p.2 "to not yield" -> do not yield
* "with scaling... width": rephrase (and, do you mean dataset size?)
* "one 1/m is then pushed" -> one 1/sqrt(m)?
* throughout: pick a consistent notation for derivative of relu (sometimes it's sigma', sometimes an indicator)
* section 2: here it seems like T(w)/(eps sqrt(m)) is just the movement from initialization and T_m,eps(w) are the final weights, while in the introduction T(w) indicates the final weights, this is confusing notation. Also, shouldn't T_m,eps appear in the bounds?
* section 3: should -> shows?
* lemma 3.1, 3.2: specify that the other coordinates are 0, also missing dG(w) in the definition of g. What do we lose from the use of truncation?
* lemma 3.5: sup_x or just L2(P)?
* section 3.3: H is just L2(G) here? Also, the kernel is not universal with only even terms, but the bias fixes that, see e.g. [2,4].
[1] Bach. On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions (2017)
[2] Bach. Breaking the Curse of Dimensionality with Convex Neural Networks (2017)
[3] Bietti and Mairal. On the Inductive Bias of Neural Tangent Kernels (2019)
[4] Basri et al. The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies (2019)
===== update post rebuttal =====
Thanks for the detailed response, I am increasing my score as the new version looks much better.
I am a bit puzzled (and surprised) by the gap in the rates between NTK and relu random features, as it seems to suggest that only training second-layer weights while leaving the first layer at random initialization yields better rates than the NTK regime, if I understand correctly? If so, is this due mainly to the linearization step, i.e. Lemma 2.6? It would be good to include some further discussion on this in the paper.
As per approximation by random features/sampling, note that [1, Prop. 1] only requires a sup control on random features (which is quite trivial here with bounded data), not on the "transport" (beta in their statement).


It is well-known that classifiers are vulnerable to adversarial perturbations. To defend against adversarial perturbations, various certified robustness results have been derived. However, existing certified robustnesses are limited to top-1 predictions. In many real-world applications, top-k predictions are more relevant. In this work, we aim to derive certified robustness for top-k predictions. In particular, our certified robustness is based on randomized smoothing, which turns any classifier to a new classifier via adding noise to an input example. We adopt randomized smoothing because it is scalable to large-scale neural networks and applicable to any classifier. We derive a tight robustness in 2 norm for top-k predictions when using randomized smoothing with Gaussian noise. We find that generalizing the certified robustness from top-1 to top-k predictions faces significant technical challenges. We also empirically evaluate our method on CIFAR10 and ImageNet. For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62.8% when the 2 -norms of the adversarial perturbations are less than 0.5 (=127/255).

This paper builds upon the random smoothing technique for top-1 prediction proposed by Cohen et al. for certifying top-k predictions with probabilistic guarantees, which enjoys good scalability to large neural networks and in principle can be applied to any classifier.
- Contributions:
1. The authors aim to provide (probabilistic) certification on top-k predictions, which to my knowledge is the first work to consider this setup. Many applications such as recommendation systems indeed use top-k predictions as a performance measure. The problem setup is new and important in the research of robustness certification.
2. In terms of technical contributions, the authors identify the difficulty of extending top-1 prediction to top-k prediction, due to the requirement of simultaneous confidence interval estimation of the bounds on the actual class predictions. To cope with this difficulty, the authors proposed simultaneous confidence interval estimation based on Clopper-Pearson method and Bonferroni correction. However, I am not sure the difficulty is caused by the necessity of estimating multiple probability bounds, or simply the limitation of the proposed algorithm. I hope the authors can address my concerns in the Questions below.
3. Experimental results on Cifar-10 and ImageNet showed improved lower bound on certified L2-norm radius when increasing k. The authors also performed an ablation study of different parameters in the proposed algorithm.
- Questions:
1. Intuitively, when extending top-1 certification to top-k certification, one would expect using ordered statistics of the prediction outputs from the randomly perturbed inputs. As long as the original label's prediction probability is in the top-k label set, the smoothed classifier is directly certified. Instead of ordered statistics, the authors tackle this problem by considering estimating upper and lower bounds of each class prediction probability. Therefore, the problem becomes more difficult as k increases, since this indirect approach needs to simultaneous estimate those probability bounds. I wonder the current approach will be suboptimal when compared to the ordered statistics approach. I would like to know the authors thoughts on this regard. That is, is the claimed difficulty an outcome when using the proposed indirect bound estimation for certification, or it's provably more difficult?
2. The discussion on Fig.3 says "We observe that  \sigma controls a trade-off between normal accuracy under no attacks and robustness. Specifically, when  is larger, the accuracy under no attacks (i.e., the accuracy when radius is 0) is larger, but the certified top-k accuracy drops more quickly as the radius increases." However, it seems that larger \sigma actually gives lower accuracy under no attacks in Figure 3. Please clarify.
Overall, this paper brings some new insights and results in robustness certification, but some claims and statements need to be further justified. I am happy to increase my rating if my concerns are addressed.


We propose a feature selection algorithm for supervised learning inspired by the recently introduced knockoff framework for variable selection in statistical regression. While variable selection in statistics aims to distinguish between true and false predictors, feature selection in machine learning aims to reduce the dimensionality of the data while preserving the performance of the learning method. The knockoff framework has attracted significant interest due to its strong control of false discoveries while preserving predictive power. In contrast to the original approach and later variants that assume a given probabilistic model for the variables, our proposed approach relies on data-driven generative models that learn mappings from data space to a parametric space that characterizes the probability distribution of the data. Our approach requires only the availability of mappings from data space to a distribution in parametric space and from parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative models from machine learning. We provide example knockoff designs using a variational autoencoder and a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff designs and the knockoff framework with respect to existing approaches.

This paper proposes a way of employing modern generative models like VAEs and GPLVMs within the recently popular “knockoff” framework for variable/feature selection. Roughly speaking, the main idea of knockoffs is to construct a duplicate (“knockoff”) of each regression variable which matches its distributional properties but is independent of the response variable when conditioning on the true input variable the knockoff is based on. Under certain assumptions, fitting a model on top of both original and knocked-off features and observing the difference between the fitted coefficients for the true and the knockoff features can then be used for variable selection with guaranteed false discovery rate. The authors of this paper suggest that many of the current methods for construction of knockoffs may not be appropriate for image, text, and other types of datasets commonly used in modern ML, and suggest a heuristic way of employing VAEs and GPLVMs for this purpose. The paper concludes with an empirical study which shows that their algorithm is competitive with existing feature selections methods in terms of post-selection accuracy of the fitted model.
I am currently leaning towards recommending rejection. While the paper is nicely written and does a good job of reviewing knockoffs, I see two main issues: (1) I am not sure about applications for the proposed algorithm; in particular, the authors allude to use on devices with limited memory and computational power, but do not discuss why the Johnson-Lindenstrauss transform or some of the many low-precision implementations of neural networks (binarised neural networks, xnor-nets, …) cannot be used; furthermore, in the case of images, I am not sure why there is no comparison to simple downsampling to a smaller resolution; (2) The reported results do not show a reliable improvements over existing methods.
Major comments:
- I am not entirely sure lemma 1 is correct. In particular, mu_{z | x} tends to be a function of the whole vector x including x_n. For example, if (a, b) are jointly distributed according to a bivariate normal, then E (b | a) = E(b) + Sigma_{ab} Sigma_{aa}^{-1} (a - E(a)) where Sigma is the corresponding covariance matrix. Hence claiming that the marginal distribution z | x_{-n} is N ( mu_{z|x} , Sigma_{z | x} ) seems wrong as mu_{z | x} will generally depend on x_n (similarly to how E(b | a) depends on “a” above) which cannot be the case when x_n is marginalised. If z depends linearly on x, there is a standard expression for the distribution you seek. However, with the non-linear dependence employed, e.g., within VAE, working out a closed form expression may be quite a challenge. Can you please clarify or drop this result from your paper if it indeed turns out incorrect?
- Can you please clarify why don’t you benchmark against the cited algorithm proposed in Lu et al. (2018)?
- Can you please explain how was the latent dimensionality for the VAEs (5) and GPLVMs (10) selected? Also, for the algorithms where random seed plays a role (e.g., VAEs), how many random seeds were used (are the numbers reported in fig.2 a result of averaging over multiple seeds)? Relatedly, have you tested whether starting from different seeds results in the same subset of variables selected (e.g., when using VAEs)?
Minor comments:
- In the 1st sentence of the introduction, “prevalence” can be high or low, increase or decrease, etc. but “becoming increasingly pervasive” does not sound right. Please consider rewording.
- In fig.1e, do you know why GPLVM leads to such a significant mode collapse?
- Just after the 1st display on p.5, “SInce” -> “Since”


We study the problem of learning associative memory -a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast. Thus, most research in energybased memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.

*Summary:*
The authors propose to tackle the associative memory problem by recasting read/write operations to read/write by optimizing the parameters/input of an energy based model. Writing is reformulated as training a parametric energy model (EBMM) to have local minima of energy w.r.t. the parameters at memorized data points. Reading is performed by performing (projected) gradient descent on the corrupted/incomplete input to minimize the energy. To ensure the operations are fast (read and write with minimal gradient steps), the authors propose to take inspiration from modern meta-learning literature and learn initialization parameters of the energy model (and other hyperparameters for GD during read/write) from which writing is fast while ensuring reading is also fast, since the models are trained to maximize read/write performance within a constrained number of gradient steps.  Experimentally, the authors show that EBMM reading performs similar to baseline methods (but better across many memory sizes) on the standard Omniglot task. On CIFAR-10 and downsized ImageNet, they show much better L2 reconstruction error of corrupted images. They also show that the learnt energy
*Recommendation:*
I believe this is a very neat idea, and utilizes large parametric models for "smart" overfitting and compression of data for the associative memory task. The proposed meta-learning approach to training the model seems to perform well across multiple simple and challenging datasets, and therefore I would recommend accept. My current recommendation is very borderline (weak accept) because of a lack of some experimental rigour (which I would love clarifications on), and missing related work, which I mention below.
*Discussion Points and Concerns from the Reviewer:*
- Dataset / batching details
Please mention how the datasets were split for training and testing the models. How much training data is utilized to meta-learn the EBMM initialization? How is batching performed? I believe these details are very important to mention in the paper for reproducibility of results.
Are there any correlations in the batch selection? Can you evaluate how good the associative memory performs across different correlation levels in the batch (A well learnt algorithm should demonstrate better reconstruction at lower memory levels for correlated batches).
- Experiments across multiple SNR and generalization on noise patterns
The authors mention at the beginning of Section 4 that a random block is corrupted, but in the end the experiments are done on a constant corruption size on the CIFAR and ImageNet images. How do the models perform across different signal-to-noise ratios? Similarly, the model is trained on simple noise patterns
- Missing related work
There is related work [1] in learning in Hopfield Networks using the implicit function theorem and finding stationary points of the dynamics. This work is not mentioned in the paper, and is a valid baseline for this paper as well.
- Mentioning Appendix D in the main paper
Appendix D is not mentioned in the main paper and has a short discussion on the mismatch between the reading process and the writing loss during meta-training. It also mentions additional tricks required for the training, and I believe it should be mentioned in the main paper like other sections are appropriately referenced.
- Large batch sizes for ImageNet
Work from [2] can be utilized to backpropagate through very long optimization sequences and therefore can be utilized to train with larger batch sizes in the ImageNet example. It is important to see how the small model utilized for ImageNet works to compress higher batch sizes, as that is one of the major practical issues with the algorithm.
- Related paper at NeurIPS this year
[3] is a related paper from Neurips this year, which the authors could consider adding as contemporary work
- Comments on scalability
The associative memory papers have often been criticized for lack of scalability, and I think the authors make progress towards making this better with the use of unconstrained energy models in the learning process. It would be nice to have a discussion of the scalability from the authors, highlighting issues in the current model and future directions
References:
[1] Reviving and Improving Recurrent Back-Propagation, ICML '18
[2] Gradient-based Hyperparameter Optimization through Reversible Learning, Maclaurin et al. ICML '15
[3] Metalearned Neural Memory, Munkhdalai et al. NeurIPS '19


We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equally well as those trained with small mini-batches but are produced in a substantially shorter time. We demonstrate the reduction in training time and the good generalization performance of the resulting models on the computer vision datasets CIFAR10, CIFAR100, and ImageNet.

This paper proposes a parallel version of the stochastic weight averaging method. It utilizes to phases to train the DNN. The first phase consists of distributed large-batch training, where the learning rate is scaled linearly with respect to the scale of the batch size. The second phase consists of using small batches with SWA to obtain the final model. Experiments verify that this method is able to achieve similar generalization performance as small-batch methods in less training time. A comparison against small-batch SWA, large-batch SWA, etc.
Strengths:
The proposed algorithm is a natural extension to SWA, and appears to give good generalization performance. It is able to utilize large-batch training effectively, which is perhaps surprising given the amount of tuning necessary in Shallue, et al. (2018) in order to achieve good performance. The experiments are well-detailed, and some interesting visualizations, graphs, and empirical analyses are provided.
Weaknesses:
I think that this paper is fairly complete; the only question is whether or not it contains enough novelty, as it is a natural extension of SWA to the parallelized setting. No theoretical analysis of the algorithm is given.
Some questions and comments:
- How sensitive is the algorithm to the choice of the transition point? How was the transition point tuned?
- How large of a batch size can one use before this approach breaks down or is no longer efficient?
- In Figure 2, LB is shown to obtain worse training error than SGD. What is the reason for this? This seems contrary to theory (assuming one has already converged to a neighborhood of the solution).
- The authors comment in the conclusion that "it is possible that regions with bad and good generalization performance are connected through regions of low training loss". Can one check if this is related to the invariances described in Dinh, et al. (2017)?
- What is the relationship between SWAP and methods on local SGD?
This paper is missing some classical optimization references on increasing batch sizes, which has been well-studied within that literature:
[1] Byrd, Richard H., et al. "Sample size selection in optimization methods for machine learning." Mathematical programming 134.1 (2012): 127-155.
[2] Bollapragada, Raghu, Richard Byrd, and Jorge Nocedal. "Adaptive sampling strategies for stochastic optimization." SIAM Journal on Optimization 28.4 (2018): 3312-3343.
[3] Bollapragada, Raghu, et al. "A progressive batching L-BFGS method for machine learning." arXiv preprint arXiv:1802.05374 (2018).
[4] Friedlander, Michael P., and Mark Schmidt. "Hybrid deterministic-stochastic methods for data fitting." SIAM Journal on Scientific Computing 34.3 (2012): A1380-A1405.
Although developing some theory for this algorithm would be beneficial, this paper performs a comprehensive set of experiments and is well-written. For these reasons, I'm inclined to accept the paper.


Training convolutional neural networks (CNNs) is time consuming. Prior work has explored how to reduce the computational demands of training by eliminating gradients with relatively small magnitude. We show that eliminating small magnitude components has limited impact on the direction of high-dimensional vectors. However, in the context of training a CNN, we find that eliminating small magnitude components of weight and activation vectors allows us to train deeper networks on more complex datasets versus eliminating small magnitude components of gradients. We propose Sparse Weight Activation Training (SWAT), an algorithm that embodies these observations. SWAT reduces computations by 50% to 80% with better accuracy at a given level of sparsity versus the Dynamic Sparse Graph algorithm. SWAT also reduces memory footprint by 23% to 37% for activations and 50% to 80% for weights.

This paper studies training neural networks with sparse weights and sparse activations (SWAT training). By using sparse weights in forward passes as well as sparse weights and activations in backward passes, SWAT can reduce the computation overhead and also reduce the training memory footprint. The primary contributions of the paper are in three folds: 1) The authors empirically compare the impact of (activation) gradient sparsity and weight + activation sparsity on the model performance---the comparison shows that the weight + activation sparsity has less influence on the model accuracy; 2) Across different models on CIFAR and ImageNet dataset, SWAP can reduce the training flops by 50% to 80% while using roughly 2 to 3x less training memory footprint saving (weight + activation); 3) The authors empirically study on why training using top-K based sparsification can attain strong model accuracy---the magnitude-based top-K approach can roughly preserve the directions of the vectors.
I think the claimed contributions are well-validated in general. The design decisions of the approach are well supported by empirical observations and the components of the approach (different top-K methods) are studied properly. Additionally, I like the authors' synthetic-data studies to shed light on why top-K based sparsity can work well. Given the above reason, I give week accept and I am willing to raise the score if the following questions / concerns can be resolved in the rebuttal / future draft:
1. In results such as in figure 4, we observe that using intermediate levels of sparsity can actually demonstrate better generalization performance than the dense baseline training approach. I was wondering if this is because the default hyperparameter produces better training loss in sparse training than in dense training, and consequently the sparse training test performance is also improved over dense training. Without showing this, it is not fully convincing that intermediate sparsity helps prevent overfitting and generalizes better (as the authors discussed in the text).
2. For "Impact on Convergence" in section 3.2, it is not clear to me what the authors are using as a metric for the degree of convergence. Thus I can not evaluate the claims here.
3. For "Efficient Top K implementation" in section 3.2, the authors suggest  only computing the K-th largest elements periodically to further improve efficiency. However the empirical evidence of whether this approach will significantly degrade the model performance at the end of training is not provided.
4. For the GFLOPS comparison in Figure 7, could the authors elaborate what operations are included into the count? As the sparse operations requires additional indexing operations for computation, I was wondering whether the GFLOPS can realistically reflect the real latency / energy efficiency of the SWAT approach.
5. How the memory access count calculated at the end of page 7? Is it counting the number of float point values (activations, activation gradients, weights) that needs to be fetched for forward and backward pass?
6. At the first paragraph in page 8 (last paragraph above section 4), do the authors imply that the activations of BN layers is not sparsified? Could the authors provide a bit more evidence on how (any why) sparsification of BN activation impacts the model performance.


Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and videos are available here.

This paper is under the topic of hierarchical reinforcement learning. The motivation of this paper is "most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task." The paper proposes a method to learn higher-level skill selection and lower-level skill improvement jointly.
What I like in this paper:
    1. The paper, in general, is well-written so that I can understand it well.
    2. Experiments are question-driven and provide interesting results.
    3. Theories are closely related to the algorithm.
Key reasons for my rejection:
    1. My biggest concern is the motivation of this paper.
    The joint learning of higher-level policy and lower-level skill discovery is not rare in modern literature. Some works are even cited in this paper, for example, option-critic, feudal network, etc. These methods fix their skills in the new task, not because they are inherently not able to do so, but because they want to demonstrate that the learned skills can be reused in new tasks, even if there is no further adaptation. I agree with the author that the agent needs to adapt its skills when faced with new tasks. But I don't think most works are limited in this aspect, as claimed by the paper in the abstract.
    2. I think the author didn't justify his key design choices well.
    This paper is under the research area of "hierarchical reinforcement learning." However, just like temporal abstraction, the HRL is a general idea instead of an existing problem formulation or a particular algorithm. It seems that the author is not aware of this point as the paper claims a particular way of achieving HRL is the HRL itself (in section 4.1 "In the context of HRL, a hierarchical policy with a manager πθh(zt|st) selects every p time-steps one of n sub-policies to execute."). I would like to see the paper takes the responsibility to justify the reason it follows this particular way. There are two more key decisions the paper proposed but not fully justified and analyzed.
        1. Why is random length a valid choice? The paper doesn't tell readers the consequence of this design choice. For example, what about the optimality of the solution? Since bounding the random length needs prior knowledge, how difficult is it to come up with the prior knowledge. Is the algorithm sensitive to prior knowledge?
        2. Why is it fine to assume "for each action, there is just one sub-policy that gives it high probability"? What would be the consequence of this assumption? Well, the extreme case is each action is only being chosen by one sub-policy. Therefore, executing the sub-policy becomes executing a repeated sequence of the same action. Obviously, this is a kind of temporal abstraction but is a very limited one.
Other small issues:
Section 2: "... maximize the discounted expected reward ..." should be "... maximize the discounted expected return ...".
Section 2: the horizon T in the definition in \eta should be H.
Section 4: the advantage function is not defined.


Current deep neural networks can achieve remarkable performance on a single task. However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget the previous learned knowledge. This phenomenon is referred to as catastrophic forgetting and motivates the field called lifelong learning. The central question in lifelong learning is how to enable deep neural networks to maintain performance on old tasks while learning a new task. In this paper, we introduce a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks. MEGA modulates the balance between old tasks and the new task by integrating the current gradient with the gradient computed on a small reference episodic memory. Extensive experimental results show that the proposed MEGA algorithm significantly advances the state-of-the-art on all four commonly used lifelong learning benchmarks, reducing the error by up to 18%.

========================== Summary of paper ==========================
The paper proposes a new loss balancing approach for lifelong learning. The proposed method dynamically balances the old task loss (from an episodic memory) and the new task loss based on their magnitude. The paper outperforms state-of-the-art method (that don't use extra attribute info), and is straightforward to understand.
========================== Decision ==========================
I vote for rejecting the paper. The paper adds a very simple dynamic loss balancing to a joint loss, which has limited novelty, yet does not discuss its relationship with loss balancing in multitask learning. Although the method outperforms state-of-the-art, a very simple baseline of their method also outperforms state-of-the-art, making the contribution ineffective. The writing of the paper may also benefit from further edits.
========================== Pros/Cons ==========================
+ The paper outperforms state-of-the-art.
+ The method is simple to explain and straightforward to implement.
+ Proposes a weighted variant of forgetting measure (although the purpose is not justified -- need to discuss what use case would make one prefer this weighted variant better than the original).
- The paper is not well-placed in the literature. Not until the bottom of page 5 can readers see very closely related methods, and despite the similarity, little is discussed about the difference. To improve, relationships with existing multitask learning weight balancing methods (e.g. https://arxiv.org/abs/1810.04650, https://arxiv.org/abs/1705.07115, https://arxiv.org/abs/1711.02257) should also be discussed, and maybe compared to.
- Motivation to use the proposed loss balancing rather than that of very similar methods (e.g. GEM/A-GEM) is lacking ("These approaches cannot capture the dynamics in the lifelong learning process", but GEM's balancing is also dynamic)
- Solving for beta using optimization (eq. 10), while (eq. 9) should have an explicit, close-form solution.
- Writing sometimes is confusing, uses absolute language, or using claims unsupported by evidence.
    - (1) page 3 "In this case, the weights are only optimized for the current task while ignoring previous tasks which leads to catastrophic forgetting." ignores the existence of regularization-based methods such as EWC
    - (2) bottom of page 3 "alpha1 and alpha2 should be adjusted adaptively" does not have experiment results supporting it (especially considering GEM is also adaptive)
    - (3) "l_ref(w; ζ) = 0 implies that there is almost no catastrophic forgetting" claim is problematic. Overfitting to the episodic memory is a common problem.
    - (3) page 3 xi and zeta not clearly defined.
    - (4) brings up NP-hardness while it is seldom of interest in this field.
    - (5) missing "∇" on the denominator in (eq. 7).
- Experiment:
    - The only ablation (the direct approach) is statistically indistinguishable from the proposed method. This also outperforms state-of-the-art, while it should not be. One can only assume that the experiment is problematic.
    - Completely uses hyperparameter from an unpublished paper.
    - Claims state-of-the-art, yet omits a state-of-the-art variant in cited A-GEM paper (with joint-embedding). This should be discussed even if the comparison is unfair.
========================== Improving the paper ==========================
- Rewrite the introduction so it is clear the paper is an improvement over A-GEM with a better loss balancing, and focus the motivation and side experiments on why this is important for lifelong learning.
- Clean up writing.
- Explain why the direct approach, while being a basic loss balancing method, outperform state-of-the-art GEM greatly. Perhaps a set of ablation studies can help.
- Replace optimizing for beta as solving for beta.
- Comparison with other loss balancing papers.
- Clarify state-of-the-art comparison.
========================== After rebuttal ==========================
I appreciate the authors addressing my concerns about the placement in the literature; it looks clearer now. However, the rebuttal did not address well the question why the simple MEGA-D variant outperforms prior state-of-the-art besides promising to publish code. The new ablation study is especially difficult to understand, since the first set of results are MEGA without using both memory and new data, but it seems to me that MEGA *needs* the memory data to work, so I have no idea what is being ablated here, and I also do not understand how this answers the question why MEGA-D performs so well. I will not be changing my rating.
Other issues:
(1) Citing A-GEM: please update the bib file so it shows the ICLR version instead of arXiv.
(2) Solving eq. 9: Here you go: https://imgur.com/a/b4QCVlv
I literally had this problem on one of my high school exams. Feel free to use this in any future version.


Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy). As a result, using manually designed action space to perform NAS often leads to sample-inefficient explorations of architectures and thus can be sub-optimal. In order to improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with concentrated performance metrics, i.e., low variance. During the search phase, as different architecture search action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions. On 3 NAS datasets, our experimental results demonstrated that LaNAS is 37×, 21.7×, 19.3×, 10.7×, 12.3×, 16.5× more sample-efficient than Random Search, Regularized Evolution, Monte Carlo Tree Search, Neural Architecture Optimization, TPE and SMAC, respectively. When applied to the open domain, LaNAS achieves 98.0% accuracy on CIFAR-10 and 75.0% top1 accuracy on ImageNet in only 803 samples, outperforming SOTA AmoebaNet with 33× fewer samples.

This paper introduces a Neural Architecture Search algorithm that attempts to solve the problems of the existing NAS only utilizing manually designed action space (not related to the performance). The paper proposes LaNAS which is based on an MCTS algorithm to partition the search space into tree nodes by the performance in the tree structure. The performance of the method is shown in the NASBench-101 dataset and Cifar-10 open domain search.
I lean to reject this paper because (1) the motivation is not well justified by the experiments, (2) the comparison on NASBench-101 is not convincing, (3) some important explanation of the method is missing.
Main arguments
The main contribution of the paper is using a learned action space in MCTS rather than a manually designed MCTS algorithm for NAS. However, as far as I know, the MCTS approach for the NAS problem is not a standard solution for NAS (which is not proved to be practically useful in other people’s papers) which diminishes the contribution of the improvements of MCTS in NAS.
Lack of the main comparison. For the motivation of the proposed method, the authors mention the drawbacks of other NAS methods used fixed action space in their RL or MCTS module. However, the authors only show that using a learned action space in MCTS is better than a fixed MCTS algorithm in the experiments. What about using a learned action space in the RL module such as PPO in NAS comparing to the fixed one?
The comparison of NASBench-101 is not convincing. The authors compared with the BO method and claimed that the method is 16.5x more efficient than BO. However, the recently released paper “BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search” said that their method is 3.8x more efficient than the one proposed here, which is quite confusing.
Some important explanation of the methods is missing. Throughout the paper, the method to sample from a leaf node is only mentioned in 3.3. However, the corresponding sampling method is unclear. The paper only mentions that MCMC has been adopted to sample from the target subspace. In my opinion, it is not so trivial to use MCMC here and should be elaborated in more detail. Otherwise, people cannot use it.
As given in figure 3, if c is set to be a very small number, the search is similar to simply using a series of predictors and always samples the models with better-predicted accuracies. Is MCTS really useful here? To show the effectiveness of MCTS, it is recommended to experiment on different values of c.
Results given in the upper row of Fig.5 is not useful. In practice, it is already painful to sample about 1000 models and train them for the Cifar-10 dataset. It is more useful to see how this method behaves in the range of (0,1000). However, in Fig.5, different methods are all overlapping in this range and hard to tell whether this method is better than other methods


Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at all. We answer both the questions affirmatively under the overparameterized twolayer neural-network parameterization. In detail, assuming independent sampling, we prove that neural natural policy gradient converges to a globally optimal policy at a sublinear rate. Also, we show that neural vanilla policy gradient converges sublinearly to a stationary point. Meanwhile, by relating the suboptimality of the stationary points to the representation power of neural actor and critic classes, we prove the global optimality of all stationary points under mild regularity conditions. Particularly, we show that a key to the global optimality and convergence is the "compatibility" between the actor and critic, which is ensured by sharing neural architectures and random initializations across the actor and critic. To the best of our knowledge, our analysis establishes the first global optimality and convergence guarantees for neural policy gradient methods.

This paper studies policy gradient where the policy is parameterized by an extremely wide neural network. The authors assume that the number of nodes ($m$) in the network is extremely large ($T^{8}R^{18}$, $T$ is the total runtime and $R$ is the radius of the function class that the network falls into), and they restrict their convergence analysis of the policy gradient algorithm in a particular function class and claims that the approximation error between the true network and the function class goes to zero when $m$ is large. The paper is written in a rigorous way and the presentation is mostly clear. I have some concerns about many of the assumptions made across the paper that are not explained or verified. This may potentially further decrease the usefulness of the analysis in this paper though the theoretical result with $m=T^8$ is already very impractical. Another major issue with this paper is that the theoretical analysis is not novel in terms of bringing new insights and results to the field given many other papers including global convergence of policy gradient (Agarwal et al., 2019;), convergence of neural TD learning (Cai et el., 2019), theory for overparameterized neural networks (Jacot et al., 2018; Allen-Zhu et al., 2018a;b; Du et al., 2018a;b; Zou et al., 2018; Chizat and Bach, 2018; Jacot et al., 2018, etc.) and other very similar papers.
There is a prior work by Agarwal et al. (2019) that proves the global convergence of both vanilla policy gradient and natural policy gradient methods. In the related work, the authors distinguish their paper from that of Agarwal et al. (2019) by claiming that they are studying the non-tabular setting and they use the actor-critic scheme. However, the first claim is incorrect because Agarwal et al. (2019) also studied the non-tabular setting (see Section 6 in Agarwal et al. (2019)) and proved an O(1/\sqrt{T}) convergence rate. Moreover, the actor-critic scheme in this paper is just a trivial modification of the nonlinear policy gradient method by calling existing result for TD learning in Cai et al. (2019). Therefore, the contribution of this paper is not so clear given existing papers.
In Algorithm 1, the policy gradient estimator $\widehat{\nabla} J(\pi_{\theta_i})$ also depends on the critic parameter $\omega_i$. It is better to show this dependency in the notation as well.
In Algorithm 1, the temperature parameter $\tau_i$ is updated in natural policy gradient but not in vanilla policy gradient. It seems that $\tau_i$ increases linearly with $i$, which makes the policy defined in eq (3.1) close to a uniform distribution when the time horizon goes to infinity. This seems to offset the update of parameter $\theta_i$.
In the update of natural policy gradient, solving eq (3.8) is really expensive in computation, especially in the setting of this paper where $m$ is chosen as $T^{8}$. It seems impossible to obtain a reasonable solution within the claimed $O(1/T^{1/4})$ runtime.
What is the function $\iota(w)$ in Assumption 4.1?
It would be better for the authors to discuss more about Assumption 4.1. It is unknown why the action-value function $Q^{\pi}$ for all policy can fall into this class.
The equation in Assumption 4.2 is exceeding the paper margin. Please make sure the paper follows the format guidelines.
Assumption 4.2 seems to be very strong. The remark after the assumption says that this condition is made on the Markov transition kernel. However, this may not be true since the assumption needs to hold for any two arbitrary policies. It is not known what kind of transition kernel $\mathcal{P}$ will satisfy this.
In each step of the neural policy gradient (Algorithm 1), the authors need to call a TD learning (Algorithm 2) to approximate the unknown action-value function $Q_{\omega_i}$ associated with the policy $\pi_{\theta_i}$ at the $i$-th step. It seems that in the learning process of ALgorithm 2, at each iteration, it samples independent data from the stationary state-action distribution which is unknown.
In the proof of Theorem 4.8, it seems that eq (D.14) and (D.15) are the same. Why it needs to be proved twice? In addition, why the equation after (D.14) holds?
The authors should provide more details about the function $u_{\hat \theta}$ defined in eq (4.4), which seems to approximate the critic function. Specifically, why are there the derivative terms instead of just the inner product term in eq (4.4).
Other comments:
In the last sentence of Section 3.1, “... approximate aompatible function approximation ...”


Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures.

The paper introduces CATER: a synthetically generated dataset for video understanding tasks. The dataset is an extension of CLEVR using simple motions of primitive 3D objects to produce videos of primitive actions (e.g. pick and place a cube), compositional actions (e.g. "cone is rotated during the sliding of the sphere"), and finally a 3D object localization tasks (i.e. where is the "snitch" object at the end of the video).  The construction of the dataset focuses on demonstrating that compositional action classification and long-term temporal reasoning for action understanding and localization in videos are largely unsolved problems, and that frame aggregation-based methods on real video data in prior work datasets, have found relative success not because the tasks are easy but because of dataset bias issues.
A variety of models from recent work are evaluated on the three proposed tasks, demonstrating the validity of the above motivation for the construction of the dataset.  The primitive action classification task is "solved" by nearly all methods and only serves for debugging purposes.  The compositional action classification task is harder and shows that incorporating LSTMs for temporal reasoning leads to non-trivial performance improvements over frame averaging.  Finally, the localization task is challenging, especially when camera motion is introduced, with much space for improvement left for future work.
I am positive with respect to acceptance of this paper.  It is a well-argued, thoughtful dataset contribution that sets up a reasonable video understanding dataset.  The authors recognize that since the dataset is synthetically generated it is not necessarily predictive of how methods would perform with real-world data, but still it can serve a useful and complementary role similar to the one CLEVR has served in image understanding.
I have a few minor comments / questions / editing notes that would be good to address:
- The random baseline isn't described in the main text, it would be good to briefly mention it (this will also help to clarify why the value is particularly high for tasks 1 and 2)
- The grid resolution ablation results presented in the supplement are actually quite important -- they demonstrate that with a small increase in granularity of the grid the traditional tracking methods begin to be the best performers. As this direction (of increased resolution to make the problem less artificial) is likely to be important, a brief discussion of this finding from the main paper text would be appropriate
- p3 resiliance -> resilience
- p4 objects is moved -> object is moved
- p6 actions itself -> actions themselves; builds upon -> build upon
- p7 looses all -> loses all; suited our -> suited to our; render's camera parameters -> render camera parameters; to solve it -> to solve the problem
- p8 (Xiong, b;a) and (Xiong, b) -> these references are missing the year; models needs to -> models need to
- p9 phenomenon -> phenomena; the the videos -> the videos; these observation -> these observations; of next -> of the next; in real world -> in the real world


Deep neural networks achieve remarkable performance in many computer vision tasks. For most networks in semantic segmentation (seg) and object detection (det) tasks, the backbone directly reuses the network manually designed for classification tasks. Utilizing a network pre-trained on ImageNet as the backbone has been a popular practice for seg/det tasks. However, because of the gaps between different tasks, adapting the network to the target task brings performance promotion. Some recent neural architecture search (NAS) methods search for the backbone of seg/det networks. ImageNet pre-training of the search space representation or the searched network bears huge computational cost. In this paper, we propose a fast neural network adaptation (FNA) method, which can adapt both the architecture and parameters of a manually designed seed network to the new seg/det tasks efficiently. A parameter remapping mechanism is designed to accelerate the whole adaptation process which takes full advantage of the knowledge from the seed network. The NAS method is utilized for architecture adaptation. In experiments, we conduct network adaptation on the MobileNetV2 network to both seg and det tasks. FNA demonstrates clear performance gains compared with both manually and NAS designed networks. The total computational cost of FNA is significantly less than many SOTA seg/det NAS methods: 1737× less than DPC, 6.8× less than Auto-DeepLab and 7.4× less than DetNAS.

In this paper, the authors take a MobileNet v2 trained for ImageNet classification, and adapt it either (i) semantic segmentation on Cityscapes, or (ii) object detection on COCO. They do this by first expanding the network into a "supernet" and copy weights  in an ad-hoc manner, then, they perform DARTS-style architecture search before fine-tuning for the task at hand.
There is no TLDR for this paper, and I must admit, on reading the abstract and introduction I wasn't entirely sure what this paper was doing at first. Perhaps I was being slow.
From a narrative perspective, one of the main selling points is not needing to perform any expensive ImageNet pre-training; however, a pre-trained MobileNetv2 is being utilised. While this was off-the-shelf, it still incurred an initial training cost, so it isn't really fair in e.g. Table 4 to put pre-training cost as zero. On a related note, the authors write that this network is used for its "generality". I'd argue that MobileNetv2 is a highly engineered network specialised for mobile computation; a standard ResNet-50 would be more general really.
I would like to see a comparison to a random search, as there are several papers (https://arxiv.org/abs/1902.07638, https://arxiv.org/abs/1902.08142) indicating that this is a very strong baseline.
As mentioned earlier, the choices for remapping weights seem very ad-hoc. I can't really tell what's going on in Table 5 (why is PR in the NE and PA row?) so the ablation study of how effective this weight mapping is lost on me.  The stuff in Table 6 is pretty interesting however, if convoluted.
I find the odd choices of hyperparameters (tau as 45, gamma as 10, lambda as 9e-3) rather alarming. How important are these? Would this technique work under any other circumstances?
Error bars would be a welcome inclusion, particularly in Table 3 where you have 0.1% separating FNA and MNasnet-92. I appreciate that this can be expensive however.
Pros:
- Some promising results
- Good figures
Cons:
- Ad-hoc design choices
- Not a fair comparison regarding pre-training.
- Very specific to one network choice
- Lack of error bars or comparison to random search.
I am giving this paper a weak reject, as there is insufficient experimental evidence that the technique works, or generalises beyond Mobilenetv2. I am also concerned about the ad-hoc hyperparmaters or weight-mapping. A comprehensive ablation study, along with error bars, and another choice of seed network would do much to strengthen this paper.


Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance objectives of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different objectives. The main idea is to regress the mapping from hyperparameter to objective quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales or outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple objectives such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods.

This paper tackles the problem of black-box hyperparameter optimization when multiple related optimization tasks are available simultaneously, performing transfer learning between tasks. Different tasks correspond to different datasets and/or metrics. Gaussian copulas are used to synchronize the different scales of the tasks.
I have several reservations with this paper. First and foremost, it seems to be lacking a fair and trivial baseline (I will describe it below) that justifies the apparently unnecessary complicated path followed in this paper. Second, there are a few small incorrect or improperly justified technical details throughout the paper.
1) Mistaken/unjustified technical details:
- In equation 1, the last term seems to be constant. For each task, the function psi is not parametric, so its gradient is also not parametric and the input is the inverse of z, i.e., y, which is also fixed. So why is it included in the cost function? This sort of probabilistic renormalization is important in e.g. warped GPs because the transformation is parametric. In this case, I don't see the point. It can be treated as a normalization of the input data, prior to its probabilistic modeling.
- Before equation 1, the text says "by minimizing the Gaussian negative log-likelihood on the available evaluations (x, z)" But then, equation 1 is not the NLL on z but on y.
- In section 4.2 the authors model the residuals of the previous model using a powerful Matern-5/2 GP. Why modeling the residuals this way and not the observations themselves? The split of modeling between a parametric and non-parametric part is not justified.
- One of the main points of the variable changes is to normalize the scales of the different tasks. However, equations 1 adds together the samples of the different tasks (which, as pointed out by the authors might have different sizes). Even if the scales of the outputs are uniform, the different dataset sizes will bias the solutions towards larger datasets. Why would that be a good thing? This is not mentioned and doesn't seem correct: there should not be a connection between a dataset size and the prior influence of the corresponding task. In fact, this will have the same effect as if the cost had different scales for different tasks, which is precisely the problem that the authors are trying to avoid.
2) Trivial baseline
Given that the authors are trying to aggregate information about the optimal hyperparameters from several tasks, they should not compare with single-task approaches, but with the simplest way to combine all the tasks. For instance:
    a) Normalize the outputs of every task. This can be accomplished in the usual way by dividing by the standard deviation, or even better, by computing the fixed transform z = psi(y), separately for each task.
    b) Collect the z of all tasks and feed them into an existing GP black-box Bayesian optimizer.
This is a very simple way to get "transfer learning" and it's unclear that the extra complexities of this paper (copulas, changes of variable with proper renormalization when the transformation is parameter free, etc) are buying much else.
Minor improvements:
- Page 2: "is the output of a multi-layer perceptron (MLP) with d hidden nodes" Is d really the number of hidden nodes of the MLP? Or the number of outputs? Given that d is also the size of w, it seems it's actually the latter.
- Explain why the EI approach is used for the second model (with the GP), but not for the first model.
Edit after rebuttal:
“The term is not constant over z” -> Sure, it’s not constant over z. But z is constant. So the term is constant.
“The NLL is minimized in z and there is indeed no y in equation 1.” -> Sure, there’s no y in the equation, that’s correct. But it is still the NLL of y, and not the NLL of z.
About the new baseline: Instead of simply renormalizing using mean and standard deviation, I suggested above using the same z=psi(y) that is used in the paper for the normalization. Is that where the advantage of the proposed method is coming from?
"Note that this is orthogonal to the scale issues we focus on: larger tasks will have larger gradient contributions but the scaling we propose still allows us to learn tied parameters across tasks as their scales are made similar. " Both issues affect the scaling of the task, so I don't see how they can be orthogonal. Their scales are not made similar precisely because of the different sample sizes.


Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries. However, due to lack of prior and inefficiency in leveraging the query and feedback information, existing methods are mostly query-intensive for obtaining effective attack patterns. In this work, we propose a meta attack approach that is capable of attacking a targeted model with much fewer queries. Its high queryefficiency stems from effective utilization of meta learning approaches in learning generalizable prior abstraction from the previously observed attack patterns and exploiting such prior to help infer attack patterns from only a few queries and outputs. Extensive experiments on MNIST, CIFAR10 and tiny-Imagenet demonstrate that our meta-attack method can remarkably reduce the number of model queries without sacrificing the attack performance. Besides, the obtained meta attacker is not restricted to a particular model but can be used easily with a fast adaptive ability to attack a variety of models. Our code will be released to the public.

This paper proposed a meta attack approach that is capable of attacking a targeted model with fewer queries. It effectively utilizes meta learning approaches to abstract generalizable prior from previous observed attack patterns. Experiments on several benchmark datasets demonstrates it performance in reducing model queries.
- The proposed meta learning approach seems to learn a universal gradient direction of an image x for all networks, which seems to be an ambitious goal. The authors did not provide any intuition or demonstrative explanations towards this. I hope the author could provide some more evidence showing that why this is achievable and is this indeed achieved by the meta learner (e.g., comparing the cosine similarity between the true gradient and the meta learner generated gradient?)
- In Algorithm 2, the authors seem to use coordinate-wise zeroth-order gradient estimation as in ZOO. I wonder have the authors considered using NES type Gaussian noise to estimate the gradient? As it has been shown to be more efficient than coordinate-wise zeroth-order gradient estimation in (Ilyas et al. 2018).
- I notice that the authors only consider L2 norm attack case and did not include more popular L-infinity norm case. Is there any reason why it can not be applied to L-infinity norm? I didn’t find anywhere in the algorithm that would restrict the choice of norm type. The author might also need to compare with the following recent papers regarding black-box attacks.
Yan, Ziang, Yiwen Guo, and Changshui Zhang. "Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks." arXiv preprint arXiv:1906.04392 (2019).
Moon, Seungyong, Gaon An, and Hyun Oh Song. "Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization." ICML 2019.
Chen, Jinghui, Jinfeng Yi, and Quanquan Gu. "A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks." arXiv preprint arXiv:1811.10828 (2018).
- In experiments part, only 100 images from each dataset may not be representative enough. I would suggest the authors to test more samples. Also please consider only the images that can be correctly classified without perturbation, as there is no need to attack those already misclassified images.


Ensuring the privacy of sensitive data used to train modern machine learning models is of paramount importance in many areas of practice. One approach to study these concerns is through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. A particular instance of this approach is through a "teacher-student" model, wherein the teacher, who owns the sensitive data, provides the student with useful, but noisy, information, hopefully allowing the student model to perform well on a given task without access to particular features of the sensitive data. Because stronger privacy guarantees generally involve more significant noising on the part of the teacher, deploying existing frameworks fundamentally involves a trade-off between utility and privacy guarantee. One of the most important techniques used in previous work involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism, which we call an Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student. Our mechanisms improve over the state-of-the-art methods on all measures, and scale to larger tasks with both higher utility and stronger privacy ( ≈ 0).

This paper studies the teacher ensembles setting for differentially private learning. In this setting, each teacher holds part of the training set and trains a local model. The student uses unlabeled examples to query teacher model. Then the student trains a model from scratch using the examples labeled by teachers.
In order to make the labeling process differentially private, previous work uses noisy argmax mechanism. Each class of label is assigned with a count number. The student first queries the same example to multiple teachers. To guarantee differential privacy, the counts are perturbed by noise before releasing. Then, because of the post-processing property of differential privacy, the argmax operator on such noisy counts are still differentially private.
This paper proposes to add a constant c to the largest count before perturbing and releasing the counts. The authors argue this would improve the accuracy of the noisy argmax operator and yield the same privacy loss as previous approach. However, adding a constant c would increase the sensitivity and therefore degenerates the privacy guarantee. The added noise cannot guarantee the privacy if all others are the same as previous work. To see this clearer, for example, if c=0, then one sample point can at most change the count by 1. If c>0, then one sample point can change the count by 1+c. Because of this, the proposed method cannot guarantee the amount of differential privacy as the paper claimed.


We study how to set the number of channels in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot approach, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods with 100× lower search cost. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs).

In this paper, the authors propose a method to perform architecture search on the number of channels in convolutional layers. The proposed method, called AutoSlim, is a one-shot approach based on previous work of Slimmable Networks [2,3]. The authors have tested the proposed methods on a variety of architectures on ImageNet dataset.
The paper is well-written and easy to follow. I really appreciate the authors for structuring this paper so well. I have the following questions:
Q1: In figure 4, the authors find that “Compared with default MobileNet v2, our optimized configuration has fewer channels in shallow layers and more channels in deep ones.” This is interesting. Because in network pruning methods, it is found that usually later stages get pruned more [1] (e.g. VGG), indicating that there is more redundancy for deep layers. However, in this case, actually deep layers get more channels than standard models. Is there any justification for this? Is it that more channels in deep layers benefit the accuracy?
Q2: In “Training Optimized Networks”, the authors mentioned that “By default we search for the network FLOPs at approximately 200M, 300M and 500M, and train a slimmable model.” Does this mean that the authors train the final optimized models from scratch as a slimmable network using “sandwich rule” and “in-place distillation” rule? Or are the authors just training the final model with standard training schedule? If it is the first case, can the authors justify why?
Q3: In Table 1, “Heavy Models”, what is the difference between “ResNet-50” and “He-ResNet-50”? Also, why the params, memory and CPU Latency of some networks are omitted?
Q4: In the last paragraph of section 4, the authors tried the transferability of networks learned from ImageNet to CIFAR-10 dataset. I am not sure how the authors transfer the networks from Imagenet to CIFAR-10? Is it the ratio of the number of channels? Can the authors provide the architecture details of MobileNet v2 on CIFAR-10 dataset?
Q5: What is the estimated time for a typical run of AutoSlim? How does it compare to network pruning methods or neural architecture search methods?
Q6: Can the methods be used to search for the number of neurons in fully connected layers? Are there any results?
[1] Rethinking the Value of Network Pruning. Zhuang et al. ICLR 2019
[2] Slimmable neural networks. Yu et al. ICLR 2019.
[3] Universally Slimmable Networks and Improved Training Techniques. Yu et al. Arxiv.


Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song. Such components include voice, bass, drums and any other accompaniments. Contrarily to many audio synthesis tasks where the best performances are achieved by models that directly generate the waveform, the state-of-the-art in source separation for music is to compute masks on the magnitude spectrum. In this paper, we first show that an adaptation of Conv-Tasnet (Luo & Mesgarani, 2019), a waveform-to-waveform model for source separation for speech, significantly beats the state-of-the-art on the MusDB dataset, the standard benchmark of multi-instrument source separation. Second, we observe that Conv-Tasnet follows a masking approach on the input signal, which has the potential drawback of removing parts of the relevant source without the capacity to reconstruct it. We propose Demucs, a new waveform-towaveform model, which has an architecture closer to models for audio generation with more capacity on the decoder. Experiments on the MusDB dataset show that Demucs beats previously reported results in terms of signal to distortion ratio (SDR), but lower than Conv-Tasnet. Human evaluations show that Demucs has significantly higher quality (as assessed by mean opinion score) than Conv-Tasnet, but slightly more contamination from other sources, which explains the difference in SDR. Additional experiments with a larger dataset suggest that the gap in SDR between Demucs and Conv-Tasnet shrinks, showing that our approach is promising.

In this work, the authors consider the task of supervised music source separation, i.e., separating out the components (bass, drums, voice, other) out of a mixed music track. In particular, the work considers the task of supervised source separation where the individual target tracks are available during training time. The main contribution of this work is the improvement of an end-to-end waveform-to-waveform separation models through a number of architectural changes that allow such waveform-to-waveform models to perform comparably with other current state-of-the-art methods that instead operate in the spectrogram domain.
Overall, the paper is generally well written and the method is easy to follow. However, I have a couple of concerns about this work, based on which I would rate the work as “weak reject”:
1. The specific architecture proposed in this task does appear to improve performance for this particular task, but it is not clear to me that the conclusions drawn based on the study in this work will be generally applicable to other related tasks. Also, the final model is still only comparable to the spectrogram-based methods overall, and appears to do a slightly worse job in separating the ‘vocal’ and ‘other’ tracks than these baseline methods. As such, I’m unsure about the impact of this work.
2. The section describing the evaluation metric SDR wasn’t very clear to me. Perhaps it would be better to just refer back to (Vincent et al., 06) and just describe what SDR captures instead? Or alternatively, more details could be added to explain the computation more clearly.
3. Section 4.2: The authors mention that they multiply each source by +/- 1. I didn’t follow why this is being done. Could the authors please clarify.
Minor comments:
1. The notation in Equation 1 is slightly confusing because ‘s’ is used to index both the mixture and the sources. I think x \in \mathcal{D} would be clearer. Similarly I wonder if x_s \in \mathbb{R}^C \times \mathbb{R}^T would be clearer than x_s \in \mathbb{R}^{C,T}
2. The reference (Oord et al., 2017) should be rendered as (van den Oord et al., 2017)
3. The Figure uses K, and S to denote Kernel Width and stride, but this isn’t explicitly mentioned in the text. Perhaps it would be useful, in Section 3.1 to write: “... convolution with kernel width, K=8, and stride, S=4, ...” or something similar, and updating the caption to explain the notation.
4. Section 3.2: “The decoder is almost the symmetric of the encoder” --> “The decoder is almost the inverse of the encoder” would perhaps be better?
5. Section 3.2: “The final layer … S.C_0 ...” --> “The final layer … S * C_0 ...”
6. The notation in Equation 1 and Equation 2 is slightly inconsistent (lower-case vs. upper-case L for the loss function)
7. Section 4.3: “additionnaly” --> “additionally”; “32 GB or RAM” --> “32 GB of RAM”


Learning an agent that interacts with objects is ubiquituous in many RL tasks. In most of them the agent's actions have sparse effects : only a small subset of objects in the visual scene will be affected by the action taken. We introduce SPECTRA, a model for learning slot-structured transitions from raw visual observations that embodies this sparsity assumption. Our model is composed of a perception module that decomposes the visual scene into a set of latent objects representations (i.e. slot-structured) and a transition module that predicts the next latent set slot-wise and in a sparse way. We show that learning a perception module jointly with a sparse slot-structured transition model not only biases the model towards more entity-centric perceptual groupings but also enables intrinsic exploration strategy that aims at maximizing the number of objects changed in the agents trajectory.

This paper introduces a model that learns a slot-based representation, along with a transition model to predict the evolution of these representations in a sparse fashion, all in a fully unsupervised way. This is done by leveraging a self-attention mechanism to decide which slots should be updated in a given transition, leaving the others untouched. The model learns to encode in a slot-wise and is trained on single step transitions.
This work tackles an important problem, and is very well motivated and presented in a very clear fashion. It reuses some known ideas and components, but combines them in a nice way. I especially liked the use of attention to select what to update, which is a good prior to have.
However, the results presented unfortunately seem to fall a bit short in this current version, and some decisions might have had too much of an effect on some of these shortcomings. Given some improvements, this work might become quite promising, but for the time being I am leaning against publication.
1.	Most modeling decisions are clear and well-motivated, however the choice to make the transition model f_trans always be applied only “slot-wise” might be too restrictive. Indeed, for a given action, this means that 2 slots have to independently learn the effect of that action (e.g. as shown in the example in Figure 2. right), and that some interactions are ~impossible to learn (e.g. in Sokoban, pushing a box requires knowing about the location of both the agent and the box). This could have been alleviated if the transition had access to “interactions outcomes” (e.g. if using a GraphNet, or in your model, if \tilde{s} was provided to the transition function f_theta). Other works (including Zambaldi et al 2018, which is cited several times), handle this appropriately.
Did you try to provide \tilde{s} to the transformation operator (e.g. in Figure 7 left)?
2.	Adding a direct comparison to pure non-slotted versions of the model/baselines would have been quite useful, as currently it is unclear why certain things are failing.
3.	Similarly, finding what was the output of the CNN encoder for pixel inputs was a bit too difficult. It is explained in the Appendix that one maps into 4x4 feature maps, but that might be too large for the current environments? Indeed for Sokoban, this means that any grid is partially supported by several “slots”, which may hurt the results more than they should (especially combined with the slot-wise transition constraints expressed above).
4.	The early state of the current results are quite visible in all examples of the “Separate training” model predictions (Figure 3, 5, 8, 9, 10 and 11). None of these actually show this model performing a “correct” prediction for t+1? They only predict no changes, or nonsensical interpolations… This is not sufficient to try to make an argument about the “joint training” helping, and most discussions about “what information they contain” is strenuous at best.
5.	The paper keeps mentioning that it “implicitly imposes transitions to be sparse”, however it is never explained how that would come about? I understand that the softmax in the self-attention may tend to become “peaky” and hence only affect a few slots (and the results do seem to confirm this observation), but I was expecting an explicit loss to enforce this fact. The current emphasis seems a bit ill-funded, so I would present more evidence to it or downplay it.
6.	Some of the results shown seem hard to interpret or provide only weak evidence for the proposed model:
	a.	Figure 2. Left does seem to indicate a benefit in using the self-attention module, but it is hard to know how much of an effect the gap between the orange and red curves actually imply. This figure is overall a bit too small to interpret, and it might be better to split the 2 conditions into sub-plots. The names of the curves in the legend do not correspond to anything described in the text/caption (but I could understand them…).
I was expecting more discussion of the results in Figure 2, for example at the end of Section 4.1.
	b.	As explained above, Figure 3 only shows that the Joint Training can perform a 1-step prediction, which is ok but is the bare minimum.
Does it handle multi-step rollouts?
	c.	Figure 4 does not seem to provide any significant results or insights. I would interpret them as showing no significant difference between the curves, and they are too small to extract any information out of them. I would remove this figure fully.
	d.	Figure 5 is unclear about what f_k=0 really is, and once again just shows that Separate training does nothing. I expected it to be exactly reconstructing s_t (given that the others are trying to predict s_t+1)? But this is only the case for the joint training, and without knowing what x_t actually was, it is hard to trust. The fact that f_k changes what it does as it is being increased makes the whole point hard to interpret.
7.	Figure 6 was quite interesting, and I feel like this could be pushed forward in a quite interesting manner. The choice of “maximizing the number of entities selected” was fair as a first try, but I could imagine it failing to generalize to more complex environments, or to be easily exploited if one ever decides to pass gradients back into the representation from the policy.
It was unfortunate that the legends do not correspond to anything expressed in the main text or caption (e.g. why do you not reuse the “valid_move”, …, ”blocked_push” names that you thoroughly introduce?). What is “random_XX”?
Could you comment on why the curves seem to have a large increase in variance along the 80000-100000 updates region?
Details:
8.	Figure 2 (right) was good to explain how the model worked, but I would actually change its location and try to move it into Figure 1. Similarly, Figure 7 in the Appendix seemed rather necessary to understand the model, and belongs in the main text in my opinion.
9.	When presenting the self-attention block, it would be good to directly state that these are MLPs receiving [s_t, a_t] (as done in the Appendix).
10.	Is sigma^2 in the decoder fixed? To which value?
11.	When presenting the “sparse” and “full” settings, having access to Figure 7 and the rest of the Appendix might be beneficial, it would be good to point forward to it.


Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks. A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive. In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it. Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families. We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks. In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO. We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

The paper proposes to improve standard variational inference by increasing the flexibility of the variational posterior by introducing a finite set of auxiliary variables. Motivated by the limited expressivity of mean field variational inference the author suggests to iteratively refine a ‘proposal’ posterior by conditioning on a sequence of auxiliary variables with decreasing variance. The key requirement to set the variance of the auxiliary variables such that the integrating over them leaves the original model unchanged. As noted by the authors this is a variant of auxiliary variables introduced by Barber & Agakov. The motivation and theoretical sections seems sound and the experimental results are encouraging, however maybe not completely supporting the claim of new ‘state of the art’ on uncertainty prediction.
Overall i find the motivation and theoretical contribution interesting. However I do not find the experimental section completely comprehensive why I currently think the paper is borderline acceptance.
Comments
1) The computational demand using the method seems quite large by adding O(NumSamples * NumAuxiliary) additional computational cost on top of the standard VI. Here each factor M is quite large e.g. 200 epochs for CIFAR10 (if i understand the algorithm correctly?)
2) For the UCI experiments the comparison is only made against DeepEnsembels or other VI methods, however to the best of my knowledge MCMC methods are superior in this setting given the small dataset size?
3) The results on CIFAR10 do seem to demonstrate that the proposed method is superior to DeepEnsembles and standard VI in one particular setting where VI is only performed over a small subset of layers in a ResNet (why doesn’t it work for when doing VI on all the parameters?). However generally looking at the best obtained results of ~86% acc this is quite far from current best probabilistic models (see e.g. Heek2019 that gets 94% acc). Some of this can probably be attributed to differences in data-augmentation and model architecture however in general it makes it very hard to compare with other methods when the baselines are not competitive.
Minor Comments:
 In relation to comment 3) above I think you should reword the sentence “It sets a new state-of-the-art in uncertainty estimation at ResNet scale on CIFAR10” in the conclusion.
“In  order  to  get  independent  samples  from  the  variational  posterior,we have to repeat the iterative refinement for each ensemble member”: Does this imply that if we want M samples we first have to optimize using the standard VI and then to M optimizations to get q_k(w)?
How sensitive is the method to sequence of variances for a?
[Heek2019]: Bayesian Inference for Large Scale Image Classification


We describe an explainable AI saliency map method for use with deep convolutional neural networks (CNN) that is much more efficient than popular gradient methods. It is also quantitatively similar and better in accuracy. Our technique works by measuring information at the end of each network scale which is then combined into a single saliency map. We describe how saliency measures can be made more efficient by exploiting Saliency Map Order Equivalence. Finally, we visualize individual scale/layer contributions by using a Layer Ordered Visualization of Information. This provides an interesting comparison of scale information contributions within the network not provided by other saliency map methods. Since our method only requires a single forward pass through a few of the layers in a network, it is at least 97x faster than Guided Backprop and much more accurate. Using our method instead of Guided Backprop, class activation methods such as Grad-CAM, Grad-CAM++ and Smooth Grad-CAM++ will run several orders of magnitude faster, have a significantly smaller memory footprint and be more accurate. This will make such methods feasible on resource limited platforms such as robots, cell phones and low cost industrial devices. This will also significantly help them work in extremely data intensive applications such as satellite image processing. All without sacrificing accuracy. Our method is generally straight forward and should be applicable to the most commonly used CNNs 1 .

The paper presents a new approach, SMOE scale, to extract saliency maps from a neural network. The approach is deemed as efficient, because it does not require one or multiple backward passes, as opposed to other approaches based on gradients. The main idea is to process the output activation tensors of a few selected layers in a deep network using an operator combining mean and std.dev. called SMOE scale. The result of this operator can be combined through different scales to obtain a global saliency map, or visualized in such a way that shows the consistency of saliency maps at different scales. Experiments show some improvement against traditional gradient-based approaches.
This paper is interesting in that it provides a different way to extract saliency maps from a network. The visualization at multiple scales is also nice and while I do not perfectly agree with the HSV encoding in Fig.2, I do see the potential. Being efficient is also a nice to have. There are three issues, however, which limits the novelty of the paper. First, the SMOE metric does not seem to bring much improvement compared to simple metrics. Second, the few comparisons made against other methods do not reveal a significant improvement. Third, at core, the paper suggests that the "high efficiency" of this approach is one of its main advantages, a statement I do not forcibly agree with. More details follow.
For the first element, we have to consider the paper as the combination of two things. 1) the use of activation maps as source of salient information, and 2) the way we should process these activation maps. 1) is relatively straightforward, so the core of the contribution should lie in 2). However, while the SMOE scale method definition (eq.2) is sound, it does not bring valuable improvement compared to other "trivial" metrics, like standard deviation. For instance, Fig.4 caption tells that "SMOE Scale differentiates itself the most early on in the network", but it is actually only for the very first scale layer. At every other scale, standard deviation (for instance) is at least as good. Same thing can be said about Table 4 in appendix, and also about Table 2 (and the scores of Trunc. Normal Entropy). Overall, while SMOE is indeed novel, it is not highly convincing.
On a side note about the SMOE description, I did not find the list of "conditions and assumptions" at the beginning of Sec. 2.1. It looks more like an after-thought over which the proposed method coincidentally fits. Moreover, point 3 is kind of conflicting in its formulation.
For the second element, the improvements in KAR and ROAR scores are quite minimal. It does seem to have an edge on KAR score, but not by a huge amount. Additionally, the methods compared are relatively old. To give just two examples of missing new techniques, Smooth Grad-CAM++ (Omeiza et al.) or even Grad-CAM++ (Chattopadhay et al.) would presumably obtain better performances. Moreover, Smooth Grad-CAM++ allows to target a particular feature map or even a specific neuron, which makes it even more relevant to this work.
Finally, a note about efficiency. Generally speaking, I agree that it is always good to be more efficient. However, I fail to see the high importance given to efficiency for this particular problem. Sure, gradient-based approaches are probably not suitable to online, in-network applications, but is it an important requisite? Computing saliency maps on a subset of the dataset once a few epochs already gives a good idea of what the network is doing. In any case, since these saliancy maps are intended for human use, I am not convinced about the importance of computing them for each training example at each epoch. Overall, in my opinion, being efficient at generating saliency maps is a nice to have, but not much more.
Some general comments:
- In Sec. 2.2, the last "con" seems a bit out of place. This could be applied to pretty much anything.
- Sec. 2.3 is interesting, but the explanations are convoluted. In essence, what should be said is 1) value encompasses the importance of a pixel, 2) saturation presents the max-min of the distribution and 3) hue shows the position in the network. Also, superimposing these HSV maps over gray scale version of the image like is Fig.2 is difficult to analyze because the "gray" of the image can be confused with the saturation channel.
- On p.2, "this is proceeded" -> "this is preceded"?
In summary, this paper presents a nice way of generating saliency maps from activations inside a network. However, the comparison to other approaches does not show a clear advantage, and the related work (and experiments) lacks recent techniques. I would thus ask this general question: what is the "selling point" of this method? The current focus on efficiency does not convince me. That being said, there are no clear flaws in the paper, so I am open to reconsider my assessment if improvements are made to the paper (better descriptions, comparison with more recent techniques, experimental justification for SMOE instead of simpler approaches, etc.).
On a final note, there is also the Score-CAM approach (Wang et al.) that looks similar (in the idea of using activation maps). I did not consider it in this review since it was published a few weeks ago on Arxiv, but it could be interesting to discuss it nevertheless.