Feature selection problems have been extensively studied in the setting of linear estimation, for instance LASSO, but less emphasis has been placed on feature selection for non-linear functions. In this study, we propose a method for feature selection in non-linear function estimation problems. The new procedure is based on directly penalizing the 0 norm of features, or the count of the number of selected features. Our 0 based regularization relies on a continuous relaxation of the Bernoulli distribution, which allows our model to learn the parameters of the approximate Bernoulli distributions via gradient descent. The proposed framework simultaneously learns a non-linear regression or classification function while selecting a small subset of features. We provide an information-theoretic justification for incorporating Bernoulli distribution into our approach. Furthermore, we evaluate our method using synthetic and real-life data and demonstrate that our approach outperforms other embedded methods in terms of predictive performance and feature selection.

The author rebuttal sufficiently addresses my concerns, so I am upgrading my score.
***
The paper considers the problem of embedded feature selection for supervised learning with nonlinear functions. A feature subset is evaluated via the loss function in a "soft" manner: a fraction of an individual feature can be "selected". Sparsity in the feature selection is enforced via a relaxation of l0 regularization. The resulting objective function is differentiable both in the feature selection and learned function making (simultaneous) gradient-based optimization possible. A variety of experiments in several supervised learning tasks demonstrates that the proposed method has superior performance to other embedded and wrapper methods.
My decision is to reject, but I'm on the fence regarding this paper. I'm not clearly seeing the motivation for an embedded feature selection method for neural network models: for the datasets considered in the paper, it would seem that training a nonlinear model that used all the features would result in performance at least as good as training the nonlinear model with a prepended STG layer. Perhaps there is evidence that filtering features, e.g., irrelevant features, results in higher accuracy and that the prepended STG layer achieves this accuracy, but that evidence is missing from the paper. Also, there could be downstream computational savings, e.g., at prediction time, if the dimension was very large, but this is not the setting tested in the experiments. I suppose interpretability could be considered motivation, but, even so, isn't there at least one simpler, deterministic approach (described below) that also "solves" the problem? Finally, it isn't clear how the method scales with increasing sample size and dimension as all the datasets tested are relatively small in these respects.
***
Questions and suggestions related to decision:
* The performance values using all features should be included in the experimental results so that the value added by STG can be assessed.
* Why not use the simpler deterministic and differentiable relaxation z = \sigma(\mu), where \sigma() is a "squashing" function from the real numbers to [0,1] applied element-by-element to the vector \mu? What specifically is/are the advantage(s) that the randomness in the definition of z at the bottom of pg. 3 provide over this deterministic alternative?
* Though well-described and methodologically rigorous, the experimental comparison is none-the-less a little disappointing: one dataset for classification and half the datasets for regression are synthetic and low-dimensional. The remaining regression datasets are real but also low-dimensional. The survival analysis dataset is also low-dimensional (as described in the supplementary material). This leaves one real classification dataset which was on the order of 20,000 examples and 2500 features. Why were larger sample-size and dimensionality datasets not tested? These should be readily available. For example, the gisette dataset from the NIPS 2003 feature selection challenge has 5000 features. See "MISSION: Ultra Large-Scale Feature Selection using Count-Sketches" by Aghazadeh & Spring et al. (2018) for other high-dimensional datasets. Even a single run for each large dataset would have provided some evidence of scalability.
***
Other minor comments not related to decision:
* "Concrete Autoencoders for Differentiable Feature Selection and Reconstruction" by Abid et al. (2019) targets unsupervised feature selection but has enough similarities in the approach that it should be considered related work.
* [Typo?] The unnumbered equation after (5) should not have a sum over d in the second term. Perhaps a sum over k was intended? Also, in this equation, the gradient of the loss wrt/ z samples, average of gradients over z samples times..., does not seem to match what the gradient would be given the algorithmic description in the supplementary material, a gradient of the (sample) average z times...
* The abstract states the paper is proposing a method for high-dimensional feature selection, but all of the experiments have datasets with max. dimensionality 2538.
* Some discussion of how the regularization parameter can be selected by a user of the proposed method would be good to include.


There are concerns that neural language models may preserve some of the stereotypes of the underlying societies that generate the large corpora needed to train these models. For example, gender bias is a significant problem when generating text, and its unintended memorization could impact the user experience of many applications (e.g., the smart-compose feature in Gmail). In this paper, we introduce a novel architecture that decouples the representation learning of a neural model from its memory management role. This architecture allows us to update a memory module with an equal ratio across gender types addressing biased correlations directly in the latent space. We experimentally show that our approach can mitigate the gender bias amplification in the automatic generation of articles news while providing similar perplexity values when extending the Sequence2Sequence architecture.

This paper focuses on the timely and important topic of unintended social bias in language models and proposes an approach to mitigate gender bias in the automatic generation of news articles. The automatic text generation is done by a sequence-to-sequence-model using word embeddings. The pattern of these embeddings in an multi-dimensional space can be associated with the stereotypes found in the corpora the models are trained on. To mitigate this bias the authors combine the network with a memory module storing additional gender information. This approach is inspired by the memory module used by Kaiser et al. (2017) to remember rare events, replacing the vector to track the age of items with a key representing the gender.
The proposed network architecture ("Seq2Seq+FairRegion") is evaluated using the bias score presented by Zhao et al. (2017) and compared to the sequence-to-sequende-models "Seq2Seq2" by Sutskever, Vinyals and Lee (2014) and "Seq2Seq+Attention" by Bahdanau, Cho and Bengio (2015).
Besides some typographical errors ("work embedding") the paper is overall well written but unfortunately the flow of the text could be improved to make it easier to follow. The authors should perhaps consider to explain the structure of the paper in the introduction. There are some errors according the brackets of the in-text citations which should be corrected.
The two figures both illustrate the memory module and could be easily combined to one because the second figure just adds the encoding stage of the architecture. The caption of figure 1 could be used to further explain the "Fair Region" in the text (section 2). In section 4 the authors explain how to compute the bias score. Because these calculations are used to evaluate their methods, this section should perhaps be a subsection to section 5 ("experiments").
To evaluate the proposed method the authors use newspaper articles from Chile, Peru and Mexiko and compare the bias amplification metric for the trained models. The results are shown for the datasets from Peru and Mexico (Chile is missing) with a larger bias amplification for the Peru dataset. It could be justified more clearly why the experiments compare the bias amplification scores of different countries while the main focus of the paper is to mitigate gender bias in general.
While this is an important topic, the contribution is rather minor and not well presented. Specifically the experiements should be extended.


Compressed forms of deep neural networks are essential in deploying large-scale computational models on resource-constrained devices. Contrary to analogous domains where large-scale systems are build as a hierarchical repetition of smallscale units, the current practice in Machine Learning largely relies on models with non-repetitive components. In the spirit of molecular composition with repeating atoms, we advance the state-of-the-art in model compression by proposing Atomic Compression Networks (ACNs), a novel architecture that is constructed by recursive repetition of a small set of neurons. In other words, the same neurons with the same weights are stochastically re-positioned in subsequent layers of the network. Empirical evidence suggests that ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks (88× to 1116× reduction) with only a fractional deterioration of classification accuracy (0.15% to 5.33%). Moreover our method can yield sub-linear model complexities and permits learning deep ACNs with less parameters than a logistic regression with no decline in classification accuracy.

This paper proposes a new way to create compact neural net, named Atomic Compression Networks (ACN). An immediate related work is LayerNet, where a deep neural net is created by replicating the same layer. Here, this paper extends replication down to the neuron level.
I am leaning towards rejecting this paper because the experimental setup is not well justified and a few important details are missing before conclusions can be drawn. I would like to ask a few clarification questions. Depending on the authors’ answers, I might be willing to adjust my rating.
(1) Is there missing a delta in the first half of line 6 in Algorithm 1?
(2) Throughout the experiments, for the same hyperparameter (e.g. Table 4 in A.2) do you run Algorithm 1 more than once and select the best sample architecture? If the answer is yes, summarizing all masks as one parameter will not be reasonable. Given a yes answer, I would also like to ask if the same number of samples have been considered for FC (for the same hyperparameter).
(3) Is there any intuition behind why FC does a much worse job of fitting curves than ACN with much less parameters? This refers to Fig. 2, if we compare FC with 41 parameters to ACN with 18 parameters. I am confused because MSE on sampled points often goes down when we increase the number of parameters for the application of curve fitting.
(4) Convolution can be thought of as a special case of ACN. ConvNet is the default architecture for working on image datasets. Since MNIST and CIFAR are considered, why not also compare to ConvNet?
(5) The claims that “ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks with only a fractional deterioration of classification accuracy” is quite misleading. Given fully-connected neural networks achieve up to 528 times with also a fractional deterioration (Sec. 4.3), by presumably having a shallower architecture.


A new variational autoencoder (VAE) model is proposed that learns a succinct common representation of two correlated data variables for conditional and joint generation tasks. The proposed Wyner VAE model is based on two information theoretic problems-distributed simulation and channel synthesis-in which Wyner's common information arises as the fundamental limit of the succinctness of the common representation. The Wyner VAE decomposes a pair of correlated data variables into their common representation (e.g., a shared concept) and local representations that capture the remaining randomness (e.g., texture and style) in respective data variables by imposing the mutual information between the data variables and the common representation as a regularization term. The utility of the proposed approach is demonstrated through experiments for joint and conditional generation with and without style control using synthetic data and real images. Experimental results show that learning a succinct common representation achieves better generative performance and that the proposed model outperforms existing VAE variants and the variational information bottleneck method.
shared common randomness. (See Fig. 1 (c,d).) In this sense, the joint distribution q(x, y) and the conditional distributions q(y|x), q(x|y) have the same common information structure characterized by the optimization problem (1), which involves learning the joint distribution in its nature. We call the joint encoder q(z|x, y) (or equivalently, the corresponding random variable Z) as the common representation of (X, Y), and the mutual information I q (X, Y; Z) then can be viewed as a measure of the complexity of Z.
The goal of this paper is to propose a new probabilistic model based on these information theoretic observations, to achieve a good performance in joint and conditional generation tasks. We apply the idea of learning succinct common representation to design a new generative model for a pair of correlated variables: seeking a succinct representation Z in learning the underlying distribution based on its sample may also help reduce the burden on the decoder's side and thereby achieve a better generative performance.
The rest of the paper gradually develops our framework as follows. We first define a probabilistic model based on the motivating problems, which we aim to train and use for generation tasks (Section 2.1), and then establish a general principle for learning the model based on the optimization problem (1) (Section 2.2). We propose one instantiation of the principle with a standard variational technique by introducing additional encoder distributions (Section 2.3). The proposed model with its training method can be viewed as a variant of variational autoencoders (VAEs) Rezende et al., 2014), and is thus called Wyner VAE. (See Appendix A for a brief introduction on VAEs.) The new encoder components introduced in Wyner VAE allow us to decompose a data vector into the common representation and the local representation, which can be used for sampling with style manipulation (Section 2.4). We carefully compare our model and show its advantages over the existing VAE variants (Vedantam et al., 2018;Suzuki et al., 2016;Sohn et al., 2015;Wang et al., 2016) and the information bottleneck (IB) principle (Tishby et al., 1999) (Section 3), which is a well-known information theoretic principle in representation learning. In the experiments, we empirically show the utility of our model in various sampling tasks and its superiority over existing models and that learning a succinct common representation achieves better generative performance in generation tasks (Section 4).

This paper presents a method for learning latent-variable models of paired data that decomposes the latent representation into common and local representations. The approach is motivated by Wyner’s common information, and is made tractable through a variational formulation. On experiments with MoG, MNIST, SVHN, and CelebA, the paper shows that penalizing the complexity of the common representation can improve style-content disentanglement and conditional sampling.
While I found the formulation and motivation from Wyner’s common information and Cuff’s channel synthesis interesting, the resulting model and experiments were unconvincing. There are many terms in the Wyner model proposed, and there are no ablations demonstrating which terms are important and which are not (e.g. do you need both L_xx and L_xy?). On the toy MoG experiment, the common information is known but trained models do not recover the right amount of information. For representation learning, accuracy decreases as the penalty on the common information increases, and for NLL, the joint and conditional NLL is often similar to existing work (CVAE and JVAE). The main win appears to be for style-content disentanglement, but the results there are qualitative and often change the content when only style is changed. It’s also puzzling as to why CVAE overfits so severely in a subset of the experiments. Without a more thorough evaluation of what terms in the loss matter, and showing that the technique recovers something like Wyner’s common information (by extracting the right information on a toy model), I cannot recommend this paper for acceptance.
Minor comments:
* Eqn 1: the constraint “Subject to X-Z-Y” is confusing. Do you mean X-Z-Y is a Markov network (undirected) and not a Markov chain (directed), in which case X -> Z <- Y does not correspond to X-Z-Y? (This is addressed in Eqn 3-4, but should be fixed here)
* How is Wyner’s common information related to the multivaraite mutual information I(X; Z; Y)?
* When describing distributed simulation, please include the variables and objective like you do for common information
* s/Markovity condition/independence assumption?
* Why are both losses in Eqn 4 and Eqn 5 needed? Couldn’t you use either to train q(z|x)?
* Eqn 10 (and most of your objectives) are still intractable due to the q(x)/q(x,y) terms, you should note that it is constant and dropped from the objective
* “Style control”: could you define what you mean by style in this context? Prior work could likely also do “style control” by e.g. interpolating subsets of dimensions.
* Related work: https://openreview.net/forum?id=rkVOXhAqY7 (CEB) that may result in a similar objective as Wyner’s common info
* When comparing to JVAE/JMVAE, it seems like the main difference is suing a latent-variable in the decoder, but the framework is still the same. It’d be useful to spend more time comparing/contrasting with this prior work.
* Fig 4: would be useful to have a picture of the samples (right now they’re just in appendix)
* Fig 4: in the data generating process, I believe I((X, Y); Z) = ln(5) = 1.6 nats, but none of your models converge to rates around there. Why not?
* Unlike other approaches (JVAE, CVAE) you have additional terms in your loss for conditional prediction at training time. It seems like these may be giving you gains, and it’d be useful to perform ablations over the terms in Eqn 15 to figure out what is impacting performance in Fig 4. E.g. if you set \alpha_{x -> y} and \alpha{y -> x} to 0
* Why do the CVAE models overfit? Have you tried optimizing the variational parameters on the test set if it’s due to amortization gap?
* Fig 5b: what’s the variance you’re plotting? In representation learning, we often just care about downstream accuracy, and it looks like \lam = 0  performs best there.
* Table 3: isn’t this showing that \lam = 0 works the best for joint NLL and close to best for conditional NLL on the MoG task? What’s the discrepancy with Fig 5 where conditional NLL is highly dependent on \lambda for MNIST?
* Fig 6 (a1-f1): for this MNIST add 1 dataset, the only common info should be the label. But it looks like the labels do change for non-zero values of lambda (b1 there’s a 2 -> 7), c1 1 -> 9)
* Table 4: would be useful to train CVAE yourself, as the small differences in numbers could just be due to tuning / experimental setup. You also should bold everything that’s within stderr (i.e. \lambda = 0 and \lambda = 0.15 are equally good)
* Fig 9: would be useful to include CVAE, and Wyner VAE in this plot, i.e. is it only using the mode information in the latent variable when doing conditional sampling?
* CelebA results look interesting, but there’s been other work on generation from attributes that presents more visually compelling results, e.g. https://arxiv.org/abs/1706.00409


Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient. Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in theory, be computed efficiently, and at the same time as the gradient. While these quantities are of great interest to researchers and practitioners, current deep-learning software does not support their automatic calculation. Manually implementing them is burdensome, inefficient if done naïvely, and the resulting code is rarely shared. This hampers progress in deep learning, and unnecessarily narrows research to focus on gradient descent and its variants; it also complicates replication studies and comparisons between newly developed methods that require those quantities, to the point of impossibility. To address this problem, we introduce BACKPACK 1 , an efficient framework built on top of PYTORCH, that extends the backpropagation algorithm to extract additional information from first-and second-order derivatives. Its capabilities are illustrated by benchmark reports for computing additional quantities on deep neural networks, and an example application by testing several recent curvature approximations for optimization.

This is a good paper. The authors present a software implementation which allows one to extend PyTorch to compute quantities that are irritatingly difficult to compute in PyTorch directly, or in other automatic differentiation frameworks, particularly if efficiency is a concern. Issues of this kind have been discussed at length within the community, particularly on GitHub, and related issues with optimization of automatic differentiation code have motivated other software developments, such as Julia's Zygote package. Having wasted a large amount of my time implementing the WGAN gradient penalty from scratch - which, to implement scalably, requires one to use both forward-mode and reverse-mode automatic differentiation simultaneously - I appreciate what the authors are trying to do here to make research that requires more sophisticated automatic differentiation more accessible.
---
Detailed remarks below.
* The paper's title is not very informative about what the paper is about. The authors should choose a different title - perhaps something like "BackPACK: user-friendly higher-order automatic differentiation in deep networks" or something similar but less long.
* The authors focus on KFAC-type methods as their key illustrated use case, but actually software frameworks like this are also helpful for certain GAN losses - WGAN-GP in particular. These losses require one to compute a term that involves the norm of a certain gradient of the output. The gradient of this gradient can be handled efficiently with Hessian-Vector products, which in principle are computable efficiently via automatic differentiation, but in practice a huge pain because of the need to use both forward-mode and reverse-mode automatic differentiation and lack of first-class support from automatic differentiation packages. Provided I've not misunderstood BackPACK and that it would help in making such computations less tedious (and I can't verify this myself, as my own implementation of WGAN-GP was not in PyTorch), I would highly recommend the authors to add an extra paragraph discussing this particular use case, because this would increase the paper's impact on the community by connecting it to another literature which is not mentioned in the paper.
* The entire paper could do with talking about backpropagation less, and automatic differentiation more, because it illustrates that the concerns addressed are not solely limited to deep networks, even if the package does primarily target them.
* P2: these issues are not limited to the Python community, and specialized automatic differentiation software has also been developed for Julia. The authors should cite Innes [1,2] and related papers from that literature.
* Figure 1: from the sample code, I worry about how generic BackPACK is. I think the package authors should be careful not to specialize too much to particular kinds of deep networks, particularly since a much wider variety of models and network architectures are starting to be used with automatic differentiation.
* P2: capital \Omega notation is confusing, please replace with capital \Theta.
* P2: L_2 regularization should more technically be \ell^2 instead.
* P4: please cite Baydin [3] who provides a very nice review of automatic differentiation. It may help explanation to introduce dual numbers, which make forward-mode much easier to understand.
* P6: please write out "with respect to" for "w.r.t.".
* P7: I really liked this section. The simplicity of implementing the example method using the authors' software framework feels compelling to me. However, genericness is still a concern: by analogy, every deep learning framework can do MNIST easily, but some of them make it much harder to do customized or advanced implementation than others. The latter cases are often the ones that matter to practitioners. It's hard to tell how easy it will be to implement something the authors did not foresee or consider - but this will necessarily be the case in any software paper.
* P8: "and in part driven" - missing a comma.
* P8: please spell out "Table" in "Tab. 1".
[1] M. Innes. Don't Unroll Adjoint: Differentiating SSA-Form Programs. NeurIPS, 2018.
[2] M. Innes. Flux: Elegant Machine Learning with Julia. Journal of Open Source Software, 2018.
[3] Baydin, Atilim Gunes and Pearlmutter, Barak A and Radul, Alexey Andreyevich and Siskind, Jeffrey Mark. Automatic differentiation in machine learning: a survey. JMLR, 2017.


Discovering causal structure among a set of variables is a fundamental problem in many empirical sciences. Traditional score-based casual discovery methods rely on various local heuristics to search for a Directed Acyclic Graph (DAG) according to a predefined score function. While these methods, e.g., greedy equivalence search, may have attractive results with infinite samples and certain model assumptions, they are less satisfactory in practice due to finite data and possible violation of assumptions. Motivated by recent advances in neural combinatorial optimization, we propose to use Reinforcement Learning (RL) to search for the DAG with the best scoring. Our encoder-decoder model takes observable data as input and generates graph adjacency matrices that are used to compute rewards. The reward incorporates both the predefined score function and two penalty terms for enforcing acyclicity. In contrast with typical RL applications where the goal is to learn a policy, we use RL as a search strategy and our final output would be the graph, among all graphs generated during training, that achieves the best reward. We conduct experiments on both synthetic and real datasets, and show that the proposed approach not only has an improved search ability but also allows for a flexible score function under the acyclicity constraint.

In this paper, the authors propose an RL-based structure searching method for causal discovery. The authors reformulate the score-based causal discovery problem into an RL-format, which includes the reward function re-design, hyper-parameter choose, and graph generation. To my knowledge, it’s the first time that the RL algorithm is applied to causal discovery area for structure searching.

The authors’ contributions are:
(1) re-design the reword function which concludes the traditional score function and the acyclic constraint
(2) Theoretically prove that the maximizing the reward function is equivalent to maximizing the original score function under some choices of the hyper-parameters.
(3) Apply the reinforce gradient estimator to search the parameters related to adjacency matrix generation.
(4) In the experiment, the authors conduct experiment on datasets which includes both linear/non-linear model with Gaussian/Non-gaussian noise.
(5) The authors public their code for reproducibility.

Overall, the idea of this paper is novel, and the experiment is comprehensive. I have the following concerns.

(1) In page 4 Encoder paragraph, the authors mention that the self-attention scheme is capable of finding the causal relationships. Why? In my opinion, the attention scheme only reflects the correlation relationship. The authors should give more clarifications to convince me about their beliefs.

(2) The authors first introduce the h(A) constraint in eqn. (4), and mentioned that only have that constraint would result in a large penalty weight. To solve this, the authors introduce the indicator function constraint. What if we only use the indicator function constraint? In this case, the equivalence is still satisfied, so I am confused about the motivation of imposing the h(A) constraint.

(3) In the last paragraph of page 5, why the authors adjust the predefined scores to a certain range?

(4) Whether the acyclic can be guaranteed after minimizing the negative reward function (the eqn.(6))? I.e., After the training process, whether the graph with the best reward can be theoretically guaranteed to be acyclic?

(5) In section 5.3, the authors mention that the generated graph may contain spurious edges? Whether the edges that in the cyclic are spurious? Whether the last pruning step contains pruning the cyclic path?

(6) In the experiment, the authors adopt three metrics. For better comparison, the author should clarify that: the smaller the FDR/SHD is, the better the performance, and the larger the TPR is, the better the performance.
(7) From the experimental results, the proposed method seems more superiors under the non-linear model case. Why? Could the authors give a few sentences about the guidance of the model selection in the real-world? i.e., when to select the proposed RL-based method? And under which case to choose RL-BIC, and which case to selection RL-BIC2?

(8) What’s training time, and how many samples are needed in the training process?

Minor:
1. In the page 4 decoder section, the notation of enc_i and enc_j is not clarified.
2. On page 5, the \Delta_1 and \Delta_2 are not explained.
3. For better reading experience, in table 1,2,3,4, the authors should bold value that has the best performance.


Neural sequence-to-sequence models are at the basis of state-of-the-art solutions for sequential prediction problems such as machine translation and speech recognition. The models typically assume that the entire input is available when starting target generation. In some applications, however, it is desirable to start the decoding process before the entire input is available, e.g. to reduce the latency in automatic speech recognition. We consider state-of-the-art "wait-k" decoders, that first read k tokens from the source and then alternate between reading tokens from the input and writing to the output. We investigate the sensitivity of such models to the value of k that is used during training and when deploying the model, and the effect of updating the hidden states in transformer models as new source tokens are read. We experiment with German-English translation on the IWSLT14 dataset and the larger WMT15 dataset. Our results significantly improve over earlier state-of-the-art results for German-English translation on the WMT15 dataset across different latency levels.

This paper extends the idea of prefix-to-prefix in STACL and proposes two different variations. The authors did some interesting experiments between caching and updating decoder.
My questions are as follows:
1) in section 3.1.2, the authors mentioned two adaptations. Is the proposed AR encoder uni-directional? If the AR encoder is uni-directional, then I would be surprised that the uni-directional encoder outperforms the bi-directional encoder in the original wait-k model. For the second bullet, I think the original wait-k also did the same thing(they mentioned this in the paper clearly). So there is nothing new about bullet 2.
2) the idea mentioned in fig 1 is very similar to [1]. I suggest the authors compare with the aforementioned methods.
3) updating the hidden state of the decoder introduces more complexity during the inference time. I recommend the authors to perform some analysis about decoding time with CPU and GPU.
4) it is also interesting to show more comparison between different models' training time with the original STACL.
[1] Zheng et al. "Simultaneous Translation with Flexible Policy via Restricted Imitation Learning" ACL 2019


Generative modelling is often cast as minimizing a similarity measure between a data distribution and a model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of a potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required to find the optimal potential. In this work, we study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, in particular, the c-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the c-transform allows for a more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not perform the best in the generative setting.
values. The discriminators are functions from the sample space to the real line, that have different interpretations in different GAN variations. The main technical issue is that the discriminators have to satisfy specific conditions, such as being 1-Lipschitz in the 1-Wasserstein case. In the original paper, this was enforced by clipping the weights of the discriminator to lie inside some small box, which, however, proved to be inefficient. The Gradient penalty WGAN (WGAN-GP) (Gulrajani et al., 2017) was more successful at this, by enforcing the constraint through a gradient norm penalization. Another notable improvement was given by the consistency term WGAN (CT-WGAN) (Wei et al., 2018), which penalizes diverging from 1-Lipschitzness directly. Other derivative work of the WGAN include different OT inspired similarity measures between distributions, such as the sliced Wasserstein distance (Deshpande et al., 2018), the Sinkhorn divergence (Genevay et al., 2018) and the Wasserstein divergence (Wu et al., 2018). Another line of work studies how to incorporate more general ground cost functions than the l 2 -metric (Adler & Lunz, 2018;Mallasto et al., 2019;Dukler et al., 2019).
Recent works have studied the convergence of estimates of the Wasserstein distance between two probability distributions, both in the case of continuous (Klein et al., 2017) and finite (Sommerfeld & Munk, 2018;Sommerfeld, 2017) sample spaces. The decay rate of the approximation error of estimating the true distance with minibatches of size N is of order O(N −1/d ) for the Wasserstein distances, where d is the dimension of the sample space (Weed & Bach, 2017). Entropic regularized optimal transport has more favorable sample complexity of order O(1/ √ N ) for suitable choices of regularization strength (see Genevay et al. 2019 andalso Feydy et al. 2018;Mena & Weed 2019). For this reason, entropic relaxation of the 1-Wasserstein distance is also considered in this work.
Contribution. In this work, we study the efficiency and stability of computing the Wasserstein metric through its dual formulation under different schemes presented in the WGAN literature. We present a detailed discussion on how the different approaches arise and differ from each other qualitatively, and finally measure the differences in performance quantitatively. This is done by quantifying how much the estimates differ from accurately computed ground truth values between subsets of commonly used datasets. Finally, we measure how well the distance is approximated during the training of a generative model. This results in a surprising observation; the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting.

[Summary]
This paper provides an empirical evaluation of commonly used discriminator training strategies in estimating the Wasserstein distance between distributions. The paper finds that methods motivated from optimal transport theory, e.g. c-transform and (c,\eps)-transform, perform better in evaluating the Wasserstein distance than methods commonly used in WGAN practice such as weight clipping and gradient penalty. However, when deployed in WGANs as the discriminator training strategy, these methods do not generate images as high-quality as the gradient penalty method.
[Pros]
The question considered in this paper, i.e. how well does various discriminator strategies (as proxies of the infeasible all 1-Lipschitz discriminator) perform for *evaluating* the Wasserstein distributions, is important for strengthening our understanding of generative models. The main result that methods that are better at computing W may not be better at generating images is interesting, and agrees with the theoretical insight (e.g. Arora et al. 2017) that *non-parametric* minimization of W may not be a good explanation for the generative power of WGANs.
[Cons]
I have concerns about the specific setting of “mini-batch distance” considered in this paper, in that whether it is really a sensible task (and can say anything about computing W) given the curse of dimensionality of estimating W from samples. The paper did not really discuss this issue, and from my own thoughts I don’t think the task avoids this issue.
From my understanding, the “Approximation” experiment does the following:
(1) Set (\mu, \nu) to be a random split of a dataset and consider them to be the populations. Let’s let f_\star denote the (ground truth) optimal discriminator between (\mu, \nu).
(2) Train discriminators (f_wc, f_gp, f_c, f_ceps) (using the different algorithms) from training batches from (\mu, \nu).
(3) Evaluate <f, \mu’_l - \nu’_l> where (\mu’_l, \nu’_l) are fresh test batches from (\mu, \nu) and f is one of the above trained discriminators.
In comparison, the “ground truth” computes W(\mu’_l, \nu’_l) = <f_l, \mu’_l, \nu’_l> from the POT package (though not necessarily through explicitly computing f_l.)
The issue with this is that we expect the method to perform well if f ~= f_l, which can be achievable if all the f_l’s are similar (and hopefully they’re all approximately equal to f_\star.) However I don’t think this is true -- as (\mu’_l, \nu’_l) are samples, and because of the curse of dimensionality, we should expect the f_l’s to be quite different from each other. (Otherwise if they’re really just ~= f_\star, then we can use standard concentration to show the W(\mu’_l, \nu’_l) ~= W(\mu, \nu), which we know is not true from curse of dim.)
Given this, I don’t think the task of comparing <f, \mu’_l, \nu’_l> with the POT results really says anything about their power in computing W. I would be glad though to hear back from the authors to see if my understanding is accurate, and adjust my evaluation from there.


Feature Map Transform Coding for Energy-Efficient CNN Inference
Convolutional neural networks (CNNs) achieve state-of-the-art accuracy in a variety of tasks in computer vision and beyond. One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements. The latter often dominates the energy footprint on modern hardware. In this paper, we introduce a lossy transform coding approach, inspired by image and video compression, designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results. Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps, which are highly correlated, with variable length coding. Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2. We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% in the memory energy footprint, compared to quantized network, with negligible impact on accuracy. When allowing accuracy degradation of up to 2%, the reduction of 60% is achieved. A reference implementation accompanies the paper.

A lossy transform coding approach was proposed to reduce the memory bandwidth of edge devices deploying CNNs. For this purpose, the proposed method compresses highly correlated feature maps using variable length coding. In the experimental analyses, the proposed method outperforms some of the previous work in terms of the compression ratio and accuracy for training ResNet-34 and MobileNetV2.
The proposed method and initial results are promising. However, the paper and the work should be improved for a clear acceptance:
- Some parts of the method need to be explained more clearly:
– In the statement “Due to the choice of 1 × 1 × C blocks, the PCA transform essentially becomes a 1 × 1 tensor convolution kernel”, what do you mean by “the PCA transform becomes a convolution kernel.”?
- Could you please further explain how you compute PCA using batch data, how you update online and how you employ that in convolution weights together with BN? Please also explain the following in detail:
(I) “To avoid this, we calculate the covariance matrix layer by layer, gradually applying the quantization.” What is the quantization method you applied, and how did you apply it gradually?
(II) “The PCA matrix is calculated after quantization of the weights is performed, and is itself quantized to 8 bits.” How did you quantize the weights, how did you calculate PCA using quantized weights and how did you quantize them to 8 bits?
- Could you please explain the following settings, more precisely: direct quantization of the activations; quantization of PCA coefficients; direct quantization followed by VLC; and full encoder chain comprising PCA, quantization, and VLC? Please note that there are various methods and algorithms which can be used for these quantization steps. Therefore, please explain your proposed or employed quantization methods more clearly and precisely.
–  Please clarify the statement “This projection helps to concentrate most of the data in part of the channels, and then have the ability to write less data that layers.”.
- Did you apply your methods to larger networks such as larger ResNets, VGG like architectures, Inception etc?
- I also suggest you to perform experiments on different smaller and larger datasets, such as Cifar 10/100, face recognition datasets etc., to examine generalization of the proposed methods at least among different datasets.


Motivated by the human's ability to continually learn and gain knowledge over time, several research efforts have been pushing the limits of machines to constantly learn while alleviating catastrophic forgetting (Kirkpatrick et al., 2017b). Most of the existing methods have been focusing on continual learning of label prediction tasks, which have fixed input and output sizes. In this paper, we propose a new scenario of continual learning which handles sequence-to-sequence tasks common in language learning. We further propose an approach to use label prediction continual learning algorithm for sequence-to-sequence continual learning by leveraging compositionality (Chomsky, 1957). Experimental results show that the proposed method has significant improvement over state of the art methods. It enables knowledge transfer and prevents catastrophic forgetting, resulting in more than 85% accuracy up to 100 stages, compared with less than 50% accuracy for baselines in instruction learning task. It also shows significant improvement in a machine translation task. This is the first work to combine continual learning and compositionality for language learning, and we hope this work will make machines more helpful in various tasks.

This paper proposes an approach for continual learning for applications in sequence-to-sequence continual learning (S2S-CL). More specifically, the paper addresses the problem of growing vocabulary. The overall approach is intuitive yet quite effective. The method assumes a split between the syntax (f) and semantics (p) of the sequence -- in other words, each token is associated with two labels. Furthermore, the syntax is assumed to be shared across all the training sequences and the sequences that are not encountered during the initial training. A sequence model (LSTM) is used for learning the syntax f over the initial training and is then frozen for all the downstream tasks (i.e. continual learning). The sequence model predicts the correspondence between the output token and input tokens. Another network (a 1-layer  MLP) is used to predict the semantic label of the selected token in the target domain. Barring some notational confusion, I believe the method is very reasonable and should work well. They perform experiments on 2 datasets (Instruction learning & machine translation) in 3 continuous learning paradigms with varying difficulties and demonstrate that the proposed method significantly outperforms the baselines by an impressive margin.
I am leaning towards accepting this paper since I believe the proposed approach can be a promising direction for continual learning in S2S settings and the empirical results are convincing.
However, I have some concerns that might require clarification or additional experiments:
    1. The novelty of the proposed method is somewhat limited in my opinion. It seems it is a very simple adaptation of Li et al. 19 and the only addition to that work is the usage of a very simple label prediction scheme by fixing everything else. However, this might be okay since the experiments are quite compelling and the method is applied to a different setting.
    2. Notations and language in section 3.2-3.4 are hard to follow:
        - At the bottom of page 4, I believe some of the equations are not correct. For example, the first term of line 2 should be P(Y^f | X^f, X^p) rather than Y^p?
        - In the second equation of page 6, the second term should be v_j = p’ \cdot a_j
        - In Figure 1, k_r, E_r, and W are never defined (although their meaning can be inferred).
        - In section 3.4, it reads that new word embedding is appended to both semantic and syntactic embedding matrices but this doesn’t make sense because the syntactic network is already fixed so it shouldn’t be able to handle new symbols; therefore, I believe only new row is added to the semantic embedding.
    3. I believe that f_predict here is parameterized by \theta and \theta is also frozen during the continual learning phase which contradicts the claim at the end of section 3.2 that only \phi is frozen. Otherwise, it’s hard to see why f_predict does not suffer from catastrophic forgetting. From the provided source code, it seems that it is indeed the case that only the new embedding is being updated. In other words, the only thing happening at this stage is that the newly added embedding are optimized to adapt to the frozen f_predict.
    4. Due to point 3, I have some doubts about how scalable this approach is if the other embedding is not allowed to co-adapt. However, perhaps this problem could be alleviated if f_predict has extremely high capacity at initialization? More on this in point 6.
    5. Assuming all my assessment above is correct, it seems that the performance of the proposed method should *not* decrease at all so I would like to see more analysis on the decreasing performance in the instruction learning task (Figure 2, left column). Is this be due to the fact that f_predict or the other embedding is not allowed to update and the newly added embedding is mapped close to existing embedding? In that case, can increasing the model capacity solve this problem? I understand the paper makes argument about regularization but I believe this warrants thorough study for gauging the significance of this approach in more realistic settings.
    6. I would like to see a discussion of the short-comings of the approach and possible ways to overcome them. For example, freezing the syntactic network seems limiting in machine translation settings if a new language, say Italian, is added. Intuitively, knowing how to translate English-French should help translating English to Italian but fixing the syntax prevents this. Another example is that prior knowledge about the syntax needs to be known about the language for labeling the f and p and this can be expensive and cannot handle words that have more than 1 usages (e.g. run can be used as a noun but also as a verb).
I am willing to increase my score to accept if the revised manuscript can address the majority of, if not all of the concerns listed above.
========================================================================================================
Minor comments that did not affect my decision:
    - These sections could greatly benefit from an overall flow chart of how everything fits together
    - It says the experiments are repeated with 5 different random seeds so why not add error bars to the plots?
    - What characteristics of the 2 tasks cause the baselines to behave so differently?


Evolutionary-based optimization approaches have recently shown promising results in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels. This paper presents a method called Deep Innovation Protection (DIP) that allows training complex world models end-to-end for such 3D environments. The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in a world model, allowing other components to adapt. We investigate the emergent representations of these evolved networks, which learn a model of the world without the need for a specific forward-prediction loss.

Review of “Deep Innovation Protection”
This paper builds on top of the World Models line of work that explores in more detail the role of evolutionary computing (as opposed to latent-modeling and planning direction like in Hafner2018) for such model-based architectures. Risi2019 is the first work that is able to train the millions of weights of the architecture proposed in Ha2018 end-to-end using modular-form of genetic algorithm (GA) to maximize the terminal cumulative reward.
This work expands on Risi2019, and proposes the use of multi-objective optimization (NSGA-II) to allow a world model to learn several subsystems by the fact that the GA optimizer is not only targeting the reward function of the environment, but it is also optimizing for diversity, similar to MAP-Elites line of literature (i.e. Cully2015, Mouret2015, and other more recent cited in this paper).
They demonstrate that end-to-end multi-objective optimization is not only able to solve both CarRacing and VizDoom tasks, but the world models learned have interesting features, i.e. the emerged world models can predict important survival events even though they were not trained explicitly for forward prediction.
While the paper is interesting, I feel in its current state, it is not the level of an ICLR conference paper. Below is some feedback I want to give the authors to help them improve the work, so hopefully it can be improved either for this conference or the next. For the record, the score I wanted to give is a 4, but since I can only choose 3, or 6, I will assign a score of 3 for this review (although should really be a 4). I will breakdown the feedback into minor and major:
Minor feedback:
- Figure 3 does not indicate which task it is solving. Is it CarRacing or VizDoom?
- What's the motivation for optimizing for both "age" and reward? Why not other multi-objectives?
Medium Feedback:
As the authors are learning world models that can predict some form of a future, can they train agents inside an open-loop environment generated by such a world model (perhaps even with some hacks), and transfer such a policy back to the real env, as done in Ha2018 for VizDoom?
Major feedback:
The contribution of this paper isn't there, compared to Risi2019 which I feel has a much larger contribution to the line of work. I'm not convinced that the approach (multi-obj + end-to-end) which solves VizDoom cannot be solved using Risi2019. Authors mentioned this in the intro, but perhaps show some convincing experiments to make sure Risi2019 will definitely fail VizDoom? I'm not really convinced that an end-to-end algorithm will fail to find a solution for a poilcy (with our without a world model component in the architecture) for VizDoom Take Cover. Honestly I don't think VizDoom Take Cover is that hard of a task, and my intuition tells me that the method outlined in Risi2019, if carefully tuned properly, will be able to solve VizDoom Take Cover (but happy to be proven wrong if there is a strong experimental argument suggesting otherwise).
I think for this paper to be convincing, the authors actually need to go beyond the CarRacing and VizDoom tasks and show the real power of multi-objective optimization for solving difficult problems that require model-based architectures. For instance, in Cully2015, the learning of several diverse sub-systems on the Pareto front allowed a robot to still be able to accomplish the walking task when the legs are disabled. I would advise the authors to look for a third task where they can clearly show that their approach can solve problems that current approaches (Ha2018, Risi2019, and possibly even DeepRL) cannot solve. Being able to do this has the potential to make this paper into an important work for the field. I can also recommend trying out some environments in the Animal Olympics (see refs) that will be very much suited for multi-obj optimization for animal-like survival environments where input space are pixels.
[Ha2018] Recurrent World Models Facilitate Policy Evolution (NeurIPS2018)
[Risi2019] Deep neuroevolution of recurrent and discrete world models (GECCO2019)
[Hafner2018] Learning latent dynamics for planning from pixels (PlaNet)
[Cully2015] “Robots that can adapt like animals” https://www.nature.com/articles/nature14422
[Mouret2015] Illuminating search spaces by mapping elites
[Animal Olympics] http://animalaiolympics.com/
*** REVISED SCORE ***
Upon looking at the revision, and the author response, I decided to increase the score of the paper (for the record: originally from a 4, which is rounded to a 4, to a 5, which is rounded to a 6).
The authors explained why Risi2019's approach cannot work on VizDoom with sufficient effort, and I buy their explanation. At the same time I feel this work is can benefit from addition, convincing tasks that highlight the full power of multi objective optimization, for this work to warrant a strong accept. Regardless of the final decision, I'm looking forward to see development of this interesting approach.


We apply the canonical forms of gradient Morse complexes (barcodes) to explore topology of loss surfaces. We present a new algorithm for calculations of the objective function's barcodes of minima. Our experiments confirm two principal observations: (1) the barcodes of minima are located in a small lower part of the range of values of loss function of neural networks and (2) increase of the neural network's depth brings down the minima's barcodes. This has natural implications for the neural network learning and the ability to generalize.

The paper aims to study the topology of loss surfaces of neural networks using tools from algebraic topology. From what I understood, the idea is to effectively (1) take a grid over the parameters of a function (say a parameters of a neural net), (2) evaluate the function at those points, (3) compute sub-levelset persistent homology and (4) study the resulting barcode (for 0/1-dim features) (i.e., the mentioned "canonical form" invariants). Some experiments are presented on extremely simple toy data.
Overall, the paper is very hard to read, as different concepts and terminology appear all over the place without a precise definition (see comments below). Given the problems in the writing of the paper, my assessment is that this idea boils down to computing persistent homology of the sub-levelset filtration of the loss surface sampled at fixed parameter realizations. I do not think that this will be feasible to do, even for small-scale real-world neural networks, simply due to the difficulty of finding a suitable grid, let alone the vast number of function evaluations involved.
The paper is also unclear in many parts. A selection is listed below:
(1) What do you mean by gradient flow? One can define a gradient flow in a linear space X and for a function F: X->R, e.g., as  a smooth curve R->X, such that x'(t) = -\nabla F(x(t)); is that what is meant?
(2) What do you mean by "TDA package"? There are many TDA packages these days (maybe the CRAN TDA package?)
(3) "It was tested in dimensions up to 16 ..." What is meant by dimension here? The dimensionality of the parameter space?
(4) The author's talk about the "minima's barcode" - I have no idea what is meant by that either; the barcode is the result of sub-levelset persistent homology of a function -> it's not associated to a minima.
(5) Is Theorem 2.3. not just a restatement of a theorem from Barannikov '94? At least the proof in the appendix seems to be .
(6) Right before Theorem 2.3., what does the notation F_sC_* mean? This needs to be introduced somewhere.
From my perspective, the whole story revolves around how to compute persistence barcodes from the sub-levelset filtration of the loss surface, obtained from function values taken on a grid over the parameters. The paper devotes quite some time to the introduction of these concepts, but not in a very clear or understandable manner. The experiments are effectively done on toy data, which is fine, but the paper stops at that point. I do not buy the argument that "it is possible to apply it [the method] to large-scale modern neural networks". Without a clear strategy to extend this, or at least some preliminary "larger"-scale results, the paper does not meet the ICLR threshold. The more theoretical part is too convoluted and, from my perspective, just a restatement of earlier results.


Deep neural networks are powerful learning machines that have enabled breakthroughs in several domains. In this work, we introduce retrospection loss to improve performance of neural networks by utilizing prior experiences during training. Minimizing the retrospection loss pushes the parameter state at the current training step towards the optimal parameter state while pulling it away from the parameter state at a previous training step. We conduct extensive experiments to show that the proposed retrospection loss results in improved performance across multiple tasks, input types and network architectures.

The paper proposes a new loss function which adds to the training objective another term that pulls the current parameters of a neural network further away from the parameters at a previous time step.
Intuitively, this aims to push the current parameters further to the local optimum.
On a variety of benchmarks, optimizing the proposed loss function achieves better results than just optimizing the training loss.
The paper is well written and easy to follow.  However, I am not entirely convinced about the intuition of the proposed method and I think further investigation are necessary.
While the method is simple and general, it also seems to be rather heuristic and requires carefully chosen hyperparameters.
Having said that, the empirical evidence shows that the proposed loss function consistently improves performance.
The following details should be addressed further:
- I am a bit confused by the definition of the loss function. In Equation 1 it seems that the term on the left represents the training objective. If that is correct than Equation 2 second case contains the training objective twice?
- F in Section 3 after Equation 2 is not properly defined
- Could it happen that the proposed loss function leads to divergence, for example if the parameter from a previous time step theta^Tp is close to the optimum theta_star?
- What is the motivation to use the L1 norm? How does this choice affect convergence compared to let's L2 norm?
- Section 4.1 typo in first paragraph: K instead of \kappa
- Section 4.1 the results would be more convincing if all networks were trained multiple times with a different random initialization and Table 1 would include the mean and std.
- Why is no warm-up period used for the GAN experiments?
- Section 4.3: why is \kappa increase by 1% for the speech recognition experiments where as by 2% for all other experiments?
- I suggest to increase the line width of all figures since they are somewhat hard to identify on a print version.
- Why is the momentum set to 0.5 for SGD in the ablation study? Most frameworks use a default value of 0.9.
- I would like to see the affect of the warm-up period to the performance in the ablation study.
- How does the choice of learning rate schedule, such as for example cosine annealing, affect the loss function?
post rebuttal
------------------
I thank the authors for clarifying my questions and providing additional experiments. I think that especially the additional ablation studies and reporting the mean and std of multiple trials make the contribution of the paper more convincing. Hence, I increased my score.


We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct confidence sets for deep neural networks with PAC guarantees-i.e., the confidence set for a given input contains the true label with high probability. We demonstrate how our approach can be used to construct PAC confidence sets on ResNet for ImageNet, and on a dynamics model the half-cheetah reinforcement learning problem.

Summary: This paper presents an approach for generating confidence set predictions from deep networks. That is, the smallest set of predictions where the true answer is included in that set. Theory is used to derive an algorithm with PAC-style bounds on the population risk.
Overall Assessment: I like the core idea of this paper---developing region-based prediction algorithms with theoretical backing by taking advantage of a small calibration dataset and simple models of uncertainty. However, there are significant clarity and evaluation issues that must be addressed before the paper is ready for publication.  In current form, I rate the paper between Weak reject and strong reject overall.
Strengths:
+ Nice general direction.
+ New algorithm for confidence set prediction with theoretical guarantees.
+ Experiments show favourable performance vs a few ablations.
Weaknesses & Questions:
1. It is unclear to me why the temperature scaling component is needed. It does not actually change the ordering of the probabilities assigned to each class, so from what I can tell all it does is change the optimal T, but not the final performance.
2. The paper presents a bound on the population risk, but the experiments do not include a comparison of the expected worst-case error rates with the empirical error rates achieved on the test set. This should be corroborated.
3. The  description for the Model-based reinforcement learning subsection is impossible to follow due to ambiguity and lack of detail overall.
3.1. Some specific confusions about this section: Paper overloads f to be both a deterministic transition function, and also a distribution over possible states. It also switches between using x_{t+1} / x_t and x^\prime / x to mean the same thing. The notation used to describe the multi-step setting also seems to use "t" for two different things: the current time-step, and some arbitrary time-step in the past.
3.2. It is unclear what metric is being used to evaluate the state transition models---L2 distance between what? Given that the predictions are not point estimates, I would expect something that takes uncertainties into account.
5. For both sets of experiments (classification and RL), there are no alternative methods used as points of reference. There are a multitude of other approaches incorporating uncertainty into predictions, as mentioned in Section 1. A trivial baseline is to heuristically generates confidence sets by trusting the probabilities produced by the model and building a set out of the classes corresponding the top 1-\epsilon probability mass should be essential. This could be improved by applying difference calibration approaches to make the probabilities more trustworthy. Model ensembles provide another easy baseline. Such simple baselines are a minimum expectation, before even getting to state of the art alternatives.
6. There seem to be no vanilla regression experiment, only the harder-to-interpret RL experiment. EG: Since we already have a vision context: If you are doing facial age estimation , or interest point tracking, could one make a PAC prediction about the true age region/interest point location?
7. Overall the paper introduction misses some explanation on the motivating scenarios where such confidence set predictions are useful. One can perhaps imagine this for vision, but some help connecting the dots to how it could be useful in regression and RL would help.
8. It is claimed that Theorem 1 provides a "better" bound than the one based on the VC dimension. What is meant by better?
Minor comments:
* There are a couple of small mistakes in the proof of theorem 1. The \tilde{X} and \tilde{Y} in the definition of \tilde{Z}_{val} are in the wrong place. The "sum of k i.i.d. random variables" should be "sum of n i.i.d. random variables".
* In general, the proof is quite hard to follow. At times it was quite unclear how you get from one step to the next, because it relies on something shown several steps early, which is not referenced.
* Notation, in general, is a bit of an issue in this paper. See comments above, but also: switching between \theta and T for the parameter used in the confidence set predictor, some confusion between T and \hat{T}.
* It is unclear how the neural network used in model-based RL predicts a PSD covariance matrix.
* \hat{T} is not defined when it is first referenced (Section 3.3).
* Algorithm 1 appears several pages before it is referenced.
-------------- POST REBUTTAL ------------
I modify my score to 6: Weak accept.


We tackle unsupervised domain adaptation by accounting for the fact that different domains may need to be processed differently to arrive to a common feature representation effective for recognition. To this end, we introduce a deep learning framework where each domain undergoes a different sequence of operations, allowing some, possibly more complex, domains to go through more computations than others. This contrasts with state-of-the-art domain adaptation techniques that force all domains to be processed with the same series of operations, even when using multi-stream architectures whose parameters are not shared. As evidenced by our experiments, the greater flexibility of our method translates to higher accuracy. Furthermore, it allows us to handle any number of domains simultaneously.

In this paper, the authors proposed to address the information asymmetry between domains in unsupervised domain adaptation. Innovatively, they resort to a multiflow network where each domain adaptatively selects its own pipeline. I quite appreciate the idea itself, while there are many essential issues to be addressed first.
Pros:
-	The way tackling the information asymmetry, or untie weights, between domains is novel and interesting.
-	The proposed network/framework can be easily extended to the multi-task setting, or multi-source/multi-target domain adaptation.
-	The paper is well-written and easy to follow.
Cons:
-	The most critical downside of this paper is its insufficient experiments to support the whole idea, where we will detail in the next.
Experimental issues:
-	Comparison with other state-of-the-art UDA methods (e.g., CDAN) is a must. This paper improves UDA in terms of adaptive parameters sharing, which is completely independent from most of the UDA contributions (including the DANN you compared) which improve the distribution alignment between feature representations. Therefore, it is imperative to compare that line of SOTA methods, otherwise why should we consider adaptative parameters sharing instead of distribution alignment? At best, the proposed multiflow network combined with the SOTA feature alignment method (e.g., CDAN other than DANN) should be considered and expected to beat CDAN itself.
-	Many ablation studies or hyperparameter sensitivity analyses are missing.
o	How do you determine the number of parallel flows, i.e., K? Is it possible that 3 or 4, more than 2 flows, are better even in the UDA between two domains?
o	Do you try any other possibilities of grouping a computational unit, and how will different configurations influence the performance?
o	Is there a possibility that none of the gates in the final layer is activated? Do you need some constraints?
-	Since the authors mentioned the potential of the multi-flow network in adaptation between multiple domains, it is necessary to investigate multi-source or multi-target domain adaptation. Only in this case may the significance of different K values be demonstrated.
-	The baseline results in Table 1 are not comparable to some reported papers, and even lower than those reported in other UDA papers.


In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy. We demonstrate that using techniques from natural language processing and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration. Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, a class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs. Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order. We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time. Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average return by 19% over a baseline that has the same number of parameters and by 9% over a stronger baseline that has far more parameters.

# UPDATE after rebuttal
I have changed my score to 8 to reflect the clarifications in the new draft.
Summary:
This paper presents a family of architectural variants for the recurrent part of an RL controller. The primary claim is that simple components which compute elementwise sum/average/max over the activations seen over time are highly robust to noisy observations (as encountered with many RL environments), as detailed with various empirical and theoretical analyses. By themselves these aggregators are incapable of storing order dependent information, but by combining an LSTM with an aggregator, and pushing half the LSTM activations through the aggregator, and concatenating the the other half of the activations with the aggregator output, the resulting output contains order dependent and order independent content.
The motivation is very clear (many of the most challenging modern video games used as RL environments clearly have noisy observations, and many timesteps for which no new useful information is observed) and the related work is comprehensive. An increase in training speed and stability, apparently without any major caveats, would be of great interest to any practitioner of Deep Reinforcement Learning.
The experiments provided are good, and vary nicely between actual RL runs and theoretical analysis, all of which convinces me that this could well become a standard Deep RL component. I do have a range of questions & requests for clarification (see below) but I believe the experiments as presented, plus some additions before camera ready, will make for a good paper of wide interest.
Desicion: Accept. I would give this a 7 if I was able to. The idea is very simple (indeed I find it slightly hard to believe no one has tried this before, but I don't know of any references myself) and the results, particularly Figure 4, are compelling. It's nice to see a very approachable more mathematical analysis as well, it would be good to see more papers proposing new neural components with this kind of rigour. I look forward to trying this approach myself, post publication.
Discussion:
AMRL-Avg coming out the best in Figure 8 makes a lot of sense to me, as I can see how average provides a stable signal of unordered information. One thing that really doesn't make sense to me is why Max and Sum would also be good - obviously their SNR / Jacobians are quite similar, but fundamentally there is a risk of huge values being built up in activations (especially with Sum), which at least have the potential to cause numerical instability, provide weird gradient magnitudes (which then mess up moment tracking optimizers like Adam, etc). Thre does not seem to be any mention of numerical stability, or whether any considerations ned to be taken for this. Maybe we could hope that 'well behaved' internal neron activations are zero mean, so the average aggregator will never get too big - but is this always the case, at all points in training, in every episode? I appreciate the straight through estimator might ameliorate this, but it is not made entirely clear to me in the text that this is the reason for using it. Addressing this point would increase the strength of the argument.
Given that DNC was determined to be the strongest baseline in a few of the metrics, and that AMRL combines these aggregators with LSTM, an experiment that I'm surprised is missing would be to make AMRL containing a DNC instead of an LSTM. Is there a reason why this wasn't attempted?
It would have been good to see a wider set of RL experiments - the Atari suite is well studied, easily available, and there are many open source implementations to which AMRL could easily be slotted into (eg IMPALA, openai baselines, etc).
The action space for Minecraft is not spelled out clearly - at first I assumed this was the recent project Malmo release, which I assume to have continuous actions (or at least, the ability to move in directions that are not compass points), but while Malmo is mentioned in the paper, the appendix implies that the action space is (north, east, west) etc. I'm aware that there is precedent in the literature (Oh et al 2016) for 'gridworld minecraft', but I think it would improve the paper to at least acknowledge this in the main text, as I feel even most researchers in 2019 would read the text and assume the game to be analogous to VizDoom / DeepMind Lab, when it really isn't. Note that most likely the proposed method would give an even bigger boost with "real" minecraft, as there are even more non-informative observations between the interesting ones, and furthermore I think the environment choice made in this paper is fine, as it still demonstrates the point. A single additional sentence should suffice.
Minor points:
NTM / DNC were not "designed with RL in mind" per se, the original NTM paper had no RL and the DNC paper contained both supervised and RL results. Potentially the statement at the bottom of page 1 was supposed to refer mainly to stacked LSTMs - either way I feel it would be better to slightly soften the statement to "...but also for stacked LSTMs and DNCs (cite), which have been widely applied in RL."
In the RL problem definition, the observation function and the state transition function are stated as (presumably?) probability distributions with the range [0,1], but for continuous state / observation spaces it is entirely possible for the probability density at a point to exceed 1.
Also in the RL section - the notation of $\tau_t \in \Omega^t$ should probably also contain the sequence of $t-1$ actions taken throughout the trajectory - this is made clear in the text, but not in the set notation.
The square bracket "slicing" notation used for slicing and concatenating is neat, but even though I spend all day writing python it still took me a while to realise what this meant, as I haven't seen this used in typeset maths before. Introducing this notation (as well as the pipe for concatenation) at first usage would help avoid tripping up readers unnecessarily.
Note that this pdf caused several office printers I tried it on to choke on later pages, and I get slow scrolling in Chrome on figures such as figure 11 - possibly the graphics are too high resolution (?)
typos: A5, feed forward section: "later" should be "layer" I believe.


There ubiquitously exist many single-objective tasks in the real world that are inevitably related to some other objectives and influenced by them. We call such task as the objective-constrained task, which is inherently a multi-objective problem. Due to the conflict among different objectives, a trade-off is needed. A common compromise is to design a scalar reward function through clarifying the relationship among these objectives using the prior knowledge of experts. However, reward engineering is extremely cumbersome. This will result in behaviors that optimize our reward function without actually satisfying our preferences. In this paper, we explicitly cast the objective-constrained task as preference multiobjective reinforcement learning, with the overall goal of finding a Pareto optimal policy. Combined with Trajectory Preference Domination we propose, a weight vector that reflects the agent's preference for each objective can be learned. We analyzed the feasibility of our algorithm in theory, and further proved in experiments its better performance compared to those that design the reward function by experts.

After Responses:
I understand the differences that authors pointed to the relevant literature. However, it is still lacking comparisons to these relevant methods. The proposed method has not been compared with any of the existing literature. Hence, we do not have any idea how does it stand against the existing approaches. Hence, I believe the empirical study is still significantly lacking. I will stick to my decision. Main reason is as follows; I believe the idea is interesting but it needs a significant empirical work to be published. I recommend authors to improve empirical study and re-submit.
-------
The submission is proposing a method for multi-objective RL such that the preference of tasks learned on the fly with the policy learning. The main idea is converting the multi-objective problem into single objective by scalar weighting. The weights are learned in a structured learning fashion by enforcing them to approximate the Pareto dominance relations.
The submission is interesting; however, its novelty is not even clear since authors did not discuss majority of the existing related work.
Authors can consult the AAMAS 2018 tutorial "Multi-Objective Planning and Reinforcement Learning" by Whiteson&Roijers for relevant papers. It is also important to note that there are other methods which learn weighting. Optimistic linear support is one of such methods. Hence, this is not the first of such approaches. Beyond RL, it is also studied extensively in supervised learning. For example, authors can see "Multi-Task Learning as Multi-Objective Optimization" from NeurIPS 2018.
The manuscript is also very hard to parse and understand. For example, Definition 2 uses but not define "p" in condition (2). Similarly, Lemma 1 states sth is "far greater" than something else. However, "far greater" is not really defined. I am also puzzled to understand the relevance of Theorem 1. It is beyond the scope of the manuscript, and also not really new.
Authors suggest a method to solve multi-objective optimization. However, there is no correctness proof. We do not know would the algorithm result in Pareto optimal solution even asymptotically. Arbitrary weights do not result in Pareto optimality.
Proposing a new toy problem is well-received. However, not providing any experiment beyond the proposed problem is problematic. Authors motivate their method using DOOM example. Why not provide experimental results on a challenging problem like DOOM?
In summary, I definitely appreciate the idea. However, it needs better literature search. Authors should position their paper properly with respect to existing literature. The theory should be revised and extended with convergence to Pareto optimality. Finally, more extensive experiments on existing problems comparing with existing baselines is needed.


Distributionally Robust Optimization (DRO) has been proposed as an alternative to Empirical Risk Minimization (ERM) in order to account for potential biases in the training data distribution. However, its use in deep learning has been severely restricted due to the relative inefficiency of the optimizers available for DRO in comparison to the wide-spread Stochastic Gradient Descent (SGD) based optimizers for deep learning with ERM. In this work, we propose SGD with hardness weighted sampling, a principled and efficient optimization method for DRO in machine learning that is particularly suited in the context of deep learning. Similar to an online hard example mining strategy in essence and in practice, the proposed algorithm is straightforward to implement and computationally as efficient as SGD-based optimizers used for deep learning. It only requires adding a softmax layer and maintaining a history of the loss values for each training example to compute adaptive sampling probabilities. In contrast to typical ad hoc hard mining approaches, and exploiting recent theoretical results in deep learning optimization, we prove the convergence of our DRO algorithm for over-parameterized deep learning networks with ReLU activation and finite number of layers and parameters. Preliminary results demonstrate the feasibility and usefulness of our approach.

This paper proposes a method for Distributionally Robust Optimization (DRO). DRO has recently been proposed (Namkoong and Duchi 2016 and others) as a robust learning framework compared to Empirical Risk Minimization (ERM).  My analysis of this work in short is that the problem that this paper addresses is interesting and not yet solved. This paper proposes a simple and efficient method for DRO. However, convergence guarantees are weak which is reflected in their weak experimental results. And for a 10-page paper, the writing has to improve quite a lot.
Summary of contributions:
The phi-divergence variation of DRO introduces a min-max objective that minimizes a maximally reweighted empirical risk. The main contribution of this work is as follows:
- Show that for two common phi-divergences (KL-divergence and Pearson-chi2 divergence), the inner maximization has a simple solution for the weights that depends on the value of the loss on the training set.
- Use the above algorithm and modify SGD by changing the sampling of data according to the weights.
- Convergence proofs for the above algorithm for wide networks (not necessarily infinite-width).
Cons:
- Fig 1, ERM should not be so bad as if predicting randomly. This suggests a problem in the experimental setting. At least more ablation study is needed, e.g. try varying the percentage of data kept and show the final test accuracy as a function of this percentage for ERM. In the worst case, the accuracy for ERM should be ~90% on 9 classes and zero on 10th which is ~80% for uniform test accuracy. Another point, what is the train accuracy on cats? ERM should be overfitting to those few cats in the training set and achieve some non-zero accuracy at test. But it seems the test accuracy is almost exactly 0 at test time.
- In Theorem 5.1 that provides convergence proofs, the learning rate eta_exact has a dependence on 1/beta, which means much smaller learning rates should be used for the proposed method. However, in the experiments it seems that the same learning rate is used for all methods. This seems to trouble the convergence very clearly as the curves start to fluctuate considerably by increasing for larger betas. There is no bounds on beta which in practice can force us to use orders of magnitude smaller learning rates.
- Section 4.3 aims at linking hard negative mining to the proposed method. What they actually do is propose a new definition for hard-negative mining which is satisfied by the proposed method. There is little basis for suggesting this definition. There are myriads of work on hard-negative mining and suggesting a new definition needs a more thorough study of related works.
- Section A.1.1 argues we would stop ERM when it plateaus, but it doesn't look like it has plateaued in fig 2 left.
- Section A.3, I'm not sure lr=1 would be stable wide resnet. Maybe there is a problem with the experimental setting.
- There is such much non-crucial details in the main body that increased the main text to 10 pages. At least 2 pages could be saved by moving details of theorems and convergence results to the appendix.
After rebuttal:
I'm keeping more score. The manuscript has been improved quite a lot.  My main concern remains the experiments. Both texts and mathematical statements still need more edits.
The authors have completely removed experiments on cifar10. Paper now has only one set of experiments on MNIST with little ablation study. I still think my suggested ablation study is interesting. As a minimal sanity check, I'd like to see the performance of DRO on the original MNIST dataset. Basically, how much do we lose by using a robust method on a balanced dataset? Even with full ablation studies, I don't think MNIST is not enough for testing methods based on hard-negative mining.


We present an efficient quantum algorithm aiming to find the negative curvature direction for escaping the saddle point, which is a critical subroutine for many second-order non-convex optimization algorithms. We prove that our algorithm could produce the target state corresponding to the negative curvature direction with query complexityÕ(polylog(d)! −1 ), where d is the dimension of the optimization function. The quantum negative curvature finding algorithm is exponentially faster than any known classical method which takes time at least O(d! −1/2 ). Moreover, we propose an efficient algorithm to achieve the classical read-out of the target state. Our classical read-out algorithm runs exponentially faster on the degree of d than existing counterparts.

I am not familiar with the quantum algorithm literature, hence I cannot judge the novelty of the proposed algorithm. Basically, this is a quantum algorithm for computing the smallest negative eigenvalue (and the associated eigenvector) of square matrices. It is worth noting that the algorithm is built on top of the existing SVE model. The authors don't compare this method with any existing work in quantum singular value transformation and decomposition. A throughout discussion and comparison should be made in Section 1.1
Despite the title, I can't find a strong connection between the proposed algorithm and machine learning. Given a loss function f and the parameters x, the paper doesn't specify how to construct the quantum state in the step 2 of Algorithm 2. Is there a cheap way to construct the quantum state from (f, x)? If not, and if one has to compute the Hessian matrix H from (f, x), then the complexity of this pre-processing step will be on the order of O(d^2), which already makes the quantum efficiency meaningless.
Escaping saddle point is usually an iterative process, because the curvature varies across from place to place. The paper focuses on an individual step of a single iteration (i.e. finding the negative curvature), but it doesn't mention how would the iterative process look like.
Finally, although the quantum complexity is poly-log in dimension d, it has a terrible dependence on the rank r and the Frobenius norm of the Hessian matrix. The potential issues of such dependencies should be discussed.


Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks -(i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform-despite their strong performance on the related but more narrowly defined task of entailment NLI-pointing to interesting avenues for future research.

This paper introduces two new natural language tasks in the area of commonsense reasoning: natural language abductive  inference and natural language abductive  generation. The paper also introduces a new dataset, ART, to support training and evaluating models for the introduced tasks. The paper describes the new language abductive tasks, contrasting it to the related, and recently established, natural language inference (entailment) task.  They go on to describe the construction of baseline models for these tasks. These models were primarily constructed to diagnose potential unwanted biases in the dataset (e.g., are the tasks partially solved by looking at parts of the input, do existing NLI models far well on the dataset, etc.), demonstrating a significant gap with respect to human performance.
The paper, and the dataset specifically, represent an important contribution to the area of natural language commonsense reasoning. It convincingly demonstrates that the proposed tasks, while highly related to natural language entailment, are not trivially addressed by existing state-of-the-art models. I expect that teaching models to perform well on this task can lead to improvements in other tasks, although empirical evidence of this hypothesis is currently absent from the paper.
Below are a set of more specific observations about the paper. Some of these comments aim to improve the presentation or content of the paper.
1. In Section “5.1 - Pre-trained Language Models” and attendant Table 1describe results of different baselines on the ART inference task. The results in the table confused me for quite some time, I’d appreciate some clarifications. With respect to the differences with columns 1 (GPT AF) and 2 (ART, also BERT AF) I would like the comparison to be made more clear. As far as I understand it, there are 2 parts of the dataset that can be varied: (1) the train+dev sets and (2) the test set. Furthermore, it seems that it makes sense to vary each of these at a time, if we are to compare results with variants. For example: we can fix the test set, and vary how we generate training and dev examples. If a model does better with the same test set, we can assume the train+dev examples were better for the model (for whatever reasons, closer distribution to test, harder or more
informative training examples, etc). We can also keep the train+set constant, and vary the test set. This allows us to evaluate which test set is harder with respect to the training examples. The caption of Table 1 implies that both columns are evaluations based on the “ART” test set. If that is correct, then the train+dev set generated from the GPT adversarial examples is of better quality, generating a BERT-ft (fully connected) model that is 3% better. But the overall analysis seems to indicate that this is not what was done in the experiment. Rather, it seems that *both* the train+dev _and_ the test sets were modified concurrently. If that is the case, I would emphasize that the text needs to make this distinction clear. Furthermore, I would say that varying both train and test sets concurrently is sub-optimal, and makes it a bit harder to draw the conclusion that BERT adversarial filtering leads to a stronger overall dataset.
2. Along the lines of the argument in (1), above, I would urge the authors to publish the *entire* set of generated hypotheses (plausible and implausible) instead of only releasing the adversarially filtered pairs. Our group’s experience with training inference models is that it is often beneficial to train using “easy” examples, not only hard examples. I suspect the adversarially filtered set will focus on hard examples only. While this is fine to do in the test set, I think if the full set of annotated/generated hypotheses are released, model designers can experiment with combining pairs of hypothesis in different ways.
3. Furthering the argument of (2): in certain few-shot classification tasks, one is typically asked to identify similarity between one test example and different class representatives. Experience shows that it is often beneficial to train the model on a larger number of distractor classes than what the model is eventually evaluated on (e.g., https://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning). In the alpha-NLI setting, have you experimented with training using multiple distractors, instead of only 1, during training (even if you end up evaluating over 2 hypotheses)?
4. One potential argument for introducing a new natural language task is of transfer learning: learning to perform a complex natural language task should lead to better natural language models more generally, or, for some other related tasks. This paper does not really touch on this aspect of the work. But, potentially, one investigation that could be conducted is through a reversal of the paper’s existing NLI entailment experiment. The paper shows that NLI entailment models do not perform well on alpha-NLI. But it would be interesting to see if a model trained on alpha-NLI, and fine-tuned or multi-tasked on NLI entailment, does better on NLI entailment (i.e., is there transfer from alpha-NLI to entailment NLI?).
5. Another option is to evaluate whether alpha-NLI helps with other commonsense tasks. One other example is Winograd Schema Challenge, which current systems also perform well below human performance. It also seems that the Winograd Schema Challenge questions are not too far from abductive inference.
6. In the abstract of the paper, before the paper defines the abductive inference/generation task specifically, the claim that abductive reasoning has had much attention in research seemed awkward. Informally, most commonsense reasoning (including NLI entailment) could be cast as abductive reasoning.
7. In at least one occasion, I found an acronym which was hard to find the definition for (“AF” used in Section “5.1 - Pre-trained Language Models”; I assumed it was “adversarial filtering”.)
8. In Section “5.1 - Pre-trained Language Models” it seems that the text quotes an accuracy for BERT-ft (fully connected) of 68.9%, but Table 1 indicates 69.6%.
9. In Section “5.1 - Learning Curve and Dataset Size”, there is a claim that the performance of the model plateaus at ~10,000 instances. This does not seem supported by Figure 5. There appears to be over 5-7% accuracy (absolute) improvements from 10k to 100k examples. Maybe the graph needs to be enhanced for legibility?
10. It is great that the paper includes human numbers for both tasks, including all the metrics for generation.
11. Period missing in footnote 8.
12. The analysis section is interesting, it is useful to have in the paper. However, Table 3 is a bit disappointing in that ~26% of the sampled examples fit into one of the categories. It would be great if the authors could comment on the remaining ~74% of the sampled dataset.


This paper proposes a new approach for step size adaptation in gradient methods. The proposed method called step size optimization (SSO) formulates the step size adaptation as an optimization problem which minimizes the loss function with respect to the step size for the given model parameters and gradients. Then, the step size is optimized based on alternating direction method of multipliers (ADMM). SSO does not require the second-order information or any probabilistic models for adapting the step size, so it is efficient and easy to implement. Furthermore, we also introduce stochastic SSO for stochastic learning environments. In the experiments, we integrated SSO to vanilla SGD and Adam, and they outperformed state-of-the-art adaptive gradient methods including RMSProp, Adam, L 4 -Adam, and AdaBound on extensive benchmark datasets.

This paper proposes a new step size adaptation in first-order gradient methods. The proposed method establishes a new optimization problem with the first-order expansion of loss function and the regularization, where the step size is treated as a variable.  ADMM is adopted to solve the optimization problem.
This paper should be rejected because (1) the proposed method does not show the convergence rate improvement of the gradient method with other step sizes adaptation methods. (2) the linearization of the objective function leads the step size to be small ($0<\eta<\epsilon$), which could slow down the convergence in some cases. (3) the experiments generally do not support a significant contribution. In table 1, the results of the competitor are not with the optimal step sizes. The limit grid search range could not verify the empirical superiority of the proposed method.
Minor comments:
The y-axis label of (a) panel in each figure is wrong. I guess it should be "Training loss ".