GRADIENT DESCENT PROVABLY OPTIMIZES OVER-PARAMETERIZED NEURAL NETWORKS
One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For an m hidden node shallow neural network with ReLU activation and n training data, we show as long as m is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function.
Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

Additional Review
This paper did NOT handle the non-differentiability and non-linearity very well. We can see this from the following three perspectives:
1. Proof idea: the proof of this paper is noisy version of the convergence analysis of  a simple convex problem --it treats the contribution of the non-linearity and non-differentiability as bounded noise.
2. The network size is of order n^6.
3. Network size requirement is dependent on \lambda_0.
1.Proof idea: The proof is essentially a noisy version of the convergence analysis of a linear regression problem provided in Appendix (at the end of this updated review). The only difference between linear regression and the problem in this paper is the changing patterns due to the non-linearity of ReLU. However, this paper views the changing patterns as noises compared to those unchanging patterns (e.g., S_i v.s. S_i^\perpendicular). The key trick is that if the actual trajectory radius (i.e.,the largest deviation from the initial point) R’ is much smaller than the desired trajectory radius R (given by a formula), then along the trajectory, the contribution of non-linearity is just O(n^2 R), which is small compared to the contribution of linearity, i.e., -\lambda_0 (shown in proof on page 9).
Following the above analysis, if the experiment shows that R’ is really small compared to R, then the approach of treating non-linearity as noise is fine. However, it is not the case for the problem studied in the experiments (Sec 5, Fig 1). In figure 1, we can easily see that the maximum distance R’ is O(1), which is far larger than R = c*\lambda_0/n^2 =10^-6 when n=1k. Therefore, the proof idea used in this paper is fundamentally not able to explain the phenomenon shown in the experiment. In fact, to address this issue, authors need to consider significant contribution of non-linearity, instead of just viewing them as noises.
2. The network size is too large. This paper requires O(n^6) neurons, that is 10^18 neurons for n=1000 samples used in the experiment. The theoretical trick to make R’< R is to note that R’ can be bounded by O(1/sqrt{m}) while R is independent of m, thus picking a sufficiently large m can make R’ very small. In a word, the reason that this paper requires so many neurons because of the inability of properly addressing non-linearity.
3. I found the dependence of the network size on the least eigenvalue funny, although the authors claim this tool is elegant. After authors add Thm 3.1 in the revision, I realize that the dependence on \lambda_0 might come from the fact that authors do NOT handle the issue of non-differentiability.
Let us see a simple example. Assume I have a dataset with \lambda_0 = 1. Now I am adding one more data point (x=0_d, y=1) to the dataset. After adding this sample, \lambda_0 clearly becomes 0. It seems I am just adding a constant 1 to the loss function and the gradient descent can also converge to the global min with a linear convergence rate since the constant does NOT contribution to the gradient. However, it seems the proof does NOT work. This is due to the fact that the “gradient” of the non-differentiable points are NOT well defined. Here is a simple example: h(w)=(y-ReLU(w*x))^2, where x= 0, y =1. By the definition provided in this paper (Eq.4), we can easily see that dh/dw = 1 for any w, even if h(w) = 1 for any w. This means that the constant can provide “fake” gradient information and make  the maximum distance become infinity, (R’=\inf). Therefore, the whole proof collapses. In fact, changing the gradient definition from I{z>=0} to I{z>0} does not address the issue and we can see this from this example w=g(w)=Relu(w)-Relu(-w) has a zero gradient at w=0.
In summary, the problem considered in this paper where the size m=O(n^6), maximum distance R’= O(1/n^2) is too easy compared to most problems in practice where m=\Theta(n), R’=O(1). To address the latter problem, we need a better definition of subgradient and need to analyze the significant contribution of non-linearity and non-differentiability, instead of just viewing them as noises.
=================================Appendix===============================
The proof basically follows from the convergence analysis of the following linear regression problem (note that u_j is fixed):
           \min_{w_1,...,w_m}\sum_{i=1}^{n}(f(x_i;w_1,...,w_m)-y_i)^2 = L(w_1,...,w_m)
where   f(x;w_1,...,w_m)=1/\sqrt{m}\sum_{j=1}^{m} a_j*(w_j^T x)*1{u_j^T x>=0}
Gradient Descent Algorithm:
-Initialization:
-For each j=1,...,m: a_j ~ U({-1,1}), u_j~N(0, I)
-Fix a_1,...,a_m, u_1,...,u_m
-Update:
-For t = 1,...,T
    w_j(t+1)  = w_j(t) - \eta* \nabla_{w_j}L(w_1,...,w_m) for j=1,..., m.
In this problem, since a_j and u_j are fixed, then model f is just a linear model w.r.t. w_j’s and the above problem is just a simple linear regression problem. Therefore, it is not difficult to prove the linear convergence rate for the gradient descent for the above problem under some mild assumptions.  Note that in this paper, u_j(t)= w_j(t) and are not fixed in iterations, i.e., patterns can change.
=========================
First, I apologize to the authors and ACs for the late review, since this paper desearves much more time to judge the quality.
Summary: This paper proves that the gradient descent/flow converges to the global optimum with zero training error under the settings (1) the neural network is a heavily over-parameterized ReLU network (i.e., requiting Omega(n^6) neurons); (2) the algorithm update rule “ignores” the non-differentiable point; (3) the parameters in the output layer (i.e., a_i’s) are fixed; (4) the data set has some non-degenerate properties and comes from a unit ball. The proof relies on the fact that the Gram matrix is always positive definite on the converging trajectory.
Pros: The proof is simple and seems to be correct. The paper is paper is written clearly  and easy to follow.
Cons:
The problem setting considered in this paper does not seem to be difficult enough. The difficulty of analyzing the landscape property of a ReLU network and proving the global convergence of the gradient descent mainly lies in the following three perspective and this paper does not try to tackle any one of them.
First, it is very hard to characterize the landscape or the convergence trajectory at/ near the non-differentiable point and this paper fails to touch it. The parameter space is separated into several regions by the hyperplanes and the loss function is differentiable in the interior of each region and non-differentiable on the boundary. I believe the very first question authors need to answer is wether there are critical points on the boundary and why the sub-gradient descent escapes from  any of these points. However, in this paper, authors avoid this problem by defining an update rule used  in practice and this rule does not use the sub-gradient at the non-differentiable point. Thus, it is totally unclear to me wether this global convergence result comes from the fact that this update rule can generally avoid the non-differentiable  points on the boundary or the fact that the landscape is so nice such that there are no critical points on the boundary or the fact that all points on the convergence trajectory is differentiable only in this unique problem.
Second, the problem is much easier if the loss is not jointly optimized over the parameters in the first and second layer. Having parameters in one layer fixed does not seem to be a big problem at first glance, but then I realize it indeed makes the problem much easier, which can be seen in the following example. If we randomly sample the weight vector w_i from N(0, I) and only optimize  over the parameters in the second layer, then it is straightforward to show the following result.
Result: If \lambda_\min(H^\inf)>0 and m=\Omega(n\log n), then with high probability, the loss function L is strongly convex with respect to a=(a_1,…, a_m) and the loss function is zero at the global minimum.
The above result shows that if we fix the parameters in the first layer and only optimize the parameters in the second layer, it is easy to prove the global convergence with a linear convergence rate. In fact, this result does not require the samples coming from a unit ball and the network size is only slightly over-parameterized. Therefore, if we are allowed to fix the parameters in some layer, how are the result presented in this paper fundamentally different from the above result.
Authors may say that the loss is not convex with respect to the weights in the first layer even if the second layer is fixed. However, when the second layer is fixed, the loss function is  smooth and convex in each parameter region and some recent works have shown that in this case, the loss function is a weakly global function. This means that the loss function is similar to a convex function except those plateaus and this further indicates that if the initial point is chosen in a strictly convex basin, the gradient descent is able to converge to a global min.  However, the problem becomes far more difficult if the loss is jointly optimized over  all parameters in the first and second layer. This can be easily seen since in each parameter region, the loss is no longer a convex function and this may lead to some high order saddle points such that the gradient descent cannot provably escape. Furthermore, the critical points on the boundary can be much more difficult to characterize for this joint optimization problem.
Third, the dataset considered in this paper does not seem to be a fundamental pattern and it seems more like a technical condition required by the proof. It is easy to see that a linearly separable dataset does not necessarily satisfy the conditions that 1) the gram matrix is positive definite and that 2) samples come from the surface of a unit ball. Therefore, I do not understand the reason why we need to analyze this pattern. Clearly, in practice, the data samples is unlikely sampled from a ball surface and it is totally unclear to me why the gram matrix is necessarily positive definite. I understand that some technical assumptions are needed in a theoretical work, but I would like to see more discussions on the dataset, e.g., some necessary conditions on the dataset such that the global convergence is possible.
Last, I understand that the over-parameterization assumption is needed. In fact, I expect the network size to be of the order Omega(n*ploylog(n)). I am wondering wether Omega(n^6) is a necessary condition or wether there exists a case such that Theta(n^6) is required.
Above all, I believe this paper is a half-baked paper with some interesting explorations. In summary, it cannot deal with non-differential points, which is considered a major difficulty for analyzing ReLU. In addition, it makes an un-justified assumption on some matrix, it requires too many neurons, and fixed 2nd layer. With so many strong assumptions, and compared to related works like [1], Mei et al., Bach and ..., its contribution is rather limited.
[1] https://arxiv.org/abs/1702.05777


Many machine learning tasks such as multiple instance learning, 3D shape recognition and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the permutation of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating increased performance compared to recent methods for set-structured data.

This paper presented an attention-based neural network, namely set transformer, a new neural model
based on original transformer designed for set inputs. The basic idea is to introduce the attention
mechanism in both learning the feature embeddings of the set inputs during “encoding” and aggregating
these embeddings during “decoding”. The paper is written clearly and well motivated. The extensive
set of experiments were conducted to demonstrate the effectiveness of the proposed method. In general,
I like reading this paper but there are some limitations or unclear parts I need authors to clarify
and explain.
i) The proposed architecture is mainly adopted from the original transformer but it is highly related
to the baselines used in the experiments. For instance, it seems like that the current set
transformer is a simple combination of Yang et al.(2018) and Mishra et al.(2018) (using Stack of
SABs) in encoder side and of Ilse et al.(2018) (using PMA and stack of SABs) in the decoder side.
This simple combination makes the novelty of this paper unclear. I would like authors to clarify
more on the originality w.r.t. these previous works.
ii) Although authors proposed a variant of SABs - ISABs using landmark points to accelerate the
computation, there are no any runtime comparisons between SABs and ISABs by fixing other components.
It would be interesting to see that ISABs can approach the performance SABs and how it approaches it.
For instance, shall we expect that ISABs approach the performance of SABs when increasing the number
of landmark points (inducing points)? Since in practice most of datasets are relatedly large, I think
understanding the behavior of ISABs is a more interesting problem.
iii) After seeing the results in table 6, I have quite concerned about the practical performance of
set transformer on relatively large datasets (like 1000 points each class in the settings.) It looks
to me that not only set transformer may have computational issues to scale up, but more importantly
that when encoder learned really expressive embeddings with a relatively large number of the set
inputs it might be little need to leverage attention in pooling anymore. I would like authors to
conduct some other experiments on relatively large datasets to verify this hypothesis, which is
important for the practical applications of the proposed model.


Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution. This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift. Ideally, a model would assign lower confidence to points unlike those from the training distribution. We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points. Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes. This has a major advantage in that it avoids the underfitting which can result from interpolating in the input space. We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice. Additionally, this concentration can be seen as making the features in earlier layers more discriminative. We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.

The paper proposes a novel method called Manifold Mixup, which linearly interpolating (with a careful selected mixing ratio) two feature maps in latent space as well as their labels during training, aiming at regularizing deep neural networks for better generalization and robust to adversarial attacks. The authors experimentally show that networks with Manifold Mixup as regularizer can improve accuracy for both supervised and semi-supervised learning, are robust to adversarial attacks, and obtain promising results on Negative Log-Likelihood on held out samples.
The paper is well written and easy to follow. Various experiments are conducted to support the contributions of the paper.  Nevertheless, the technical novelty seems a bit weak to me. The method basically moves the interpolating process from input space as in MixUp to randomly selected hidden states. More importantly, some of the paper’s claims are not very convincing to me in its current form.
Major remarks:
1.	The authors suggest that Mixup can suffer from interpolations intersecting with a real sample, but how Manifold Mixup can avoid this issue is not very clear to me.
The authors theoretically prove that with the proposed training cost in Manifold Mixup, the representation for each class will lie on a subspace of dimension dim (h) –d +1 (h and d are the hidden dimension and number of classes, respectively). I did not get the idea of how such dimension reduction relates to the ‘’flattening’’ of the manifold and in particular how such representations (representations for each class “concentrating into local regions”) can avoid the class collision issues as that in Mixup.
Experimentally, from Figures 3 and 4, it seems the class collision issue could be worse than that of Mixup. For example, for mixing ratio of 0.6 (meaning the created image has almost half labels from the two original images), MixUp clearly shows, for instance in the second row, that there are two overlapped images (Horse and Plane), but Manifold Mixup seems to have only the Plane in the mixed image with a soft label.
2.	The observations of mixing in the hidden space is better than mixing in the input space seem to contradictive to the observations by Mixup, it would be very useful if the paper can make that much clear to the readers. I would suggest that the authors fully compare with MixUp in the supervised learning tasks, namely using all the datasets (including ImageNet) and networks architectures used in MixUp for supervised learning. In this way, the paper would be much more convincing because the proposed method is so close to MixUp and the observation here is contradictive.
3.	I wonder how sensitive is the parameter Alpha in Manifold Mixup. For example, how the mixing rate Alpha impacts the results for NLL and Semi-supervised learning in section 5.2?
4.	It would be useful to also present the results for SVHN for supervised learning since the Cifar10 and Cifar100 datasets are similar, and the authors have already used SVHN for other task in the paper.
Minor remarks:
1.	In Table2, the result from AdaMix seems missed.
2.	Why not using Cifar100, but with a new dataset SVHN for the semi-supervised learning in section 5.2?
3.	In related work, regarding regularizing deep networks by perturbing the hidden states, the proposed method may relate to AgrLearn (Guo et al., Aggregated Learning: A Vector Quantization Approach to Learning with Neural Networks) as well.


For problem solving, making reactive decisions based on problem description is fast but inaccurate, while search-based planning using heuristics gives better solutions but could be exponentially slow. In this paper, we propose a new approach that improves an existing solution by iteratively picking and rewriting its local components until convergence. The rewriting policy employs a neural network trained with reinforcement learning. We evaluate our approach in two domains: job scheduling and expression simplification. Compared to common effective heuristics, baseline deep models and search algorithms, our approach efficiently gives solutions with higher quality.

Summary:
Search-based policies are stronger than a reactive policies, but the resulting time consumption can be exponential. Existing solutions include designing a plan from scratch given a complete problem specification or performing iterative rewriting of the plan, though the latter approach has only been explored in problems where the action and state spaces are continuous.
In this work, the authors propose a novel study into the application of iterative rewriting planning schemes in discrete spaces and evaluate their approach on two tasks: job scheduling and expression simplification. They formulate the rewriting task as a reinforcement learning problem where the action space is the application of a set of possible rewriting rules to modify the discrete state.
The approach is broken down into two steps. In the first step, a particular partition of the discrete state space is selected as needing to be changed by a score predictor. Following this step, a rule selector chooses which action to perform to modify this state space accordingly.
In the job scheduling task, the partition of the state space corresponds to a single job who’s scheduled time must be changed. the application of a rule to rewrite the state involves switching the order of any two jobs to be run. In the expression simplification task, a state to be rewritten corresponds to a subtree in the expression parse tree that can be converted to another expression.
To train, the authors define a mixed loss with two component:
1. A mean squared error term for training the score predictor that minimizes the difference between the benefit of the executed action and the predicted score given to that node
2. An advantage actor critic method for training the rule selector that uses the difference between the benefit of the executed action and the predicted score given to that node as a reward to evaluate the action sampled from the rule set
Pros:
-The approach seems to be relatively novel and the authors address an important problem.
-The authors don’t make their approach more complicated than it needs to be
Cons:
Notation: The notation could be a lot clearer. The variable names used in the tasks should be directly mapped to those defined in the theory in Section 2. It wasn’t clear that the state s_t in the job scheduling problem was defined as the set of all nodes g_j and their edges and that the {\hat g_t} corresponds to a single node. Also, there are some key details that have been relegated to the appendix that should be in the main body of the paper (e.g., how inference was performed)
Evaluation: The authors perform this evaluation on two automatically generated synthetic datasets. It’s not clear that the method would generalize to real data. Why not try the approach on a task such as grammar error correction? Additionally, I would have liked to see more analysis of the method. Apart from showing the comparison of the method with several baselines, the authors don’t provide many insights into how their method works. How data hungry is the method? Seeing as the data is synthetically generated, how effective would the method be with 10X of the training data, or 10% of it? Were any other loss functions attempted for training the model, or did the authors only try the Advantage Actor Critic? What about a self-critical approach? I'd like to see more analysis of how varying different components of the method such as the rule selector and score predictor affect performance.


For neural sequence model training, maximum likelihood (ML) has been commonly adopted to optimize model parameters with respect to the corresponding objective. However, in the case of sequence prediction tasks like neural machine translation (NMT), training with the ML-based cross entropy loss would often lead to models that overgeneralize and plunge into local optima. In this paper, we propose an extended loss function called dual skew divergence (DSD), which aims to give a better tradeoff between generalization ability and error avoidance during NMT training. Our empirical study indicates that switching to DSD loss after the convergence of ML training helps the model skip the local optimum and stimulates a stable performance improvement. The evaluations on WMT 2014 English-German and English-French translation tasks demonstrate that the proposed loss indeed helps bring about better translation performance than several baselines.

This paper describes an alternative training objective to cross-entropy loss for sequence-to-sequence models. The key observation is that cross-entropy is minimizing KL(P|Q) for a data distribution P and a model distribution Q; they add another loss that minimizes the inverse KL(Q|P) to create their dual-skew divergence. The idea is tested in the context of neural MT, using a model similar to that proposed by Bahdanau et al. (2015) with results on English-to-French and English-to-German WNT 2014. In the context of beam search, improvements are small (<=0.5 BLEU) but statistically significant.
This is an interesting idea, and one I certainly wouldn’t have thought of on my own, but I think it is currently lacking sufficient experimental support to warrant publication. The paper feels strangely dated, with most experiments on two-layer models, and only two citations from 2017. The experiments compare against an in-house maximum likelihood baseline (varying greedy-vs-beam search and model depth), and against a number of alternative training methods (minimum risk, scheduled sampling, RL) with numbers lifted from various papers. These latter results are not useful, as the authors (helpfully) point out that the baseline results in this paper are universally higher than the baselines from these other papers. Furthermore, it feels like methods designed to address exposure bias and/or BLEU-perplexity mismatch are not the right comparison points for this work, as it does not attempt to address either of these issues. I would instead be much more interested to see a comparison to label smoothing (Szegedy et al., 2015), which perhaps addresses some of the same issues, and which produces roughly the same magnitude of improvements. Also, the literature review should likely be updated to include Edunov et al., 2017. In general, the improvements are small (though technically statistically significant), the baseline models are somewhat shallow and the deltas seem to be decreasing as model depth grows, so it is hard to get too excited.
Smaller concerns:
For Table 1, it would be helpful to explain why Baseline is not equal to \Beta=1. With some effort, I figured out that this was due to the alpha term modifying the cross-entropy objective when \Beta=1.
It would also be useful to tell us what “switching point” was used for Table 1 and Figure 2.
Christian Szegedy, Vincent Vanhoucke, SergeyIoffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. CoRR abs/1512.00567. http://arxiv.org/abs/1512.00567.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Classical structured prediction losses for sequence to sequence learning. In Proceedings of NAACL-HLT 2018.


General comment
==============
The authors used policy gradient optimization for generating RNA sequences that fold into a target secondary structure, reporting clear accuracy and runtime improvements over the previous state-of-the-art. The authors used BOHR for optimizing hyper-parameters and present a new dataset for evaluating RNA design methods. The paper is well motivated and mostly clearly written. However, the methodological contributions are limited and I have some important concerns about their evaluation. Overall, I feel it’s a good paper for an ICLR workshop or biological journal if the authors address the outstanding comments.
Major comments
=============
1. The methodological contributions are limited. The authors used existing approaches (policy gradient optimization and BOHR for hyperparameter optimization) but do not report new methods, e.g. for sequence modeling. Performing hyper-parameter optimization is in my eyes not novel, but common practice in the field. It would me more informative if the authors compared reinforcement learning to other approaches for (conditional) sequence generations, e.g. RNNs, autoregressive models, VAEs, or GANs, which have been previously reported for biological sequence generation (e.g. http://arxiv.org/abs/1804.01694).
2. Did the authors split all three datasets (Eterna, Rfam-Taneda, Rfam-learn-test) into train, eval, and test set, trained their method on the training set, optimized hyper-parameters on the eval set, and measured generalization and runtime on the test set? This is not described clearly enough in section 5. I suggest to summarize the number of sequences for each dataset and split in a table.
3. Did the authors also optimize the most important hyperparameters of RL-LS and other methods? Otherwise it is unclear if the performance gain is due to hyperparameter optimization or the method itself.
4. The time measurement (x-axis figure 3) is unclear. Is it the time that methods were given to solve a particular target structure and does figure 3 show the average number of solved structures in the test for a the time shown on the x-axis?
5. Were all methods compared on the same hardware (section 5; 20 cores; Broadwell E5-2630v4 2.2 GHz CPUs) and can they be parallelized over multiple CPU or GPU cores? This is essential for a fair runtime comparison.
6. The term ‘run’ (“unreliable outcomes in single runs”, section 4) is unclear. Is it a single sample from the model (one rollout), a particular hyperparameter configuration, or training the model once for a single target structure? This must be clarified for understanding the evaluation.
7. How does the accuracy and runtime or LEARNA scale depending on the sequence (structure) length?
8. How sensitive is the model performance depending on the context size k for representing the current state? Did the authors try to encode the entire target structure with, e.g. recurrent models, instead of using a window centered on the current position?
9. The authors should more clearly describe the local optimization step (section 3.1; reward). Were all nucleotides that differ mutated independently, or enumerated exhaustively? The latter would have a high runtime of O(3^d), where d is the number of nucleotides that differ. When do the authors start with the local optimization?
Minor comments
=============
10. The authors should replace ‘450x’ faster in the abstract by ‘clearly’ faster since the evaluation does not show that LEARNA is 450x faster than all other methods.
11. Does “At its most basic form” (introduction) mean that alternative RNA nucleotides exist? If so, this should be cited.
12. The authors should more clearly motive in the introduction why they created a new dataset.
13. The authors should mention in section 2.1 that the dot-bracket notation is not the only notation for representing RNA structures (https://www.tbi.univie.ac.at/RNA/ViennaRNA/doc/html/rna_structure_notations.html).
14. The authors should define the hamming distance (section 2.1). Do other distance metrics exist?
15. For the Traveling Salesman Problem (section 2.2) should the reward be the *negative* tour length?
16. The authors should more clearly describe the embedding layer (section 4). Are nucleotides one-hot encoded or represented as integers (0, 1  for ‘(‘ and ‘.’)?


The Adversarially Learned Mixture Model (AMM) is a generative model for unsupervised or semi-supervised data clustering. The AMM is the first adversarially optimized method to model the conditional dependence between inferred continuous and categorical latent variables. Experiments on the MNIST and SVHN datasets show that the AMM allows for semantic separation of complex data when little or no labeled data is available. The AMM achieves unsupervised clustering error rates of 3.32% and 20.4% on the MNIST and SVHN datasets, respectively. A semi-supervised extension of the AMM achieves a classification error rate of 5.60% on the SVHN dataset.

The paper presents a method for un-/semi- supervised clustering which combines adversarial learning and Mixture of Gaussian.
Cons:
It is interesting to incorporate the generative model into GAN.
Probs:
1.	The author claims the method is the first one to generative model inferring both continuous and categorical latent variables. I think that such a conclusion is overclaimed, there are a lot of related works, e.g., Variational deep embedding: An unsupervised generative approach to Clustering, IJCAI17; Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks, ICML2017. Multi-Modal Generative Adversarial Networks for Diverse Datasets, ICLR19 submission. In fact, these methods are also very competitive approaches to clustering.
2.	Is adversarial learning do helpful for clustering with the generative model? Some theoretical should be given, at least, some specified experiments should be designed.
3.	The novelty should be further summarized by highlighting the difference with most related works including but not limited the aforementioned ones. The current manuscript makes the work seem like a straightforward combination of many existing approaches.
4.	In fact, the paper is hard to follow. I would recommend improving the logic/clarity.


KERNEL RNN LEARNING (KERNL)
We describe Kernel RNN Learning (KeRNL), a reduced-rank, temporal eligibility trace-based approximation to backpropagation through time (BPTT) for training recurrent neural networks (RNNs) that gives competitive performance to BPTT on long time-dependence tasks. The approximation replaces a rank-4 gradient learning tensor, which describes how past hidden unit activations affect the current state, by a simple reduced-rank product of a sensitivity weight and a temporal eligibility trace. In this structured approximation motivated by node perturbation, the sensitivity weights and eligibility kernel time scales are themselves learned by applying perturbations. The rule represents another step toward biologically plausible or neurally inspired ML, with lower complexity in terms of relaxed architectural requirements (no symmetric return weights), a smaller memory demand (no unfolding and storage of states over time), and a shorter feedback time. Finally, we show that KeRNL can learn long time-scales more efficiently than BPTT in an online setting.

This paper proposes a simple method for performing temporal credit assignment in RNN training.  While it seems somewhat naive and unlikely to work (in my opinion), the experimental results surprisingly show reasonable performance on several reasonably challenging artificial tasks.
The core of the approach is based on equation 7, which approximates the Jacobian between different hidden states at different time-steps as a single adaptively-learned matrix times a decay factor that depends on the time gap.  While this seems like a very severe approximation to make the authors speculate that some kind of feedback alignment-like mechanism might be at play.
The presentation needs work in several areas, and the experimental results require more explanation, but otherwise this seems like a solid paper.  I would probably increase my rating if the authors could address my issues satisfactorily.
See below for more detailed comments:
Abstract & Section 1:
Is "sensitivity tensor" or "credit assignment tensor" common term?  Because I've never heard them before.  Consider defining them before you discuss it, and using consistent jargon.  Later in Section 2 you seem to call this the "RTRL tensor" (whose meaning I can infer).
Section 2:
Gradient vanishing isn't so much a problem in itself, but a symptom that the sensitivity of the network's output to the action of some neuron in the past is very low. Ths gradient is just relaying this information, so I don't really see vanishing gradients as the problem to overcome, but rather low sensitivity on past activations.
Section 3:
Did you mean to write (W^out h^t + b^out) instead of (W^out h + b^out)^t ?
"[equation] represents the gradient of the cost with respect to the current hidden state".  The RHS of this equation makes no sense to me.  Not only does this not depend on the nonlinearity in any way, it doesn't include any consideration of future outputs on which the current h surely depends.
It would make the paper much more pleasant to read if you gave your derivation of the learning rule before you stated it in gory detail.  It feels almost completely arbitrary reading it first without any justification. This might be fine if it were compact and elegant, but it's not.
Consider using exp(x) instead of e^x since the symbol e already means something else in your notation.
Section 4:
Please define "temporal variation"
Section 5:
You should elaborate on the experimental setup you used.  Especially for the Addition and MNIST problems.  For example, what consistutes a "step" in figure 2?  Does KeRL take "one" step per time-step?  Or does "step" mean a complete gradient computation from running from t = 1 to t= T?  Is the BPTT truncated?  Are you counting one step of BPTT to be one complete forwards and backwards pass?
You should include some basic description of what an IRNN is.
When you say that for MNIST that KeRL "does not converge to as good of an optimum" this seems like unjustified inference.  You don't really know that it is converging to a minimum of the original objective at all.  It could be converging to the minimum of some other objective it is implicitly optimizing due to your approximations (if one even exists).  Or it could be simply cycling around and failing to converge.  The fact that the loss plateaus isn't direct evidence of convergence in any sense.  If you wanted to measure this more directly you could look at the (true) gradient magnitude.
"only requires a few tensor operations at each time step" -> this is also true of UORO


The paper describes a question about discretize continuous features or group discrete features in the preprocessing step, which they call feature quantification. It considers that a joint training of feature quantification and a discriminative model can lead to a better performance than treating feature quantification as a preprocessing step.
This paper has many typos, grammar mistakes and question marks, which make it hard to follow. The question proposed is simple and easy to understand. However, I don't convinced by the solution in this paper. Since it is a hard optimization question, the authors proposed a relaxation approach in section 3.1. I do not think that exp(a+bx) is able to approach step functions since exp(a+bx) is monotone. I think Figure 1 is misleading. For grouping discrete features, the author propose to use exp(\alpha_{x_j, j}^h) and hoping that some \alpha parameters can be optimized to be equal, which is too simple. The exponential transformation here does not have an effect. It is more interesting to consider how to add some constraints. For example, if the discrete feature is ordinal, how one can assure that the grouped discrete feature is still ordinal. The relaxation in this paper is too much without handling any interesting constraints and the proposed exp(a+bx) can not approach step functions. The authors do not provide a good way to select number of cut points, which I think is a hard but interesting question.
The work also lacks value in literature review, optimization and experiments.


LEARNING A META-SOLVER FOR SYNTAX-GUIDED PROGRAM SYNTHESIS
A general formulation of program synthesis called syntax-guided synthesis (SyGuS) seeks to synthesize a program that follows a given grammar and satisfies a given logical specification. Both the logical specification and the grammar have complex structures and can vary from task to task, posing significant challenges for learning across different tasks. Moreover, supervision is often unavailable for domain-specific synthesis tasks. To address these challenges, we propose a metalearning framework that learns a transferable policy using only weak supervision. Our framework consists of three components: 1) an encoder, which embeds both the logical specification and grammar at the same time using a graph neural network; 2) a grammar adaptive policy network which enables learning a transferable policy; and 3) a reinforcement learning algorithm that jointly trains the embedding and adaptive policy with sparse reward. We evaluate the framework on 214 cryptographic circuit synthesis tasks. It solves 141 of them in the out-of-box solver setting, significantly outperforming a similar search-based approach but without learning, which solves only 31. The result is comparable to two state-of-the-art classical synthesis engines, which solve 129 and 153 respectively. In the metasolver setting, the framework can efficiently adapt to unseen tasks and achieves speedup ranging from 2× up to 100×.

This paper presents a (meta-)solver for particular program synthesis problems, where the model has access to a (logic) specification of the program to be synthesized, and a grammar that can change from one task instance to another. The presented model is an RL-based model that jointly trains 1) the joint graph-based embedding of the specification and the grammar, and 2) a policy able to operate on different (from instance to instance) grammars. Interestingly, not only can the model operate as a stand-alone solver, but it can be run as a meta-solver - trained on a subset of tasks, and applied (with tuning) on a new task. Experiments show that the model outperforms two baselines (one being a (near-to-)SOTA model) in the stand-alone setting and that the model successfully transfers knowledge (considers fewer candidates) in the meta-solving mode.
First, I enjoyed reading the paper. I think the problem is interesting, particularly due to the model being able to train and operate on various grammars (from task to task), and not on a single, pre-specified grammar. The additional bonus is that the problem the paper solves does not require program as supervision, but an external verifier.
The evaluation shows that this approach not only makes sense but (significantly) outperforms, under same conditions, specialized program synthesis programs. However, there’s one issue here, and that’s what the comparison hasn’t been done to SOTA model but to a less performant model (see issues).
The particular approach of jointly training a specification+grammar graph embedding and learning a policy that acts on different grammars seems original and significant enough for publication.
The paper is well (with a few kinks) written, and mostly clear. There are still some issues in the paper.
Issues:
- The dataset used is 210 cryptographic circuit synthesis tasks from SyGuS 2017. Why only this particular subset of all the tasks, and not the other tasks/categories (there is 569 of them in total, no)?
- Alur et al mention 214 examples in the said tasks, yet the paper says 210. Why?
- The SyGuS results paper https://arxiv.org/abs/1711.11438 mentions EUSolver as the SOTA model, solving 152 tasks (out of 214). Why didn’t you compare your model to EUSolver?
- The same paper reports CVC4 solving 117 tasks (out of 214), as opposed to 129 (out of 210) reported in your paper. Could you comment on the (possible) differences in the experimentation protocol?
- you mention global graph embedding, but you never describe how you calculate it
- abstract mentions outperforming two SOTA engines, but later you say ESymbolic is a baseline (which it seems by description)
Questions:
- W for different edge types and different propagation steps t? Why is there a need for such a large number of parameters? What is the number of propagation steps?
- In the extreme case where all inputs can be enumerated - how often does this happen in the tasks you solve?
- figure 2 is not clear. There is too much information on one side (grammar) and too little on the other (what is the meaning of \tau^(t-1)?)? Is the tree on the right a generated subtree?
- details of the state s are unclear - it is tracked by an LSTM? Is there a concrete training signal for s, or is it a part of the architecture and everything is end-to-end trainable from the final reward? The same for s0=MLP(h(G)) - is that also trained in the same way?
- can you provide some intuition on why you chose that particular architecture (state-tracking LSTM,  s0 as such, instead of something simpler?)
- can you provide details on the state value estimator MLP architecture, as well as the s0 MLP, and the state-tracking LSTM?
- the probability of each action (..) is defined as ….H_\alpha^(i) - what does the i stand for? Was that supposed to be the t or \alpha_t was supposed to be \alpha_i?
Minor stuff:
- Figure 5a is referred to as Table 5a in the text
- out-of-out-solver
- global graph embedding, figure 1 - G(phi, G), figure 2 - h(G)
- a figure of the policy architecture would be beneficial
- Figure 1
  - d_1 ->X OR Y in the graph is d1T, why isn’t it d1_OR, and connected to the OR node?
  - why isn’t d1_OR connected to OR node?
  - AST edge - but grammar is a DAG - (well, multigraph)
  - what are the reversed links? e.g. if A->B, reversed link is B->A ?
  - what is the meaning of the concrete figures in ‘one step’?
- consider relating to ‘DREAMCODER: Bootstrapping Domain-Specific Languages for Neurally-Guided Bayesian Program Learning’ (https://uclmr.github.io/nampi/extended_abstracts/ellis.pdf), as it’s another model that steps away from the fixed-DSL story


In this paper we propose to view the acceptance rate of the Metropolis-Hastings algorithm as a universal objective for learning to sample from target distribution -given either as a set of samples or in the form of unnormalized density. This point of view unifies the goals of such approaches as Markov Chain Monte Carlo (MCMC), Generative Adversarial Networks (GANs), variational inference. To reveal the connection we derive the lower bound on the acceptance rate and treat it as the objective for learning explicit and implicit samplers. The form of the lower bound allows for doubly stochastic gradient optimization in case the target distribution factorizes (i.e. over data points). We empirically validate our approach on Bayesian inference for neural networks and generative models for images.

One of the main contributions of the paper is showing how maximizing the acceptance rate in Metropolis Hastings (MH) translates in minimizing the symmetric KL divergence of the target and the proposal distribution. The result aligns with the intuition that if this variance is 0, detailed balanced is satisfied and, hence, we can always accept the proposal. Also Equation 11 nicely fits the intuition that a good (independent) proposal should minimize the KL divergence between proposal and target (as in VI) under the constraint that the proposal has full support compared to the target distribution, which is enforced by the last term. Theorem 1 and its proof are great.
However, the proposed algorithms leveraging these findings are problematic. Algorithm 1 suggest independent Metropolis-Hastings in order to avoid the proposal to collapse to a point distribution, that is, a Dirac delta function centered at the current position. However, in the experiments, the authors study a "modified" version using a random walk proposal parameterized with the current (diagonal) covariance estimate. This is surprising as the authors explicitly avoided this proposal when motivating MH acceptance rate maximization.
In any case, the correctness of the algorithm is neither shown for an independent proposal nor a Markov Chain proposal. Indeed, I would argue that Algorithm 1 generally does not find a sample distributed according to the target distribution. The algorithm assumes that we can create a sample from the target p(x) that can be used to approximate the loss (a bound on the expected acceptance rate). However, if we could find an unbiased sample in the very first iteration of the algorithm, we could immediately stop and there wouldn't be a need for the algorithm at all. Hence, we must assume that the sample drawn from the target is biased (in the beginning); which is indeed a realistic assumption as neither independent MH nor random walk MH will yield an unbiased sample in any reasonable time (for any practical problem). However, this would bias the loss and, consequently, the parameter update for the proposal distribution. In particular, for random walk MH, I would expect the covariance to contract such that the final proposal indeed collapses to a point distribution. This is because the proposal is only required to have support around the observed samples and this area will become smaller over time. I would expect a proof that the sample at iteration k is "better" than a sample drawn at iteration k-1, to show that the bias vanishes over time. Though, I assume that this is hard to show as the proposal parameters form a Markov Chain itself. So at least a rigor empirical study is needed.
Therefore, I would expect a metric measuring the quality of the final sample. The log-likelihood is not such a measure. While the marginalized log-likelihood could measure the quality of the sample, we cannot compute it for any real data/model (which is why we use sampling in the first place). So we need some artificial settings. However, the 1-dimensional toy example is insufficient as MH primarily suffers in high-dimensional spaces. It would be interesting to also report the acceptance rate depending on the number of dimensions of the target distribution. I would assume an exponential decay; even with learning, which might be the reason why the authors only report random walk MH in Section 4.2.
Algorithm 2 does not require to sample from some target distribution but can leverage the observed sample. While the algorithm nicely connects GANs and sampling, the actual value of the algorithm is not fully clear to me. Learning an independent proposal reduces the problem to learning a GAN; and learning a Markov Chain seems only relevant for sampling-based inference; however, we already have a sample from the target distribution, and we can sample more data using a trained GAN.
Minor comments:
- The prefix/in-fix notation of integrals is mixed, e.g. in Eq 19, "dx" appears before the integrand, but "du" appears after the integrand of the inner integral.
UPDATE:
The revised version is much better in empirically demonstrating the value of the method; though, there is still some work needed. First, the artificial examples are still rather low-dimensional where independent MH is expected to perform well. Second, ESS does not help to assess the biasedness of the sample; maybe [1] can help with this. Third, NUTS might be a better baseline than standard HMC which is know to be sensitive to the stepsize/number of leapfrog steps. An effective sample size of 1 suggests that the method did not even start to sample - likely because of a bad choice of the stepsize and/or mass matrix. I would suggest using PyMC3's NUTS implementation. Finally, to show the correctness of the method, I would suggest to 1) let alpha converge to zero such that \phi will be fixed at some point, and 2) ensure that the proposal has full support under the target for any value of \phi. In this case, the sample drawn from the target will be unbiased for large enough n (same arguments as for adaptive MCMC should apply).
The idea of reusing the samples from previous iterations for approximating the loss is interesting and worth exploring.
[1] Jackson Gorham, Lester Mackey. "Measuring Sample Quality with Kernels", https://arxiv.org/abs/1703.01717


For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes. However, this decomposition can fall short in the finite bit-length regime, as it requires non-trivial tuning of hand-crafted codes and assumes infinite computational power for decoding. In this work, we propose Neural Error Correcting and Source Trimming (NECST) codes to jointly learn the encoding and decoding processes in an end-to-end fashion. By adding noise into the latent codes to simulate the channel during training, we learn to both compress and error-correct given a fixed bit-length and computational budget. We obtain codes that are not only competitive against several capacity-approaching channel codes, but also learn useful robust representations of the data for downstream tasks such as classification. Finally, we learn an extremely fast neural decoder, yielding almost an order of magnitude in speedup compared to standard decoding methods based on iterative belief propagation.

Summary of paper: For the finite-bit case of the noisy communication channel model, it is suboptimal to optimize source coding (compression of input) and error correction (fault tolerance for inherent noise in the channel) separately. The authors propose a neural network model (NECST) that is very similar to the standard VAE, except using binary latents with corruption (e.g.,  random bit flipping in the style of a binary symmetric channel). They use VIMCO to optimize through the discrete units. In their experiments, they show that they can outperform a JPEG+ideal channel code model, but perform similarly to a VAE+LDPC (LDPC is a classic error correcting code) setup.
First of all, the paper is quite well written and easily readable. Great work on explaining the motivation and the model -- the writing is clear and explains background knowledge extremely well.
The main contribution in the model is the use of discrete binary latents, instead of the standard continuous latents in a VAE. However, I am uncertain about the novelty of this contribution. There have been numerous works examining discrete latent variables in autoencoders (a random sampling: [1, 2, 3, 4]) and beyond. Furthermore, the method of training through discrete latents is also standard (VIMCO, though one can also imagine using more recent advances like REBAR or RELAX). The only difference would be the addition of noise to the discrete. I would be curious to see how that compares to recent works that have also added noise to discrete latents [5].
Thus, it strikes me that the main contribution of this work would be in comparing against the current best techniques for coding. However, the experiments section is weak, and does not provide significant evidence that the NECST model is better than the alternatives. NECST outperforms JPEG+ideal channel coding, but doesn't do much better than a VAE+LDPC baseline. This suggests that most of the gains comes from the encoder (source coding) model q(\hat{y} | x), instead of the joint training of source coding and error correcting code. It is not surprising that using a neural network to generate codes would provide significant gains. It's not clear that error correcting code aspect (noise in the latents) is particularly important.
Furthermore, in the classification results, the MLP model trained on the discrete codes gets 93% accuracy on noiseless MNIST inputs. You can easily get this accuracy by training logistic regression directly on the pixels. Despite what the authors write, this result suggests that the codes are not very useful for downstream learning. Furthermore, it is unclear why adding random noise to the inputs would significantly improve some of the weaker classifiers. The only reason I can think of is data augmentation, but this has nothing to do with the NECST model.
In conclusion, this is a well written paper, but the novelty is not apparent and the experimental results are weak, and so I am not convinced this is suitable for ICLR.
Additional Questions:
* How is the runtime computed? Specifically, for NECST, do you batch the data and then divide the forward pass time by the batch size? If this is how runtime is computed, it's not surprising that NECST does better, given that batching is cheap with modern hardware. If the actual forward pass time for a single example is cheaper than that of LDPC's belief propagation, then that would be quite promising.
* The authors state that VAEs optimize a lower bound on the marginal log-likelihood p(X), whereas NECST optimizes a lower bound on the mutual information I(X, Y), where Y is the noised code. The authors however do not discuss why one should optimize for mutual information compared to marginal log-likelihood. What are the advantages and disadvantages between the two?
[1] Semi-Supervised Learning with Deep Generative Models (https://arxiv.org/abs/1406.5298)
[2] Discrete Variational Autoencoders (https://arxiv.org/abs/1609.02200)
[3] Neural Discrete Representation Learning (https://arxiv.org/abs/1711.00937)
[4] Discrete Autoencoders for Sequence Models  (https://arxiv.org/abs/1801.09797)
[5] Theory and Experiments on Vector Quantized Autoencoders (https://arxiv.org/pdf/1805.11063.pdf)


INITIALIZED EQUILIBRIUM PROPAGATION FOR BACKPROP-FREE TRAINING
Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation). Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way. In response to this, Scellier & Bengio (2017) proposed Equilibrium Propagation -a method for gradient-based training of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient. Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network. In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation. This feed-forward network learns to approximate the state of the fixed-point using a local learning rule. After training, we can simply use this initializing network for inference, resulting in a learned feedforward network. Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation while requiring fewer steps to converge. This shows how we might go about training deep networks without using backpropagation.

This paper presents an improvement on the local/derivative-free learning algorithm equilibrium propagation. Specifically, it trains a feedforward network to initialize the iterative optimization process in equilibrium prop, leading to greater stability and computational efficiency, and providing a network that can later be used for fast feedforward predictions on test data. Non-local gradient terms are dropped when training the feedforward network, so that the entire system still doesn't require backprop. There is a neat theoretical result showing that, in the neighborhood of the optimum, the dropped non-local gradient terms will be correlated with the retained gradient terms.
My biggest concern with this paper is the lack of significant literature review, and that it is not placed in the context of previous work. There are only 12 references, 5 of which come from a single lab, and almost all of which are to extremely recent papers. Before acceptance, I would ask the authors to perform a literature search, update their paper to include citations to and discussion of previous work, and better motivate the novelty of their paper relative to previous work. Luckily, this is a concern that is addressable during the rebuttal process! If the authors perform a literature search, and update their paper appropriately, I will raise my score as high as 7.
Here are a few related topic areas which are currently not discussed in the paper. *I am including these as a starting point only! It is your job to do a careful literature search. I am completely sure there are obvious connections I'm missing, but these should provide some entry points into the citation web.*
- The "method of auxiliary coordinates" introduces soft (often quadratic) couplings between post- and pre- activations in adjacent layers which, like your distributed quadratic penalty, eliminate backprop across the couplings. I believe researchers have also done similar things with augmented Lagrangian methods. A similar layer-local quadratic penalty also appears in ladder networks.
- Positive/negative phase (clamped / unclamped phase) training is ubiquitous in energy based models. Note though that it isn't used in classical Hopfield networks. You might want to include references to other work in energy based models for both this and other reasons. e.g., there may be some similarities between this approach and continuous-valued Boltzmann machines?
- In addition to feedback alignment, there are other approaches to training deep neural networks without standard backprop. examples include: synthetic gradients, meta-learned local update rules, direct feedback alignment, deep Boltzmann machines, ...
- There is extensive literature on biologically plausible learning rules -- it is a field of study in its own right. As the paper is motivated in terms of biological plausibility, it would be good to include more general context on the different approaches taken to biological plausibility.
More detailed comments follow:
Thank you for including the glossary of symbols!
"Continuous Hopfield Network" use lowercase for this (unless introducing acronym)
"is the set non-input" -> "is the set of non-input"
"$\alpha = ...$ ... $\alpha_j \subset ...$" I could not make sense of the set notation here.
would recommend using something other than rho for nonlinearity. rho is rarely used as a function, so the prior of many readers will be to interpret this as a scalar. phi( ) or f( ) or h( ) are often used as NN nonlinearities.
inline equation after "clamping factor" -- believe this should just be C, rather than \partial C / \partial s.
Move definition of \mathcal O up to where the symbol is first used.
text before eq. 7 -- why train to approximate s- rather than s+? It seems like s+ would lead to higher accuracy when this is eventually used for inference.
eq. 10 -- doesn't the regularization term also decrease the expressivity of the Hopfield network? e.g. it can no longer engage in "explaining away" or enforce top-down consistency, both of which are powerful positive attributes of iterative estimation procedures.
notation nit: it's confusing to use a dot to indicate matrix multiplication. It is commonly used in ML to indicate an inner product between two vectors of the same shape/orientation. Typically matrix multiplication is implied whenever an operator isn't specified (eg x w_1 is matrix multiplication).
eq. 12 -- is f' supposed to be h'? And wasn't the nonlinearity earlier introduced as rho? Should settle on one symbol for the nonlinearity.
This result is very cool. It only holds in the neighborhood of the optimum though. At initialization, I believe the expected correlation is zero by symmetry arguments (eg, d L_2 / d s_2 is equally likely to have either sign). Should include an explicit discussion of when this relationship is expected to hold.
"proportional to" -> "correlated with" (it's not proportional to)
sec. 3 -- describe nonlinearity as "hard sigmoid"
beta is drawn from uniform distribution including negative numbers? beta was earlier defined to be positive only.
Figure 2 -- how does the final achieved test error change with the number of negative-phase steps? ie, is the final classification test error better even for init eq prop in the bottom row than it is in the top?
The idea of initializing an iterative settling process with a forward pass goes back much farther than this. A couple contexts being deep Boltzmann machines, and the use of variational inference to initialize Monte Carlo chains
sect 4.3 -- "the the" -> "to the"


Positive Unlabeled (PU) learning consists in learning to distinguish samples of our class of interest, the positive class, from the counter-examples, the negative class, by using positive labeled and unlabeled samples during the training. Recent approaches exploit the GANs abilities to address the PU learning problem by generating relevant counter-examples. In this paper, we propose a new GAN-based PU learning approach named Divergent-GAN (D-GAN). The key idea is to incorporate a standard Positive Unlabeled learning risk inside the GAN discriminator loss function. In this way, the discriminator asks the generator to converge towards the unlabeled samples distribution while diverging from the positive samples distribution. This enables the generator convergence towards the unlabeled counterexamples distribution without using prior knowledge, while keeping the standard adversarial GAN architecture. In addition, we discuss normalization techniques in the context of the proposed framework. Experimental results show that the proposed approach overcomes previous GAN-based PU learning methods issues, and it globally outperforms two-stage state of the art PU learning performances in terms of stability and prediction on both simple and complex image datasets.

This paper proposed another GAN-based PU learning method. The mathematics in this paper is not easy to follow, and there are many other critical issues.
*****
The clarity is really an issue. First of all, I cannot easily follow the meanings behind the equations. I guess the authors first came up with some concrete implementation and then formalize it into an algorithm. Given the current version of the paper, I am not sure whether this clarity of equations can be fixed without an additional round of review or not.
Moreover, the logic in the story line is unclear to me, especially the 3rd paragraph that seems to be mostly important in the introduction. There are two different binary classification problems, of separating the positive and negative classes, and of separating the given and generated data. I cannot see why the generated data can serve as negative data. This paragraph is discussing GenPU, PGAN and the proposed method, and consequently the motivation of the current paper does not make sense at least to me.
*****
The paper classified PU learning methods into two categories, one-stage methods and two-stage methods. This is interesting. However, before that, they should be classified into two categories, for censoring PU learning and for case-control PU learning. The former problem setting was proposed very early and formalized in "learning classifiers from only positive and unlabeled data", KDD 2008; the latter problem setting was proposed in "presence-only data and the EM algorithm", Biometrics 2009 and formalized in "analysis of learning from positive and unlabeled data", NIPS 2014. Surprisingly, none of these 3 papers was cited. By definition, GAN-based PU learning belongs to the latter problem setting while Rank Prune can only be applied to the former but was included as a baseline method.
The huge difference between these two settings and their connections to learning with noisy labels are known for long time. To be short, class-conditional noise model corrupts P(Y|X) and covers censoring PU, mutual contamination distribution framework corrupts P(X|Y) and covers case-control PU, and mathematically mutual contamination distribution framework is more general than class-conditional noise model and so is case-control PU than censoring PU. See "learning from corrupted binary labels via class-probability estimation", ICML 2015 for more information where the above theoretical result has been proven. An arXiv paper entitled "on the minimal supervision for training any binary classifier from only unlabeled data" has some experimental results showing that methods for class-conditional noise model cannot handle mutual contamination distributions. The situation is similar when applying censoring PU methods to case-control PU problem setting.
Furthermore, the class-prior probability pi is well-defined and easy to estimate in censoring PU, see "learning classifiers from only positive and unlabeled data" mentioned above. However, it is not well-defined in case-control PU due to an identifiability issue described in "presence-only data and the EM algorithm" mentioned above. Thus, the target to be estimated is defined as the maximal theta such that theta*P(X|Y)<=P(X) following "estimating the class prior and posterior from noisy positives and unlabeled data", NIPS 2016. BTW, "mixture proportion estimation via kernel embedding of distributions" is SOTA in class-prior estimation; the previous NIPS paper was written earlier and accepted later.
In summary, as claimed in the paper and shown in Table 1 in the introduction, all discriminative PU methods and GenPU require to know pi for learning. This is true, but this is because they are designed for a more difficult problem setting---learning classifiers and estimating pi are both more difficult. Lacking some basic knowledge of PU learning is another big issue.
*****
The novelty is to be honest incremental and thus below the bar of ICLR. The significance is similarly poor, due to that the experiments mixed up methods for censoring PU and those for case-control PU. What is more, F1-score is a performance measure for information retrieval rather than binary classification. We all know GANs are pretty good at MNIST but not CIFAR-10. In fact, GenPU has a critical issue of mode collapse, and this is why GenPU reports 1-vs-1 rather than 5-vs-5 on MNIST. Even though, I still think GenPU makes much more sense than PGAN and D-GAN.


LEARNING TO SOLVE CIRCUIT-SAT: AN UNSUPERVISED DIFFERENTIABLE APPROACH
Recent efforts to combine Representation Learning with Formal Methods, commonly known as Neuro-Symbolic Methods, have given rise to a new trend of applying rich neural architectures to solve classical combinatorial optimization problems. In this paper, we propose a neural framework that can learn to solve the Circuit Satisfiability problem. Our framework is built upon two fundamental contributions: a rich embedding architecture that encodes the problem structure, and an end-to-end differentiable training procedure that mimics Reinforcement Learning and trains the model directly toward solving the SAT problem. The experimental results show the superior out-of-sample generalization performance of our framework compared to the recently developed NeuroSAT method.

The paper makes a nice contribution to solving Circuit-SAT problem from a Neuro-Symbolic approach, particularly, 1) a novel DAG embedding with a forward layer and a reverse layer that captures the structural information of a circuit-sat input. 2) Compared with Selsam et al.'s work of Neuro-SAT, the proposed model in this paper, DG-DAGRNN, directly produces an assignment of variables, and the method is unsupervised and end-to-end differentiable. 3) Empirical experiments on random k-SAT and random graph k-coloring instances that support the authors' claim on better generalization ability.
The paper is lucid and well written, I would support its acceptance at ICLR. Though I have a few comments and questions for the authors to consider.
- In figure 1 (a), what are x11, x12, etc?
- When comparing the two approaches of Neuro-Symbolic methods, besides the angles of optimality and training cost, it is worth to mention that the first one that based on classical algorithms always has a correctness guarantee, while the second one (learning the entire solution from scratch) usually does not.
- Section 4.1, as a pure decision problem, solving SAT means that giving a yes/no answer (i.e., a classification); while for practical purposes, solving SAT means that producing a model (i.e., a witness) of the formula if it is SAT. This can be misleading for some readers when the authors mentioning "solving SAT", and it would be clear if the authors could make a distinction when using such terms.
- Section 4.1, "without requiring to see the actual SAT solutions during training", again, what is the meaning of "solutions" is not very clear at this point. Readers may realize the experiments in the paper only train with satisfiable formulae from the afterward description, so the "solutions" indicates the assignments of variables. But it would be better to make it clear.
- Section 4.1/The Evaluator Network, "one can show also show that min() < S_min() <= S_max() < max()", what is the ordering relation (i.e., < and <=) here? It is a bit confusing if a forall quantifier for inputs (a_1, ... a_n) is required here.
- Section 4.1/The Evaluator Network, how does the temperature affect the results of R_G? It would be helpful to show their dynamics.
- Section 4.1/Optimization, "if the input circuit is UNSAT, one can show that the maximum achievable values for S_\theta is 0.5", it would be better to provide a brief description of how it is guaranteed. Also, this seems to be suggesting the DG-SAGRNN solver has no false positives, i.e., it will never produce a satisfiable result for unsatisfiable formulae? This would be interesting toward some semi-correctness if the answer is yes.
- Section 5.1, are the testing data all satisfiable formulae? If yes, then the figure 2 shows there is a number of satisfiable formulae but both the models cannot produce correct results -- is that a correct understanding of figure 2? If not, then what is the ground truth?
- I would love to see more experiments on SAT instances with a moderate number of variables but from real-world applications. It would be interesting to see how the model utilizes the rich structural information of instances from real applications (instead of randomly generated formulae).
- The training time and testing time(per instance) are not reported in the experiments.


Deep artificial neural networks with spatially repeated processing (a.k.a., deep convolutional ANNs) have been established as the best class of candidate models of visual processing in the primate ventral visual processing stream. Over the past five years, these ANNs have evolved from a simple feedforward eight-layer architecture in AlexNet to extremely deep and branching NASNet architectures, demonstrating increasingly better object categorization performance. Here we ask, as ANNs have continued to evolve in performance, are they also strong candidate models for the brain? To answer this question, we developed Brain-Score, a composite of neural and behavioral benchmarks for determining how brain-like a model is, together with an online platform where models can receive a BrainScore and compare against other models. Despite high scores, typical deep models from the machine learning community are often hard to map onto the brain's anatomy due to their vast number of layers and missing biologically-important connections, such as recurrence. To further map onto anatomy and validate our approach, we built CORnet-S: an ANN guided by Brain-Score with the anatomical constraints of compactness and recurrence. Although a shallow model with four anatomically mapped areas and recurrent connectivity, CORnet-S is a top model on Brain-Score and outperforms similarly compact models on ImageNet. Analyzing CORnet-S circuitry variants revealed recurrence as the main predictive factor of both Brain-Score and ImageNet top-1 performance.

This is an interesting paper in which the authors propose a shallow neural network, the architecture of which was optimized to maximize brain score, and show that it outperforms other shallow networks at imagenet classification. The goal of this paper — allowing brain data to improve our neural nets — is great. The paper is well written (except for a comment on clarity right below), and presents an interesting take on solving this problem.
It is a little unclear how the authors made CORnet optimize brain score: “However, note that CORnet-S was developed using Brain-Score as a guiding benchmark and although it was never directly used in model search or optimization, testing CORnet-S on Brain-Score is not a completely independent test.” Making these steps clearer is crucial for evaluating better what the model means. In the discussion “We have tested hundreds of architectures before finding CORnet-S circuitry and thus it is possible that the proposed circuits could have a strong relation to biological implementations.” implies that the authors trained models with different architectures until the brain score was maximized after training. A hundred(s) times of training on 2760 + 2400 datapoints are probably plenty to overfit the brainscore datasets. The brain score is probably compromised after this, and it would be hard to make claims about the results on the brain modeling side. The author acknowledge this limitation, but perhaps a better thing is to add an additional dataset, perhaps one from different animal recordings, or from human fMRI?
Arguably, the goal of the paper is to obtain a model that overpowers other simpler models, and not necessarily to make claims about the brain. The interesting part of the paper is that the shallow model does work better than other shallow models. The authors mention that brain score helps CORnet be better at generalizing to other datasets. Including these results would definitely strengthen the claim since both brain score and imagenet have been trained on hundreds of times so far.
Another way to show that brain score helps is to show it generalizes above or differently from optimizing other models. What would have happened if the authors stuck to a simple, shallow model and instead of optimizing brain score optimized performance (hundreds of times) on some selected image dataset (this selection dataset is separate from imagenet, but the actual training is done on imagenet) and then tested performance on imagenet? Is the effect due to the brain or to the independent testing on another dataset?


Hierarchically embedding smaller networks in larger networks, e.g. by increasing the number of hidden units, has been studied since the 1990s. The main interest was in understanding possible redundancies in the parameterization, as well as in studying how such embeddings affect critical points. We take these results as a point of departure to devise a novel strategy for escaping from flat regions of the error surface and to address the slow-down of gradient-based methods experienced in plateaus of saddle points. The idea is to expand the dimensionality of a network in a way that guarantees the existence of new escape directions. We call this operation the opening of a tunnel. One may then continue with the larger network either temporarily, i.e. closing the tunnel later, or permanently, i.e. iteratively growing the network, whenever needed. We develop our method for fully-connected as well as convolutional layers. Moreover, we present a practical version of our algorithm that requires no network structure modification and can be deployed as plug-and-play into any current deep learning framework. Experimentally, our method shows significant speed-ups.
• We develop a practical version of our algorithm that requires no network structure modification and can be deployed as plug-and-play into any current deep learning framework.
• We show experimentally that our method escapes effectively from flat areas during training.
In Section 2, we provide background, motivation, and explain in detail how to open the tunnel by solving an interposed optimization problem. We discuss the fully connected case as well as the practically relevant case of convolutional layers. In Section 3, we describe how to close the tunnel, i.e. how to project back the expanded network onto one of the original dimensionality. We also propose a practical version of our idea that does not require an explicit modification of the network structure, based on a well-chosen reorganization of the weights. Finally, experiments and related work are presented in Sections 4 and 5 respectively.

This is an interesting submission and I appreciate especially connecting to a body of literature which is not normally well known in our community (e.g. Fukumizu&Amari). I think the perspective is definitely new and probably quite relevant not only for practical approaches to escape saddles but also to understand learning in deep learning.  I have a few notes and suggestions:
1) Name of the paper:
I think is not descriptive of the approach and actually the words “magic” makes it sound strange. I think this will reduce the amount of people reading the work. Please consider something more descriptive like: “Escaping saddle points by increasing capacity”. Or something else more inspired, but that also hints what the work is about.
2) Notation:
The notation used is not ML friendly (or generically) to the average reader. I strongly suggest to use b_v for bias, W_uv for weights, and not theta_v and theta_uv which is not typical notation. ‘u’ and ‘v’ are somewhat non-typical choices either, though I understand that they come from the graph notation. Transfer functions are usual sigma. In the text you explain the process by starting with a u’ and then add the clone which is u. Normally you should have started with u and add the clone that is u’. x_u for the value of unit u (assuming this is in the middle of a deep net) is also quite a strange notation. I can guess the authors might be from a slightly different community, but I’m worried about people from the target audience (ICLR) being turned away from the work or even worse confused because of notations.
3) Related work
There is the Net2Net work that is related to what is going on here that is not cited (https://arxiv.org/pdf/1511.05641.pdf). I think there was some follow-up work after this.
4) Symmetry breaking
I do not understand how symmetry is broken. If I clone a unit, and have a new variant of it u’ that now has the same outbound connections but multiplied by alpha (while the original unit by 1-alpha) then while the norm of the gradients differ, their direction does not. Wouldn’t this mean that the units will track each other and hence no tunnel is open? In Net2Net dropout was used to break symmetry (i.e. a source of noise that would pick one path over the other). There is no source of noise here to break symmetry.
5) Diagrams and analysis
Connected to this, I feel like this could have been represented clearly with a diagram showing the net before and after. There could be some analysis, a more extended discussion of where the symmetry breaking comes from, empirical evidence that it does. I’m not necessarily worried that experiments are not scaled up, I’m more concerned that the hypothesis and solution is only tested by means of change in performance. What is this tunnel doing? How does it change the Hessian at the saddle? Any visualization to reinforce the intuition of what the approach is doing?
6) Closing tunnels & re-organizing
I don’t understand the mechanism for reorganizing weights and closing tunnels. It seems first of all to confirm my intuition that there is no symmetry breaking since we can “close” the tunnel by simple algebraic manipulation. So if those two units always stay in sync how do you actually change the error surface? How do you take advantage of this extra capacity to solve anything. Regardless, when it comes to re-organizing, it seems you pick two of these units that are in sync (previous tunnel I guess) and collapse them to open a new tunnel, right? How does this change anything? Which unit needs to be cloned? Any? So then why is the previous tunnel not efficient anymore?


A convolutional neural network for image classification can be constructed mathematically since it can be regarded as a multi-period dynamical system. In this paper, a novel approach is proposed to construct network models from the dynamical systems view. Since a pre-activation residual network can be deemed an approximation of a time-dependent dynamical system using the forward Euler method, higher order Runge-Kutta methods (RK methods) can be utilized to build network models in order to achieve higher accuracy. The model constructed in such a way is referred to as the Runge-Kutta Convolutional Neural Network (RKNet). RK methods also provide an interpretation of Dense Convolutional Networks (DenseNets) and Convolutional Neural Networks with Alternately Updated Clique (CliqueNets) from the dynamical systems view. The proposed methods are evaluated on benchmark datasets: CIFAR-10/100, SVHN and ImageNet. The experimental results are consistent with the theoretical properties of RK methods and support the dynamical systems interpretation. Moreover, the experimental results show that the RKNets are superior to the state-of-the-art network models on CIFAR-10 and on par on CIFAR-100, SVHN and ImageNet.

The paper takes the perspective that a ResNet can be seen as a discrete time dynamical system approximation, and then uses that insight to introduce a Runge Kutta style continuous dynamical system.  Results are achieved that outperform DenseNet and CliqueNet
Quality
- The connection between deep nets and dynamical systems is an important one, as a small but growing literature demonstrates.
- The quality of this paper is difficult to extract, given rather long and unclear definitions of various key choices, at key points opting instead for a speculative connection to neurosciences (see clarity below).
- The empirical results could make this paper compelling, but the comparisons are rather shallow, in that they compare on only a few datasets and don't compare extensively to many of the state of the art methods.   For example, in Table 2 DenseNet outperforms the IRKNets, but this fact is not highlighted.  Overall this and other key omissions create the concern that the results are not particularly strong.
- One key reference that is missing is the careful work Chen et al https://arxiv.org/abs/1806.07366.  This work provides an excellent basis for how to present these ideas rigorously.
Clarity
- The connections to the brain -- such as "visual cortex", "ventral stream", etc -- are not rigorous and are unrelated to the paper at hand; really all connections to brain should be removed.
- Figures 2 and 3 are the central architectural choices, but the writing around them does not clarify why all the choices have been made, and what the implications are.
Originality
- This is one of a small but growing number of papers connecting deep nets to ODE solvers.  Chen et al should be cited (Chen et al https://arxiv.org/abs/1806.07366).
- There appears to be sufficient originality.
Significance
- hard to determine with unclear exposition and rather incomplete empirical results.


This paper proposes Dirichlet Variational Autoencoder (DirVAE) using a Dirichlet prior for a continuous latent variable that exhibits the characteristic of the categorical probabilities. To infer the parameters of DirVAE, we utilize the stochastic gradient method by approximating the Gamma distribution, which is a component of the Dirichlet distribution, with the inverse Gamma CDF approximation. Additionally, we reshape the component collapsing issue by investigating two problem sources, which are decoder weight collapsing and latent value collapsing, and we show that DirVAE has no component collapsing; while Gaussian VAE exhibits the decoder weight collapsing and Stick-Breaking VAE shows the latent value collapsing. The experimental results show that 1) DirVAE models the latent representation result with the best log-likelihood compared to the baselines; and 2) DirVAE produces more interpretable latent values with no collapsing issues which the baseline models suffer from. Also, we show that the learned latent representation from the DirVAE achieves the best classification accuracy in the semi-supervised and the supervised classification tasks on MNIST, OMNIGLOT, and SVHN compared to the baseline VAEs. Finally, we demonstrated that the DirVAE augmented topic models show better performances in most cases.

In this paper, authors proposes an algorithm to use Dirichlet prior on the variational auto-encoder (VAE). They used this prior as natural conjugate to likelihood distributtion of multinomial (categorical). The paper proposes a way to use scalability power of VAE for data distributed by categorical distribution. In order to apply reparametrization trick, authors have used iid Gamma random variable to construct draw from Dirichlet distribution and have used approximation with inverse gamma CDF,  it is discussed how this method has better performance than other approximations method for gamma distribution such as Weibull and logistic Gaussian.
Authors pointed out, one of the weak points in competing models such as  Guassian softmax prior or Griffith -Engen-McCloskey prior which has been used for Stick breaking VAE is to not encouraging of having multi-modal posteriori, while this prior empower having multi-modal posteriori distribution which give them advantage over previous papers.
 In experimental results, paper has used different datasets of MNIST, MNIST+rotation , OMNIGLOT , 20newsgroup and RCVI and used different measures to compare the existing method with the baselines.
To summarize the contribution of this paper, following three points can be named as main contribution of this paper:
- proposed a Dirichlet prior, for categorical likelihood which encourages having multi-modal posteriori. paper demonstrates couple of techniques  to apply the reparametrization trick on Dirichlet distribution, by using sum of iid Gamma random variables.
- used method of moments estimator to update the hyper parameter of the Dirichlet distribution which helps to have closer approximation of log likelihood. They update hyper-parameters after every few updates of VAE parameters.
-discussed how to overcome  Stick-breaking VAE “component collapse” issue. Experiments show superior results on supervised and semi supervised, and authors claimed the main reason of this superiority being due to not having disadvantage of component collapse which happens in SBVAE.
Quality and Novelty:
claims in paper are supported by proofs and/or experimental results and there does not exist significant technical issues with the details of claims made in this paper and proofs provided. There are following issues with novelty and quality of paper that I would like discuss them under following three points:
- Authors need to be clear about the motivation of the paper, if the motivation of the paper is to encourage the multi-modality in posteriori distribution, using Gaussian prior and methods like normalizing flow Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." arXiv preprint arXiv:1505.05770 (2015) or similar may be able to do the same work in which case paper should compare its results to those ideas which has not been done in this paper.
- second appealing point that this paper can make is to use Dirichlet prior for the purposes like community detection, topic modeling and LDA  etc etc. In this case, I did not find significant difference between the proposed method and what is found in Srivastava, Akash, and Charles Sutton. "Autoencoding variational inference for topic models." arXiv preprint arXiv:1703.01488 (2017), but due to the encourages of multi-modality authors show in average DirVAE performs better in measures like perplexity and NPMI. Under this condition, my main concern is interpretablity of posteriori. That will be discussed under next point
- Main motivation behind using Dirichlet prior, is to have posteriori with a few significant related topic and many unrelated topic for every word. By changing the concentration parameter in stick-breaking, it is possible that performance of stick-breaking method increase in perplexity and NPMI scores in cost of loosing interpretability of the model. So having higher concentration parameter can show better performance in the cost of interpretablity that put second point of the paper at risk
Clarity:
The paper is well written and previous relevant methods have been reviewed well. The organization of paper is good, experiments well explained and proofs and mathematical reasoning are clear.
Significance of experiments:
As discussed,in previous sections, the results show superior performance and compared to other methods on semi-supervised and supervised classification on different datasets. Also it has shown in average better perplexity and NPMI score for topic modeling, the only issue can be these scores come as cost of interpretablity of the model. Also it is possible that other competing models can be matching to this results if they do not aim for sparse posteriori.


We propose a preconditioned accelerated stochastic gradient method suitable for large scale optimization. We derive sufficient convergence conditions for the minimization of convex functions using a generic class of diagonal preconditioners and provide a formal convergence proof based on a framework originally used for on-line learning. Inspired by recent popular adaptive per-feature algorithms, we propose a specific preconditioner based on the second moment of the gradient. The sufficient convergence conditions motivate a critical adaptation of the perfeature updates in order to ensure convergence. We show empirical results for the minimization of convex and non-convex cost functions, in the context of neural network training. The method compares favorably with respect to current, first order, stochastic optimization methods.

This paper presents a preconditioned variant of Nesterov's Accelerated Gradient (NAG) for use with Stochastic Gradients. The appears to be an interesting direction given that Nesterov's Acceleration empirically works better than the Heavy Ball (HB) method. There are a few issues that I'd like to understand:
[0] The authors make the assumption A 1-3, wherein,
- why should the momentum parameter mu drop exponentially? This is not required for the convex case, see Nesterov's (1983) scheme for the smooth case and the smooth+strongly convex case.
- why should the bounded rate of change (A3) even be satisfied in the first case? Does this hold true even in simple settings such as optimizing a quadratic/for log loss with stochastic and/or deterministic gradients? In short, is this a reasonable assumption (which at the least holds for certain special cases) or one that helps in obtaining a convergence statement (and which is not true even in simple situations)?
[1] Convergence under specific assumptions aside, I am not sure what is the significance of the regret bound provided. In particular, can the authors provide any reasoning as to why this regret bound is better compared to ADAM or its variants [Reddi et al, 2018]? Is there a faster rate of convergence that this proposed method obtains? I dont think a regret bound is reflective of any (realistic) practical behavior and doesn't serve as a means to provide a distinction between two algorithms. What matters is proving a statement that offers a rate of convergence. Other forms of theoretical bounds do not provide any forms of distinction between algorithms of the same class (adagrad, rmsprop, adam and variants) and are not reflective of practical performance.
[2] The scope of empirical results is rather limited. While I like the notion of having experiments for convex (with least squares/log loss) and non-convex losses, the experiments for the non-convex case are fairly limited in scope. In order to validate the effectiveness of this scheme, performing experiments on a suitable benchmark of some widely used and practically applicable convnet with residual connections/densenet for cifar-10/imagenet is required to indicate that this scheme indeed works well, and to show that it doesn't face issues with regards to generalization (see Wilson et al, 2017 - marginal value of adaptive gradient methods in machine learning).
[3] The paper claims of an "accelerated stochastic gradient" method - this really is a loosely used term for the paper title and its contents. There are efforts that have specifically dealt with acclerating SGD in a very precise sense which the paper doesn't refer to:
- Ghadimi and Lan (2012, 2013), Dieuleveut et al (2017) accelerate SGD with the bounded variance assumption.
- Accelerating SGD is subtle in a generic sense. See Jain et al. (2017) "Accelerating Stochastic Gradient Descent for Least Squares Regression", Kidambi et al. (2018) "On the insufficiency of existing momentum schemes for stochastic optimization". The former paper presents a rigorous understanding of accelerating SGD. The latter paper highlights the insufficiencies of existing schemes like HB or NAG for stochastic optimization.