FFJORD: FREE-FORM CONTINUOUS DYNAMICS FOR SCALABLE REVERSIBLE GENERATIVE MODELS
Reversible generative models map points from a simple distribution to a complex distribution through an easily invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the logdensity. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, improving the state-ofthe-art among exact likelihood methods with efficient sampling.
Reversible generative models use cheaply invertible neural networks to transform samples from a fixed base distribution. Examples include NICE (Dinh et al., 2014), Real NVP (Dinh et al., 2017), and Glow (Kingma & Dhariwal, 2018). These models are easy to sample from, and can be trained by maximum likelihood using the change of variables formula. However, this requires placing awkward restrictions on their architectures, such as partitioning dimensions or using rank one weight matrices, in order to avoid an O(D 3 ) cost determinant computation.

Summary:
This paper discusses an advance in the framework of normalizing flows for generative modeling, named FFJORD. The authors consider normalizing flows in the form of ordinary differential equations, as also discussed in [1]. Their contributions are two-fold: (1) they use an unbiased estimator of the likelihood of the model by approximating the trace of the jacobian with Hutchinson’s trace estimator, (2) they have implemented the required ODE solvers on GPUs.
The models are evaluated on a density estimation task on tabular data and two image datasets (MNIST and CIFAR10), as well as on variational inference for auto-encoders, where the datasets MNIST, Omniglot, Freyfaces and Caltech Silhouettes are considered.
The authors argue that the trace estimator, in combination with reverse-mode automatic differentiation to compute vector-Jacobian products, leads to a computational cost of O(D), instead of O(D^2) for the exact trace of the jacobian.
They compare this to the cost of computing a Jacobian determinant for finite flows, which is O(D^3) in general. They argue that in general all works on finite flows have adjusted their architectures for the flows to avoid the O(D^3) complexity, and that FFJORD has no such restriction.
However, I would like the authors to comment on the following train of thought: autoregressive models, such as MAF, as well as IAF (inverse of an autoregressive model) do not require O(D^3) to compute jacobian determinants as the jacobian is of triangular form. Note however, they are still universal approximators if sufficient flows are applied, as any distribution can be factorized in an autoregressive manner. With this in mind, I find the red cross for MAF under free-form Jacobian slightly misleading. Perhaps I misunderstood something, so please clarify.
Another topic that I would like the authors to comment on is efficiency and practical use. One of the main points that the authors seem to emphasise, is that contrary to autoregressive models, which require D passes through the model to sample a datapoint of size D, FFJORD is a ‘single-pass’ model, requiring only one pass through the model. They therefore indicate that they can do efficient sampling. However, for FFJORD every forward pass requires a pass through an ODE solver, which as the authors also state, can be very slow. I could imagine that this is still faster than an autoregressive model, but I doubt this is actually of comparable speed to a forward pass of a finite flow such as glow or realNVP.
On the other hand, autoregressive models do not require D passes during training, whereas, if I understand correctly, FFJORD relies on two passes through ODE solvers, one for computing the loss, and a second to compute the gradient of the loss with respect to model parameters. So autoregressive models should train considerably faster. The authors do comment on the fact that FFJORD is slower than other models, but they do not give a hint as to how much slower it is. This would be of importance for practical use, and for other people to consider using FFJORD in future work.
For the density estimation task, FFJORD does not have the best performance compared other baselines, except for MNIST, for which the overall best model was not evaluated (MAF-DDSF). For variational inference FFJORD is stated to outperform all other flows, but the models are only evaluated on the negative evidence lower bound, and not on the negative log-likehood (NLL). I suspect the NLL to be absent from the paper as it requires more computation, and this takes a long time for FFJORD. Without an evaluation on NLL the improvement over other methods is questionable. Even if the improvement still holds for the NLL, the relative improvement might not weigh heavily enough against increased runtime. FFJORD does require less memory than its competitors.
The improved runtime by implementing the ODE solvers on GPU versus the runtime on a CPU would be useful, given that this is listed as one of the main contributions.
Besides these questions/comments, I do think the idea of using Hutchinsons trace estimator is a valid contribution, and the experimental validation of continuous normalizing flows is of interest to the research community. Therefore, in my opinion, the community will benefit from the information in this paper, and it should be accepted. However I do wish for the authors to address the above questions as it would give a clearer view of the practical use of the proposed model.

See below for comments and questions:
Quality
The paper has a good setup, and is well structured. The scope and limitations section is very much appreciated.
Clarity
The paper is clearly written overall. The only section I can comment on is the related work section, which is not the best part of the paper. The division in normalizing flows and partitioned transformations is a bit odd. Partitioned transformations surely are also normalizing flows. Furthermore IAF by Kingma et al. is put in the box of autoregressive models, whereas it is the inverse of an autoregressive model, such that it does not have the D-pass sample problem. For a reader who is not too familiar with normalizing flows literature, I think this section is a little confusing. Furthermore, there is no related work discussed on continuous time flows, such as (but not limited to) [2].
Originality
The originality of the paper is not stellar, but sufficient for acceptance.
Significance
The community can benefit from the experimental analysis of continuous time flows, and the GPU implementation of the ODE solver. Therefore I think this work is significant.
Detailed questions/comments:
1. In section 4.2, as an additional downside to MAF-DDSF, the authors argue that sampling cannot be performed analytically. Since FFJORD needs to numerically propagate the ODE, I do not think FFJORD can sample analytically either. Is this correct?
2. The authors argue that they have no restriction on the architecture of the function f, even if they have O(D) estimation of the trace of the jacobian. However, they also say they make use of the bottle-neck trick to reduce the variance that arises due to Hutchinson’s estimate of the trace. This seems like a limitation on the architecture to me. Can the authors comment?
3. In B.1 in the appendix, the street view house numbers dataset is mentioned, but no results appear in the main text, why not?
4. In the results section, it is not clear to me which numbers of the baselines for different datasets are taken from other papers, and which numbers are obtained by the authors of this paper. Please clarify.
5. In the conclusions, when discussing future work, the authors state that they are interested in reducing the number of function evaluations in the ODE solvers. In various disciplines many people have worked on this problem for a long time. Do the authors think major improvements are soon to be made?
6. In section 5.2 the dependence of the number of function evaluations (NFE) on the data dimension D is discussed. As a thought experiment they use the fact that going from an isotropic gaussian distribution (in any D), to an isotropic gaussian distribution has a corresponding differential equation of zero. This should convince the reader that NFE is independent of D. However, this seems to me to be such a singular example, that I gain no insight from it, and it is not very convincing. Do the authors agree that this particular example does not add much? If not, please explain.
[1] Chen et al. Neural ordinary differential equations. NIPS 2018
[2] Chen et al. Continuous-time flows for deep generative models.
**** EDIT *****
I have read the response of the authors and appreciate their clarifications and the additional information on the runtimes. See my response below for the concern that remains about the absence of the estimate of the log likelihood for the VAE experiments. Besides this issue, the other comments/answers were satisfactory, and I think this paper is of interest to the research community, so I will stick with my score.


In image classification tasks, the ability of deep convolutional neural networks (CNNs) to deal with complex image data has proved to be unrivalled. Deep CNNs, however, require large amounts of labeled training data to reach their full potential. In specialized domains such as healthcare, labeled data can be difficult and expensive to obtain. One way to alleviate this problem is to rely on active learning, a learning technique that aims to reduce the amount of labelled data needed for a specific task while still delivering satisfactory performance. We propose a new active learning strategy designed for deep neural networks. This method improves upon the current state-of-the-art deep Bayesian active learning method, which suffers from the mode collapse problem. We correct for this deficiency by making use of the expressive power and statistical properties of model ensembles. Our proposed method manages to capture superior data uncertainty, which translates into improved classification performance. We demonstrate empirically that our ensemble method yields faster convergence of CNNs trained on the MNIST and CIFAR-10 datasets.

The paper shows that Bayesian neural networks, trained with Dropout MC (Gal et al.) struggle to fully capture the posterior distribution of the weights.
This leads to over-confident predictions which is problematic particularly in an active learning scenario.
To prevent this behavior, the paper proposes to combine multiple Bayesian neural networks, independently trained with Dropout MC, to an ensemble.
The proposed method achieves better uncertainty estimates than a single Bayesian neural networks model and improves upon the baseline in an active learning setting for image classification.
The paper addresses active deep learning which is certainly an interesting research direction since in practice, labeled data is notoriously scarce.
However, the paper contains only little novelty and does not provide sufficiently new scientific insights.
It is well known from the literature that combining multiply neural networks to an ensemble leads to better performance and uncertainty estimates.
For instance, Lakshminarayanan et al.[1] showed that Dropout MC can produce overconfident wrong prediction and, by simply averaging prediction over multiple models, one achieves better performance and confidence scores. Also, Huand et al. [2] showed that by taking different snapshots of the same network at different timesteps performance improves.
It would also be great if the paper could related to other existing work that uses Bayesian neural networks in an active learning setting such as Bayesian optimization [3, 4] or Bandits[5].
Another weakness of the paper is that the empirical evaluation is not sufficiently rigorous:
1) Besides an comparison to the work by Lakshminarayanan et. al, I would also like to have seen a comparison to other existing Bayesian neural network approaches such as stochastic gradient Markov-Chain Monte-Carlo methods.
 2) To provide a better understanding of the paper, it would also be interesting to see how sensitive it is with respect to the ensemble size M.

 3) Furthermore, for the experiments only one neural network architecture was considered and it remains an open question, how the presented results translate to other architectures. The same holds for the type of data, since the paper only shows results for image classification benchmarks.

 4) Figure 3: Are the results averaged over multiple independent runs? If so, how many runs did you perform and could you also report confidence intervals? Since all methods are close to each other, it is hard to estimate how significant the difference is.

[1] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundel
NIPS 2017
[2] Gao Huang and Yixuan Li and Geoff Pleiss and Zhuang Liu and John E. Hopcroft and Kilian Q. Weinberger
    Snapshot Ensembles: Train 1, get {M} for free}
    ICLR 2017
[3] Bayesian Optimization with Robust Bayesian Neural Networks
    J. Springenberg and A. Klein and S.Falkner and F. Hutter
    NIPS 2016

[4] J. Snoek and O. Rippel and K. Swersky and R. Kiros and N. Satish and N. Sundaram and M. Patwary and Prabhat and R. Adams
    Scalable Bayesian Optimization Using Deep Neural Networks
    ICML 2015
[5] Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
    Carlos Riquelme, George Tucker, Jasper Snoek
    ICLR 2018


RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING
Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay. We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy. Using a single network architecture and fixed set of hyperparameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and matches the state of the art on DMLab-30. It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

In this submission, the authors investigate using recurrent networks in distributed DQN with prioritized experience replay on the Atari and DMLab benchmarks. They experiment with several strategies to initialize the recurrent state when processing a sub-sequence sampled from the replay buffer: the best one consists in re-using the initial state computed when the sequence was originally played (even if it may now be outdated) but not doing any network update during the first k steps of the sequence (“burn-in” period). Using this scheme with LSTM units on top of traditional convolutional layers, along with a discount factor gamma = 0.997, leads to a significant improvement on Atari over the previous state-of-the-art, and competitive performance on DMLab.
The proposed technique (dubbed R2D2) is not particularly original (it is essentially “just” using RNNs in Ape-X), but experiments are thorough, investigating several important aspects related to recurrence and memory to validate the approach. These findings are definitely quite relevant to anyone using recurrent networks on RL tasks. The results on Atari are particularly impressive and should be of high interest to researchers working on this benchmark. The fact that the same network architecture and hyper-parameters also work pretty well on DMLab is encouraging w.r.t. the generality of the method.
I do have a couple of important concerns though. The first one is that a few potentially important changes were made to the “traditional” settings typically used on Atari, which makes it difficult to perform a fair comparison to previously published results. Using gamma = 0.997 could by itself provide a significant boost, as hinted by results from “Meta-Gradient Reinforcement Learning” (where increasing gamma improved results significantly compared to the usual 0.99). Other potentially impactful changes are the absence of reward clipping (replaced with a rescaling scheme) and episodes not ending with life loss: I am not sure whether these make the task easier or harder, but they certainly change it to some extent (the “despite this” above 5.1 suggests it would be harder, but this is not shown empirically). Fortunately, this concern is partially alleviated by Section 6 that shows feedforward networks do not perform as well as recurrent ones, but this is only verified on 5 games: a full benchmark comparison would have been more reassuring (as well as running R2D2 with more “standard” Atari settings, even if it would mean using different hyper-parameters on DMLab).
The second important issue I see is that the authors do not seem to plan to share their code to reproduce their results. Given how time consuming and costly it is to run such experiments, and all potentially tricky implementation details (especially when dealing with recurrent networks), making this code available would be tremendously helpful to the research community (particularly since this paper claims a new SOTA on Atari). I am not giving too much weight to this issue in my review score since (unfortunately) the ICLR reviewer guidelines do not explicitly mention code sharing as a criterion, but I strongly hope the authors will consider it.
Besides the above, I have a few additional small questions:
1. “We also found no benefit from using the importance weighting that has been typically applied with prioritized replay”: this is potentially surprising since this could be “wrong”, mathematically speaking. Do you think this is because of the lack of stochasticity in the environments? (I know Atari is deterministic, but I am not sure about DMLab)
2. Fig. 3 (left) shows R2D2 struggling on some DMLab tasks. Do you have any idea why? The caption of Table 3 in the Appendix suggests the absence of specific reward clipping may be an issue for some tasks, but have you tried adding it back? I also wonder if maybe training a unique network per task may make DMLab harder, since IMPALA has shown some transfer learning occurring between DMLab tasks? (although the comparison might be to the “deep experts” version of IMPALA — this is not clear in Fig. 3 — in which case this last question would be irrelevant)
3. In Table 1, where do the IMPALA (PBT) numbers on DMLab come from? Looking at the current arxiv version of their paper, their Fig. 4 shows it goes above 70% in mean capped score, while your Table 1 reports only 61.5%. I also can’t find a median score being reported on DMLab in their paper, did you try to compute it from their Fig. 9? And why don’t you report their results on Atari?
4. Table 4’s caption mentions “30 no-op starts” but you actually used the standard “random starts” setting, right? (not a fixed number of 30 no-ops)
And finally a few minor comments / suggestions:
- In the equation at bottom of p. 2, it seems like theta and theta- (the target network) have been accidentally swapped (at least compared to the traditional double DQN formula)
- At top of p. 3 I guess \bar{delta}_i is the mean of the delta_i’s, but then the index i should be removed
- In Fig. 1 (left) please clarify which training phase these stats are computed on (whole training? beginning / middle / end?)
- p. 4, “the true stored recurrent states at each step”: “true” is a bit misleading here as it can be interpreted as “the states one would obtain by re-processing the whole episode from scratch with the current network” => I would suggest to remove it, or to change it (e.g. “previously”). By the way, I think it would have been interesting to also compare to these states recomputed “from scratch”, since they are the actual ground truth.
- I think you should mention in Table 1’s caption that the PBT IMPALA is a single network trained to solve all tasks
- Typo at bottom of p. 7, “Indeed, Table 1 that even...”
Update: score updated to 8 (from 7) following discussion below


We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, procedures that are much slower than a SGD step. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration with just two passes over the network. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained. We first validate our method, called CURVEBALL, on small problems with known solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. We also show our optimiser's generality by testing on a large set of randomly-generated architectures.

This paper proposes an approximate second-order method with low computational cost. A common pitfall of second-order methods is the computation (and perhaps inversion) of the Hessian matrix. While this can be avoided by instead relying on Hessian-vector products as done in CG, it typically still requires several iterations. Instead, the authors suggest a simpler approach that relies on one single gradient step and a warm start strategy. The authors points out that the resulting algorithm resembles a momentum method. They also provide some simple convergence proofs on quadratics and benchmark their method to train deep neural networks.
While I find the research direction interesting, the execution is rather clumsy and many details are not sufficiently motivated. Finally, there is a lot of relevant work in the optimization community that is not discussed in this paper, see detailed comments and references below.
1) Method
The derivation of the method is very much driven on a set of heuristics without theoretical guarantees. In order to derive the update of the proposed method, the authors rely on three heuristics:
a) The first is to reuse the previous search direction z as a warm-start. The authors argue that this might be beneficial if If z does not change abruptly. In the early phase, the gradient norm is likely to be large and thus z will change significantly. One might also encounter regions of high curvature where the direction of z might change quickly from one iteration to the next.
The "warm start" at s_{t-1} is also what yields the momentum term, what interpretation can you give to this choice?
b) The second step interleaves the updates of z and w instead of first finding the optimum z. This amounts to just running one iteration of CG but it is rather unclear why one iteration is an appropriate number. It seems one could instead some adaptive strategy where CG with a fixed accuracy. One could potentially see if allowing larger errors at the beginning of the optimization process might still allow for the method to converge. This is for instance commonly done with the batch-size of first-order method. Gradually increasing the batch-size and therefore reducing the error as one gets close to the optimum can still yield to a converging algorithm, see e.g.
Friedlander, M. P., & Schmidt, M. (2012). Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3), A1380-A1405.
c) The third step consists in replacing CG with gradient descent.
"If CG takes N steps on average, then Algorithm 2 will be slower than SGD by a factor of at least N, which can easily be an order of magnitude".
First, the number of outer iterations may be a lot less for the Hessian-free method than for SGD so this does not seem to be a valid argument. Please comment.
Second, I would like to see a discussion of the convergence rate of solving (12) inexactly with krylov subspace methods. Note that Lanczos yields an accelerated rate while GD does not. So the motivation for switching to GD should be made clearer.
d) The fourth step introduces a factor rho that decays z at each step. I’m not really sure this makes sense even heuristically. The full update of the algorithm developed by the author is:
w_{t+1} = w_t - beta nabla f + (rho I - beta H) (w_t - w_{t-1}).
The momentum term therefore gets weighted by (rho I - beta H). What is the meaning of this term? The -beta H term weights the momentum according to the curvature of the objective function. Given the lack of theoretical support for this idea, I would at least expect a practical reason back up by some empirical evidence that this is a sensible thing to do.
This is especially important given that you claim to decay rho therefore giving more importance to the curvature term.
Finally, why would this be better than simply using CG on a trust-region model? (Recall that Lanczos yields an accelerated linear rate while GD does not).
2) Convergence analysis
a) The analysis is only performed on a quadratic while the author clearly target non-convex functions, this should be made clear in the main text. Also see references below (comment #3) regarding a possible extension to non-convex functions.
b) The authors should check the range of allowed values for alpha and beta. It appears the rate would scale with the square root of the condition number, please confirm, this is an important detail. I also think that the constant is not as good as Heavy-ball on a quadratic (see e.g. http://pages.cs.wisc.edu/~brecht/cs726docs/HeavyBallLinear.pdf), please comment.
c) Sub-sampling of the Hessian and gradients is not discussed at all (but used in the experiments). Please add a discussion and consider extending the proof (again, see references given below).
3) Convergence Heavy-ball
The authors emphasize the similarity of their approach to Heavy-ball. They cite the results of Loizou & Richtarik 2017. Note that they are earlier results for quadratic functions such as
Lessard, L., Recht, B., & Packard, A. (2016). Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1), 57-95.
Flammarion, N., & Bach, F. (2015, June). From averaging to acceleration, there is only a step-size. In Conference on Learning Theory (pp. 658-695).
The novelty of the bounds derived in Loizou & Richtarik 2017 is that they apply in stochastic settings.
Finally, there are results for non-convex functions such convergence to a stationary point, see
Zavriev, S. K., & Kostyuk, F. V. (1993). Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling, 4(4), 336-341.
Also on page 2, "Momentum GD ... can be shown to have faster convergence than GD". It should be mentioned that this only hold for (strongly) convex functions!
4) Experiments
a) Consider showing the gradient norms.
b) it looks like the methods have not yet converged in Fig 2 and 3.
c) Second order benchmark:
It would be nice to compare to a method that does not use the GN matrix but the true or subsampled Hessian (like Trust Region/Cubic Regularization) methods given below.
Why is BFGS in Rosenbrock but not in NN plots?
d) "Batch normalization (which is known to improve optimization)"
This statement requires a reference such as
Towards a Theoretical Understanding of Batch Normalization
Kohler et al… - arXiv preprint arXiv:1805.10694, 2018
5) Related Work
The related work should include Cubic Regularization and Trust Region methods since they are among the most prominent second order algorithms. Consider citing Conn et al. 2000 Trust Region,  Nesterov 2006 Cubic regularization, Cartis et al. 2011 ARC.
Regarding sub-sampling: Kohler&Lucchi 2017: Stochastic Cubic Regularization for non-convex optimization and Xu et al.: Newton-type methods for non-convex optimization under inexact hessian information.
6) More comments
Page 2
Polyak 1964 should be cited  where momentum is discussed.
"Perhaps the simplest algorithm to optimize Eq. 1 is Gradient Descent". This is technically not correct since GD is not a global optimization algorithm. Maybe mention that you try to find a stationary point
rho (Eq. 2) and lambda (Eq. 4) are not defined
Page 4:
Algorithm 1 and 2 and related equations in the main text: it should be H_hat instead of H.
Background
“Momemtum GD exhibits somewhat better resistance to poor scaling of the objective function”
To be precise the improvement is quadratic for convex functions. Note that Goh might not be the best reference to cite as the article focuses on quadratic function. Consider citing the lecture notes from Nesterov.
Section 2.2
This section is perhaps a bit confusing at first as the authors discuss the general case of a multivalue loss function. Consider moving your last comment to the beginning of the section.
Section 2.3
As a side remark, the work of Dauphin does not rely on the Gauss-Newton approximation but a different PSD matrix, this is probably worth mentioning.
Minor comment: The title is rather bold and not necessarily precise since the stepsize of curveball is not particularly small e.g. in Fig 1.


Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving stochastic gradient methods named predictive local smoothness (PLS). First, we create a convergence condition to build a learning rate which varies adaptively with local smoothness. Second, the local smoothness can be predicted by the latest gradients. Third, we use the adaptive learning rate to update the stochastic gradients for exploring linear convergence rates. By applying the PLS method, we implement new variants of three popular algorithms: PLS-stochastic gradient descent (PLS-SGD), PLS-accelerated SGD (PLS-AccSGD), and PLS-AMSGrad. Moreover, we provide much simpler proofs to ensure their linear convergence. Empirical results show that the variants have better performance gains than the popular algorithms, such as, faster convergence and alleviating explosion and vanish of gradients.

# Unrealistic assumptions and trivial theory
This papers proposes a method to adjust the learning rate of stochastic gradient methods. The problem is of great importance but the theoretical results and presentation contain many issues that make the paper unfit for publication.
The main issue that I see is that the assumption made are unrealistic and make the theory trivial. First, for gradient descent, the authors assume that the gradient is of the form L(x_t) (x_t - x*). Under this assumption, gradient descent converges on a single step with step size 1 / L(x_t). In the stochastic setting, they assume that *each* stochastic gradient is of the form L_i(x_t) (x_t - x*), Eq. (11). Again, SGD in this scenario converges in a single iteration with step size 1 / L_i(x_t).
No wonder in this scenario the authors are able to obtain linear convergence of SGD to arbitrary precision (which is known to be impossible even for quadratics).
# Other Issues
* Motivation of Eq. (9) is not discussed in sufficient detail. It is unclear to me how to obtain (9) from (7) as the authors mention. Regarding notation, L(x_t) is a scalar, hence (9) could be written more simply as \nabla f(x_t) = L(x_t) (x_t - x*). Why the need for the Kronecker product?
* The authors should clearly state what are the assumptions in the theorem statement. For theorem 1 these are not clearly stated, and phrases like "Theorem 1 provides a simple condition for the linear convergence of SGD" give the wrong impression that the Theorem is widely applicable.
# Minor
  * Belo Eq. (10): "where \epsilon_1 is a parameter to prevent ||x_t - x_{t-1}|| going to zero: . I guess what the authors meant is to prevent *the denominator* going to zero, you do want ||x_t - x_{t-1}|| to go to zero as you approach a stationary point


Large-scale datasets may contain significant proportions of noisy (incorrect) class labels, and it is well-known that modern deep neural networks poorly generalize from such noisy training datasets. In this paper, we propose a novel inference method, Deep Determinantal Generative Classifier (DDGC), which can obtain a more robust decision boundary under any softmax neural classifier pre-trained on noisy datasets. Our main idea is inducing a generative classifier on top of hidden feature spaces of the discriminative deep model. By estimating the parameters of generative classifier using the minimum covariance determinant estimator, we significantly improve the classification accuracy, with neither re-training of the deep model nor changing its architectures. In particular, we show that DDGC not only generalizes well from noisy labels, but also is robust against adversarial perturbations due to its large margin property. Finally, we propose the ensemble version of DDGC to improve its performance, by investigating the layer-wise characteristics of generative classifier. Our extensive experimental results demonstrate the superiority of DDGC given different learning models optimized by various training techniques to handle noisy labels or adversarial samples. For instance, on CIFAR-10 dataset containing 45% noisy training labels, we improve the test accuracy of a deep model optimized by the state-of-the-art noise-handling training method from 33.34% to 43.02%.

The paper proposes a new method for robustifying a pre-trained model improving its decision boundaries. The goal is to defend the model from mistakes in training labels and to be more robust to adversarial examples at test time. The main idea is to train a LDA on top of the last-layer, or many layers in its ensemble version, making use of a small set of clean labels after training the main model. Additionally, robustness to outliers is achieved by the minimum covariance determinant estimator for the LDA covariance matrix.
While I find this idea interesting and of potential practical use, I have concerns about novelty and the experimental results and overall I recommend rejection.
== Method
At a high level, the idea of imposing a mixture of gaussian structure in the feature space of a deep neural network classifier is not new. See for example [A, B]. In particular, [B] performs experiments on adversarial examples. Moreover, in spite of the authors writing that their goal is “completely different” from [Lee at al 18a, Ma et al 18a], I found the two cited papers having a similar intent and approach to the problem, but a comparison is completely missing. Without a proper comparison (formal and experimental) with these lines of work, the paper is incomplete.
Theorem 1 well supports the proposed method and it is well explained. I did not check the proofs in appendix.
Regarding the presentation, I found odd having some experimental results (page 5) before the Section on experience even have started.
== Experiments
The authors did not comment on the computational overhead of training their LDA. But I assume it is very cheap compared to training e.g. the ResNet, correct?
I also did not find an explanation of which version backward/forward losses [Patrini et al. 17] is used in the experiments: are the noise transition matrices estimated on the data or assumed to be known (for fair comparison, I would do the former).
I disagree on the importance of the numbers reported on the abstract: DenseNet on Cifar10 with 60% goes from 53.34 to 74.72. This is the improvement with the weakest possible baseline, i.e. no method to defend for noise! Looking at Table 3, which is on ResNets, I will make this point clear. Noise 60% on CIFAR10, DDGC improves 60.05-> 71.38, while (hard) bootstrap and forward do better. Even more, it seems that forward does always better than DDGC with noise 60% on every dataset. Therefore, I don’t find interesting to report how DDGC improve upon “no baseline”, because known methods do even better. Yet, it is interesting --- and I find this to be a contribution of the paper --- that DDGC can be used in combination with prior work to boost performance even further.
A missing empirical analysis is on class-conditional noise (see for example Patrini et al. 17 for a definition). An additional column on the table showing that the algorithm can also work in this case would improve the confidence that the proposed method is useful in practice. Uniform noise is the least realistic assumption for label noise.
Regarding the experiments on adversarial examples, I am not convinced of their relevance at all. There are now dozens of defence methods that work (partially) for improving robustness. I don’t think it is of any practical use to show that a new algorithm (such at DDGD) provide some defence compared to no defence. A proper baseline should have been compared.
One more unclear but important point: is Table 3 obtained by white-box attacks on the Resnet/Denset but oblivious of the MCD? Is so, I don’t think such an experiment tells the whole story: as the the MCD would arguably also be deployed for classification, the attacker would also target it.
Additionally, the authors state “we remark that accessing the parameters of the generative classifiers […] is not a mild assumption since the information about training data is required to compute them”. I don’t follow this argument: this is just part of the classifier. White box attacks are by definition performed with the knowledge of the model, what is the difference here?
Table 8 rises some concerns. I appreciate the idea of testing full white-box adversarial attacks here. But I don’t understand how it is possible that DDGC is more robust, with higher adversarial test accuracy, than in Table 3.
[A] Wen, Yandong, et al. "A discriminative feature learning approach for deep face recognition." European Conference on Computer Vision. Springer, Cham, 2016.
[B] Wan, Weitao, et al. "Rethinking feature distribution for loss functions in image classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.


Adversarial examples are modified samples that preserve original image structures but deviate classifiers. Researchers have put efforts into developing methods for generating adversarial examples and finding out origins. Past research put much attention on decision boundary changes caused by these methods. This paper, in contrast, discusses the origin of adversarial examples from a more underlying knowledge representation point of view. Human beings can learn and classify prototypes as well as transformations of objects. While neural networks store learned knowledge in a more hybrid way of combining all prototypes and transformations as a whole distribution. Hybrid storage may lead to lower distances between different classes so that small modifications can mislead the classifier. A onestep distribution imitation method is designed to imitate distribution of the nearest different class neighbor. Experiments show that simply by imitating distributions from a training set without any knowledge of the classifier can still lead to obvious impacts on classification results from deep networks. It also implies that adversarial examples can be in more forms than small perturbations. Potential ways of alleviating adversarial examples are discussed from the representation point of view. The first path is to change the encoding of data sent to the training step. Training data that are more prototypical can help seize more robust and accurate structural knowledge. The second path requires constructing learning frameworks with improved representations.

Strength:
The proposed approach is architecture-independent: the attack is constructed only from the dataset.
Weaknesses:
Paper is not sufficiently well written for a venue like ICLR.
Attack has very low success rate.
To the exception of Figures 4 and 5, many experiments are conducted on MNIST.
Feedback:
Experimental results show that the attack is able to degrade a classifier’s performance by inserting perturbations that are computed on the data only. However, there are no baselines included to compare adversarial evasion rates achieved here to prior efforts. This makes it difficult to justify the fairly low success rate. In your rebuttal, could you clarify why baselines were not used to offer comparison points in the experiments?
Furthermore, strong conclusions are made from the results despite the lack of supporting evidence. For instance, on P10, the attack is said to “also explains why adversarial examples can be universal.”. However, not only does the attack achieve less success than universal adversarial examples would (so it cannot explain all of them) but also does it not share any characteristics with the way universal adversarial examples are crafted. Drawing such a strong conclusion thus most likely needs a lot more supporting evidence.
Several directions would improve the content of the paper:
* Complete existing experimental results by being more systematic. For instance, in Section 3.1, measurements are only performed on one pair of MNIST images. Without studying a significant portion of the test set of two datasets, it is very difficult to draw any conclusions from the limited evidence.
* Perform a human study to have all perturbed images labeled again. Indeed, because of the ways images are perturbed here, it is unclear how much perturbation can be added without changing the label of the resulting input.
* Study how existing adversarial example techniques modify internal representations. This would help support conclusions made (e.g., about universal perturbations---see above).
* Rewrite the related work section to scope it better: for instance, Sabour et al. in Adversarial Manipulation of Deep Representations and Wicker et al. in Feature-Guided Black-Box Safety Testing of Deep Neural Networks explore adversaries operating in the feature space. This will also help build better baselines for the evaluation.
Additional details:
TLDR: typo in the first word
P1: The following definition of adversarial examples is a bit restrictive, because they are not necessarily limited to vision applications (e.g., they could be found for text or speech as well). “Adversarial examples are modified samples that preserve original image structures”
P1: The following statement is a bit vague (what is obvious impact referring to?): “Experiments show that simply by imitating distributions from a training set without any knowledge of the classifier can still lead to obvious impacts on classification results from deep networks.”
P1: References do not typeset properly (the parentheses are missing: perhaps, the \citep{} command was not used?)
P2: What is the motivation for including references to prior work in the realm of image segmentation and more generally-speaking multi-camera settings in the related work section?
P2: Typo in “ linear vibration”
P2: It remains difficult to make a conclusion about humans being robust to the perturbations introduced by adversarial examples. For instance, Elsayed et al. at NIPS 2018 found that time-constrained humans were also misled by adversarial examples crafted to evade ML classifiers: see Adversarial Examples that Fool both Computer Vision and Time-Limited Humans.
P2: Prior work suggests that the following conclusion is not entirely true: “Most of these kinds of examples are generated by carefully designed algorithms and procedures. This complexity to some extent shows that adversarial examples may only occupy a small percentage for total image space we can imagine with pixel representations.” For instance, Tramer et al. in ICLR 2018 found that adversarial subspaces were often large: “Ensemble Adversarial Training: Attacks and Defenses”.
P2: Others have looked at internal representations of adversarial examples so the following statement would be best softened: “To the best of our knowledge, this paper should be the first one that discusses adversarial examples from the internal knowledge representation point of view.”. See for instance, Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning by Papernot and McDaniel.
P3: Could you add pointers to support the description of human abstraction and sparsity? It reads a bit speculative as is, and adding some pointers would help relate the arguments made to relevant pointers for readers that are less familiar with this topic.
P3: What is the motivation for including the discussion of computations performed by a neural network layer-by-layer in Section 2?
P4: Given that saliency maps can be manipulated easily and are only applicable locally, it appears that Figure 1 is too limited to serve as sufficient evidence for the following conclusion: “This, in some way, proves the point that the knowledge storage and representation of current neural networks are not exactly sparse prototype based.”
P5: The error rate reported on MNIST is quite low (45%). Even using the Fast Gradient Method, one should be able to have the error rate be as high as 90% on a standard CNN.
P7: Would you be able to provide references to backup the following statement? “This is a network structure that is very common.”
P10: How does the discussion in Section 4.2 relate to the attack described in the submission?


We flip the usual approach to study invariance and robustness of neural networks by considering the non-uniqueness and instability of the inverse mapping. We provide theoretical and numerical results on the inverse of ReLU-layers. First, we derive a necessary and sufficient condition on the existence of invariance that provides a geometric interpretation. Next, we move to robustness via analyzing local effects on the inverse. To conclude, we show how this reverse point of view not only provides insights into key effects, but also enables to view adversarial examples from different perspectives.

Review
This paper discusses invariances in ReLU networks. The discussion is anchored around the observation that while the spectral norm of neural networks layers (their product bounds the Lipschitz constant) has been investigated as a measure of robustness of nets, the smallest singular values are also of interest as these indicate directions of invariance.
The paper consists mostly of a theoretical analysis with little empirical support, focusing on a property of matrices called omnidirectIonality. The definition given seems weird — an A \in R^{m \times n} is omnidirectional if there exists a unique x \in R^n such that Ax \leq 0.
If there is a *unique* x then that x must be 0. Else if there were a nonzero x for which Ax \leq 0, then A(cx) also \leq 0 for any positive scalar 0 and thus x is not unique. Moreover if x must be equal to 0 Ax \leq 0 and at that point Ax = 0, then that means there exists no x for which Ax < 0, so why not just say this outright? Perhaps a cleaner definition would just be “A is full rank and there does not exist any X such that Ax < 0? Also perhaps better to use the curly sign for vector inequality.
Overall the paper, while interesting is unacceptably messy.
The first two pages have no paragraph breaks!!! This means either that the author are separating paragraphs with \\ \noindent or that they have modified the style file to remove paragraph breaks to save space. Either choice is unreadable and unacceptable. The paper is also littered with typos and vague statements (many enumerated below under *small issues*). In this case, they add up to make a big issue.
The notation at the top of page 4 — see (1) and (2) — comes out of nowhere and requires explanation. |_{y>0} x + b |_{y>0}  <— what is the purpose of the subscripts here? Why is this notation never introduced?
Ultimately this paper focuses on the question on whether the pre-image of a ReLU layer can be concluded (based on the post-image) to be a singleton,  a compact polytope, or if it has infinite volume. The paper offers some analysis, suggesting when each of the conditions occurs, upper bounds the smallest singular value of D A (where the example dependent diagonal matrix D incorporates the ReLU activation (shouldn’t this be more clearly introduced and notated?).
Ultimately this paper is interesting but falls well below the standards of exposition that I expect from a theory paper and doesn’t go very far at connecting the analysis back to the claimed motivation of investigating practical invariances. If the authors significantly improve the quality of the draft, I’ll be happy to revisit it and re-evaluate my score.
Small issues
The following is a *very* incomplete list of small bugs found in the paper:
“From a high-level perspective both of these approaches” --> missing comma after “perspective”
"as well as the gradient correspond to the highest
possible responds for a given perturbation" --> incomprehensible "corresponding?" "possible responds?" do you mean "response", and if so what is the precise technical meaning here?
"analyzing the lowest possible response" what does "response' mean here?
"We provide upper bounds on the smallest singular value" -- the singular value of what? This hasn't been stated yet.
"reverse view on adversarial examples" --- what this means isn't clear from the preceding text.
"we aim to theoretically derive means to uncover mechanisms of rectifier networks without assumptions on the weights" -- what does "mechanisms" mean here?
Notation section -- need a sentence here at the beginning, can't just have a section heading followed by bullets.
"realated"


In recent years softmax, together with its fast approximations, has become the defacto loss function for deep neural networks with multiclass predictions. However, softmax is used in many problems that do not fully fit the multiclass framework and where the softmax assumption of mutually exclusive outcomes can lead to biased results. This is often the case for applications such as language modeling, nextevent prediction, and matrix factorization, where many of the potential outcomes are not mutually exclusive, but are more likely to be conditionally independent given the state. To this end, for the set of problems with positive and unlabeled data, we propose Partially Mutual Exclusive Softmax (PMES), a relaxation of the original softmax formulation, where, given the observed state, each of the outcomes are conditionally independent but share a common set of negatives. Since we operate in a regime where explicit negatives are missing, we create a cooperatively-trained model of negatives, and derive a new negative sampling and weighting scheme which we call Cooperative Importance Sampling (CIS). We show empirically the advantages of our newly introduced negative sampling scheme by plugging it into the Word2Vec algorithm and evaluating it extensively against other negative sampling schemes on both language modeling and matrix factorization tasks, where we show large lifts in performance.

This paper presents Partially Mutual Exclusive Softmax (PMES), a relaxation of the full softmax that is commonly used for multi-class data. PMES is designed for positive-unlabeled learning, e.g., language modeling, recommender systems (implicit feedback), where we only get to observe positive examples. The basic idea behind PMES is that rather than considering all the non-positive examples as negative in a regular full softmax, it instead only considers a "relevant" subset of negatives. Since we actually don't know which of the negatives are more relevant, the authors propose to incorporate a discriminator which attempts to rate each negative by how hard it is to distinguish it from positives, and weight them by the predicted score from the discriminator when computing the normalizing constant for the multinomial probability. The motivation is that the negatives with higher weights are the ones that are closer to the decision boundary, hence will provide more informative gradient comparing to the negatives that are further away from the decision boundary. On both real-world and synthetic data, the authors demonstrate the PMES improves over some other negative sampling strategies used in the literature.
Overall the idea of PMES is interesting and the solution makes intuitive sense. However, the writing of the paper at the current stage is rather subpar, to the extend that makes me decide to vote for rejection. In details:

1. The motivation of PMES from the perspective of mutual exclusivity is quite confusing. First of all, it is not clear to me what exactly the authors mean by claiming categorical distribution assumes mutual exclusivity -- does it mean given a context word, only one word can be generated from it? Some further explanation will definitely help. Further more, no matter what mutual exclusive means in this context, I can hardly see that PSME being fundamentally different given it's still a categorical distribution (albeit over a subset).
The way I see PMES from a positive-unlabeled perspective seems much more straight-forward -- in PU learning, how to interpret negatives is the most crucial part. Naively doing full softmax or uniform negative sampling carry the assumption that all the negatives are equal, which is clearly not the right assumption for language modeling and recommender systems. Hence we want to weight negatives differently (see Liang et al., Modeling user exposure in recommendation, 2016 for a similar treatment for RecSys setting). From an optimization perspective, it is observed that for negative sampling, the gradient can easily saturate if the negative examples are not "hard" enough. Hence it is important to sample negatives more selectively -- which is equivalent to weighting them differently based on their relevance. A similar approach has also been explored in RecSys setting (Rendle, Improving pairwise learning for item recommendation from implicit feedback, 2014). Both of these perspectives seem to offer more clear motivation than the mutual exclusivity argument currently presented in the paper.
That being said, I like the idea of incorporating a discriminator, which is something not explored in the previous work.
2. The rigor in the writing can be improved. In details:
* Section 3.3, "Multivariate Bernoulli" -> what is presented here is clearly not multivariate Bernoulli
* Section 3.3, the conditional independence argument in "Intuition" section seems no difference from what word2vec (or similar models) assumes. The entire "Intuition" section is quite hand-wavy.
* Section 3.3, Equation 4, 5, it seems that i and j are referred both as binary Bernoulli random variables and categorical random variables. The notation here about i and j can be made more clear. Overall, there are ambiguously defined notations throughout the paper.
* Section 4, the details about the baselines are quite lacking. It is worth including a short description for each one of them. For example, is PopS based on popularity or some attenuated version of it? As demonstrated from word2vec, a attenuated version of the unigram (raised to certain power < 1) works better than both uniform random, as well as plain unigram. Hence, it is important to make the description clear. In addition, the details about matrix factorization experiments are also rather lacking.
3. On a related note, the connection to GAN seems forced. As mentioned in the paper, the discriminator here is more on the "cooperative" rather than the "adversarial" side.
Minor:
1. There are some minor grammatical errors throughout.
2. Below equation 3, "\sigma is the sigmoid function" seems out of the context.
3. Matt Mohaney -> Matt Mahoney


Spatio-Temporal processes bear a central importance in many applied scientific fields. Generally, differential equations are used to describe these processes. In this work, we address the problem of learning spatio-temporal dynamics with neural networks when only partial information on the system's state is available. Taking inspiration from the dynamical system approach, we outline a general framework in which complex dynamics generated by families of differential equations can be learned in a principled way. Two models are derived from this framework. We demonstrate how they can be applied in practice by considering the problem of forecasting fluid flows. We show how the underlying equations fit into our formalism and evaluate our method by comparing with standard baselines.

I very much like the aim of this work. This is a problem of interest to a wide community, which as far as I'm aware hasn't yet had much focus from the deep learning community. However, perhaps in part because of this, the paper reads as naive in places. Pages 1-4 are all background saying nothing new, but ignoring the effort made on this problem by other communities. There has been some work done o this problem within statistics, and within the Gaussian process community, to which no reference is made at all by the paper.
There are two novelties as far as I can see (these may or may not be novel - but they were novel to me). The first is use of NNs to model the system. The second is the multiple state restimation (MRSE) on page 5. I struggled to get a feeling about how successful these two aspects of the work are. The results section is difficult to follow, and doesn't compare the method to existing methods and so there is no baseline to say that this is successful or not. Thus I find it hard to judge the execution of the idea. What I really want to know reading a paper like this is should I use this approach? Because there is no comparison to existing methods, it leaves me unsure.
Other comments:
- Is the title correct? I don't see how these are PDE guided NNs? You've used data from a PDE to train the network and as a test problem. A PDE guided NN would, for me, know something about the dynamics (compare with recently work in the GP community where kernels are derived that lead to GPs that analytically obey simple PDEs).
- There is an obvious link to work in the uncertainty quantification community, particularly around the use of multi-fidelity/ multi-level simulation. This paper is likely to be of interest to them and the link could be more explicit.
- Page 3, after eq 2 - there is notation used here that is undefined Y_{t-k}^t
- The simplifying assumption on page 3 is very strong and unlikely to hold for many systems. But it isn't clear to me whether this is necessary or not? Presumably if it doesn't hold then we may still get an approximation that could be useful, but it is just that we lose any guarantee the method will work.
- I thought the MSRE idea was interesting. It wasn't very well explained or motivated, and it was unclear to me whether it works well or not from the results, or whether it is novel to this paper or not. But I'd like to have read more about it.
- Is the trick in Section 8.2 original to this paper? If so, it seems a nice idea (I've not checked the detail).
- Most of section 8.1 strikes me as unnecessary.
- There are quite a few typos. In particular, words such as Markovian, Newtonian should be capitalised.


There is a rising interest in studying the robustness of deep neural network classifiers against adversaries, with both advanced attack and defence techniques being actively developed. However, most recent work focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. In this paper, we propose and investigate the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs with low likelihood under the generative model. Experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and that the proposed detection methods are effective against many recently proposed attacks.

In this work the authors propose and analyse generative models as defences against adversarial examples. In addition, three detection methods are introduced and an extension to deep features is suggested.
My main concerns are as follows (details below):
* Important prior work is not mentioned.
* Evaluation with direct attacks is only based on (very few) gradient-based techniques, many results are not reliable.
* There are signs of gradient masking (the common problem of robustness evaluation, in particular of only gradient-based techniques are used).
* The way detection rates are taken into account in the perfect knowledge scenario is confusing.
### Style
I like the idea of testing many different factorisation structures. However, that comes with the drawback that one needs to constantly check back what the abbreviations mean. Together with the three detection methods, the manuscript is quite confusing at times and should definitely be streamlined. One suggestion: remove the detection methods: I did not find any real conclusion about them but they are definitely side-tracking users away from the main results.
### Prior work
There is at least one closely related prior work not mentioned here: the analysis by synthesis model [1]. This model uses a variational auto encoder to learn class conditional distributions and shows high robustness on MNIST. Please make clear what your contribution is over this paper (other than testing several other factorisations).
### Evaluation problems
The robustness of models should be evaluated on different direct attacks ranging from gradient-based to score-based (e.g. NES [2]) to decision-based attacks [3]. Please take a look at [1] to see how a very extensive evaluation might look like. The results can be astonishingly different for different attacks, and so basing conclusion on only one or two attacks is dangerous (in particular if you only use gradient-based ones). One can also see that in your results, just check the variations you get between MIM and PGD. Also, rather then discussing (and showing in detail) results for individual attacks, the minimum adversarial distance for a given sample that can be found by any attack is much more comparable between models (which can also streamline the manuscript).
One can see signs of gradient masking in your results. For example, in Figure 3 the MIM attacks levels out at 20% for the DBX model. That can happen for iterative attacks if the gradient is masked. Similarly, in Figure 5 DBX-ZK (zero knowledge) is better in both accuracy and detection rate than DBX-PKK (which takes the KL-detection method into account and should thus either be better in accuracy or detection rate).
More generally, the perfect knowledge case, in which the attacker knows about the detector, should only count samples as adversarials which evade the detector and change the model decision. Thus, the detection rate should be zero. Otherwise I have no idea what trade-off between accuracy and detection rate you are actually targeting and how to compare the results.
Also, some intermediate results are conflicting with each other. E.g. in 4.1 you state “the usage of bottleneck is beneficial for better robustness”, but for L2 this is not true.
Also, I am not sure how conclusive the grey-box and black-box scenarios really are: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models.
### Minor
 * In 4.1 you say “as they fail to find near manifold adversarials”, but I don’t see how there can be L-infty adversarials on MNIST that are on-manifold (remember, MNIST pixel values are basically binary). Plus, in the zero-knowledge scenario there is nothing that enforces staying on this manifold.
 * Result presentation (Figure 3/5 & Table 1) is very different for different attack scenarios, which makes them hard to compare. Please unify.
 * Is the L2 distance you report in Table 1 the mean (or median) distance to adversarial examples. If so, GBZ (for which you state that C&W “failed on attacking” has actually a smaller mean adversarial distance than some other models (for which C&W is actually quite successful).
 * Grey-box scenario doesn’t make a lot of sense: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models. A similar confounder makes the black-box results difficult to interpret.
* Also, taking into account that the paper is two pages longer and thus calls for higher standards
Taken together, I find the general direction of the paper very interesting and I’d definitely encourage the authors to go further. At the current stage, however, I feel that (1) contributions are not sufficiently delineated to prior work, (2) the evaluation is not convincingly supporting the claims and that  (3) the manuscript needs to be streamlined (both in terms of text and figures).
[1] Schott et al. (2018) “Towards the first adversarially robust neural network model on MNIST” (https://arxiv.org/abs/1805.09190)
[2] Ilyas et al. (2018) “Black-box Adversarial Attacks with Limited Queries and Information” ( [https://arxiv.org/abs/1804.08598)](https://arxiv.org/abs/1804.08598))
[3] Brendel et al. (2018) “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models” (https://arxiv.org/abs/1712.04248)


Contrary to most reinforcement learning studies, which emphasize on training a deep neural network to approximate its output layer to certain strategies, this paper proposes a reversed method for reinforcement learning. We call this "Deep Deducing". In short, after adequately training a deep neural network according to a strategy-environment-to-payoff table, then we initialize randomized strategy input and propagate the error between the actual output and the desired output back to the initially-randomized strategy input in the "input layer" of the trained deep neural network gradually to perform a task similar to "human deduction". And we view the final strategy input in the "input layer" as the fittest strategy for a neural network when confronting the observed environment input from the world outside.

The paper proposes a method of searching for a Nash equilibrium strategy in games where the strategy-to-payoff mapping is defined by a neural network. The idea is to perform gradient optimization of the payoff w.r.t. the strategy. Preliminary results on tic-tac-toe and variations of the prisoner’s dilemma task are presented. The paper has an interesting idea at the core. However, it is poorly written, does not properly discuss the related works and does not present a convincing method or experimental results.
Pros:
* The paper considers an interesting question of exploring the applications of neural networks to the game theory problems.
* The idea of the paper is reasonable. It makes sense to me to perform gradient-based search over the strategies (assuming that the payoff is differentiable).
Cons:
* Writing
- The paper is over the mandatory length limit of 10 pages.
- The paper makes a grandiose claim: “this paper provides a revolutionary way for reinforcement learning and a possible road toward general A.I.” However, there are arguably no revolutionary ideas, and certainly no reinforcement learning experiments!
- Despite the claim, the novelty of the paper is limited. There is no discussion of the related work: optimization of the neural networks w.r.t. the inputs [1]; related RL ideas such as model-based learning [2,3] and Monte-Carlo Tree Search [4].
- The problem being solved is never formally stated. As far as I understand, Nash equilibrium is usually defined (1) in mixed strategies, while the paper seems to consider pure strategies; (2) in the scenario where every player attempts to maximize their payoff, while in the paper the players attempt to achieve some pre-fixed value of the payoff.
- The flow of the paper is generally poor. Instead of presenting a general solution and then showcasing its applications, the paper iterates on similar ideas multiple times. For example, all four algorithms are just gradient-based optimization of either the weights or the inputs to a model.
- The paper provides extremely misleading analogies and explanations. I am quite sure that a mosquito brain is not a one hidden layer fully-connected neural network! Also, the example of avoiding a moving hand is poor: since the outcome is life or death, the learning should happen via evolution, not during the lifetime of a single insect. The claim that the neural networks with sigmoid activation functions are less prone to local optima is questionable as well.
* Method and experiments
- The proposed method is essentially a greedy gradient-based planning procedure. For this to work, we need to have a very good environment model. This is a strong assumption that is not discussed.
- The experiments are performed on very simple synthetic problems: matrix games and tic-tac-toe. They do not suggest that the method is general and can work on harder problems, say, Sokoban [2].
- The experiments do not present any baselines, so it is unclear how well the method performs compared to the alternatives. One obvious candidate is gradient-free optimization, such as Nelder-Mead, and gradient descent with momentum, which can be less prone to local optima.
[1] Brandon Amos, Lei Xu, J. Zico Kolter “Input Convex Neural Networks”, ICML 2017
[2] Racanière et al. “Imagination-Augmented Agents for Deep Reinforcement Learning”, NIPS 2017
[3] David Ha, Jürgen Schmidhuber “Recurrent World Models Facilitate Policy Evolution”, NIPS 2018
[4] Thomas Anthony, Zheng Tian, David Barber “Thinking Fast and Slow with Deep Learning and Tree Search”, NIPS 2017


Extensive work on compressed sensing has yielded a rich collection of sparse recovery algorithms, each making different tradeoffs between recovery condition and computational efficiency. In this paper, we propose a unified framework for accelerating various existing sparse recovery algorithms without sacrificing recovery guarantees by exploiting structure in the measurement matrix. Unlike fast algorithms that are specific to particular choices of measurement matrices where the columns are Fourier or wavelet filters for example, the proposed approach works on a broad range of measurement matrices that satisfy a particular property. We precisely characterize this property, which quantifies how easy it is to accelerate sparse recovery for the measurement matrix in question. We also derive the time complexity of the accelerated algorithm, which is sublinear in the signal length in each iteration. Moreover, we present experimental results on real world data that demonstrate the effectiveness of the proposed approach in practice.

Clarity: Paper is generally well written; however, certain theoretical statements (e.g. Theorem 1) are not very precise.
Originality: Contribution seems to be incremental; the proposed method seems to be a straightforward concatenation of well-known existing results in sparse recovery and nearest-neighbor search.
Significance: Unclear whether the techniques significantly advance the state of the art.
Quality: Overall, I think this is a promising direction but the idea might not have fully fleshed out.
----
Summary:
the paper proposes a scheme to accelerate popular sparse recovery methods that rely on hard thresholding (specifically, CoSaMP and IHT, but presumably other similar methods can also be used here). The key idea is that if the measurement matrix is normalized, then the k-sparse thresholding of the gradient update can be viewed as solving a k-nearest neighbor problem. Therefore, one can presumably use fast k-NN methods instead of exact NN methods. Specifically the authors propose to use the prioritized DCI method of Li and Malik.
Pros:
reasonable idea to use fast (sublinear) NN techniques in the k-sparse projection step.
Cons:
* It appears that the running time improvement over the baseline IHT (which has Otilde(mn) complexity) heavily depends on the intrinsic dimensionality of A. However, the authors do not characterize this.
* The authors neglect to mention in the paper that prioritized DCI has a pre-processing time of O(mn), so the final algorithm isn't really asymptotically faster.
* I cannot parse Theorem 1 (especially, the second sentence). Is epsilon the failure probability of DCI?
* Experimental results are far too synthetic. In real-life problems k itself is big, so there may be other bottlenecks (least squares, gradient updates, etc) and not necessarily the hard thresholding step.


Deep neural networks (DNNs), although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work has been conducted on DNN model compression or pruning. However, most of the previous work has taken heuristic approaches. This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34× pruning rate for ImageNet data set and 167× pruning rate for MNIST data set, significantly higher than those reached by existing work in the literature. Under the same number of epochs, the proposed method also achieves better convergence and higher compression rates. The codes and pruned DNN models are avilable in the link: bit.ly/2zxdlss.

This paper focus on weight pruning for neural network compression. The proposed method is based on ADMM optimization method for neural network loss with constraint on the l_0 norm of weights, proposed in Zhang et al. 2018b. Two improvements, masked retraining and progressive pruning, are introduced. Masked retraining set the weights to zero at early stages and stop updating those weights. Progressing pruning keeps a buffer of partial pruning results and select the best performed model for further pruning. The proposed method achieves 30x compression rate for AlexNet and VGG for ImageNet.
I have the following concerns about the proposed method.
- It is unclear to me what is the benefit of ADMM for solving the sparse regularized NN optimization problem. Why is it better than projected gradient descent or proximal gradient method used in previous network pruning? I understand the proposed method is based on Zhang et al. 2018b, but a strong argument will support the draft.
- I fail to understand the claim ``at convergence, the pruned DNN model will not be exactly sparse’’ in section 2.3. Z will always be sparse after the projection step in (5). At convergence, the linear constraint should be satisfied, which makes W = Z to be sparse.
- Please describe the the proposed method in detail. The current description is very vague and I do not think it can be reimplemented based on the current draft. In each outer loop of ADMM, (4)(5) and dual update is applied (I consider solving (4) is the inner loop). How is the mask generated and fit into these equations? For progressive pruning, it looks to me there is an outer loop outside the outer loop of ADMM. Please provide details on how many iterations, and how the compression rate is decided for each iteration.
- The hyper-parameters of the proposed method is unclear. It is a bit strange the optimization parameter \rho could control the pruning rate (section 3.1). As described before, I guess the proposed method has three loops. How is the iterations counted, like for Figure 3. Please clarify the experiments are fair comparison, the better results are not because of more weight updates from the three loops.
- It is unclear what is the benefit of masked retraining. It looks to me this kind of greedy approach will harm the performance (I have to guess if a weight is masked to be zero, it will never be updated or recovered). What happens if there are a lot of weights (Z in (5)) are zero at the early stage?
- The progressive pruning looks heuristic and I am not convinced the buffer is necessary. There is always a progressive pruning trace that can directly lead to the results without selecting from candidates. For example, in Figure 2, we can just train model from 15x to 24x to 30x.
- The following works are related.
Li et al. Pruning Filters for Efficient ConvNets. ICLR 2017
Alvarez et al. Learning the number of neurons in deep networks. NIPS 2016
=============== after rebuttal ===================
I appreciate the authors' feedback and slightly raise the score.
Though the compression results look good, I still have some concerns about the method. The motivation of the proposed method is not strong. The proposed mask is greedy and sounds ad-hoc. The proposed progressive pruning looks expensive.
The proposed method looks time consuming. For the experiments, I would love to see the training time comparing with baselines in table 1, not only the ADMM method in table 2. A fair comparison could be wall-clock time, or number of gradient updates for neural networks.


Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models. We propose Mahé, a novel approach to provide Model-agnostic hierarchicaléxplanations of how powerful machine learning models, such as deep neural networks, capture these interactions as either dependent on or free of the context of data instances. Specifically, Mahé provides context-dependent explanations by a novel local interpretation algorithm that effectively captures any-order interactions, and obtains context-free explanations through generalizing contextdependent interactions to explain global behaviors. Experimental results show that Mahé obtains improved local interaction interpretations over state-of-the-art methods and successfully provides explanations of interactions that are contextfree.

Summary
=======
The authors extended the linear local attribution method LIME for interpreting black box models by non-linear functions to more accurately approximate black box models locally and identifying interactions between model input variables using the previously published neural interaction detection (NID) framework. They further propose a method to discern between context-dependent and context-free interactions. I found the paper hard to understand without being familiar with previously published literature on detecting interactions and could not understand their approach to detect context-free interactions as well as some aspects of their evaluation. I have also concerns about the practically of their method due to the high runtime and the notion of locality in the light of high-dimensional inputs. In the following, I will briefly summarizing my major criticism and give further details below.
Summary of major criticism
=====================
1) The paper is hard to understand without being familiar with previously published literature in the field.
The authors do not describe how they define the interaction sets X_I in equation (3).
2) I could not understand their approach for detecting context-free interactions (section 4.2).
3) It is unclear how discrete variables are locally modified. Can the approach be used for a combination of differently distributed variables, e.g. categorical and continuous variables?
4) Evaluation metrics such as MSE and R-precision are not described and I could not understand other important aspects of their evaluation, e.g. superpixels (figure 6), why and how they modified network architectures for evaluating context-free interactions (section 5.3), or how they asked Amazon Medical Turk users.
5) The authors did not compare the runtime of Mathe with LIME, which is presumably high due to the need of fitting multiple non-linear models (equation 3) and sampling the local neighborhood.
6) The authors  did not compare the total number of model parameters of Mathe with LIME, which might also account for the higher accuracy (lower MSE).
7) The authors did not evaluate the accuracy and runtime of Mathe on high-dimensional inputs, e.g. large images.
Details
=====
Abstract
---------------------------
1. The abstract is hard to understand since ‘context-dependent’ and ‘context-free’ are undefined. The authors should also not use ‘dependencies’ as synonym for ‘interactions’.
Introduction
---------------------------
2. The difference between interactions and context-free and context dependent interactions is unclear. Is a variable without interactions context-free, e.g. Buffalo does not interact with water, and a variable with interactions context-dependent? Also, ‘classes of data’ is unclear, which can be misinterpreted as class labels for classification problems. The authors should also clarify what they mean by ‘performance and generality’ in the last section of the introduction.
Section 3.1
---------------
3. The description of the model function f(x) and attribution scores phi(x) is unclear. Since f(x) is undefined when it is first mentioned after equation (1), I suggest to first define f(x) and afterwards define phi(x). The purpose of the attribution score function phi(x) is also unclear without prior knowledge. The authors should more clearly describe that f(x) is the target function (model) of interest, e.g. a classifier, and phi(x) locally approximates f(x) and is interpretable in contrast to f(x).
Section 4.1
---------------
4. The authors should justify why they are sampling x ~ N(x, sigma * I), which assumes that data instances are iid Normal. This is not the case, e.g., if x is categorical or contains a combination of categorical and continuous variables, and variables are correlated. How is sigma chosen? How many samples are used depending on the dimension of x?
5. The authors should briefly describe the basic idea of NID.
6. How are points sampled from the epsilon neighborhood of x? What is a link function?
7. The subsection ‘Hierarchical Interaction Attribution’ is hard to understand without being familiar Tsang et al. The authors should give an example of a hierarchical explanation with different layers.
Section 4.2
---------------
8. How is the ‘local vicinity’ defined? Which distance metric is used? This is in particular problematic if x is high-dimensional due the the curse of dimensionality. How are continuous and categorical variables locally modified? Did the authors meant to use a lowercase ‘k’ in equation (4), i.e. ‘phi_k =’ instead of ‘phi_K’? I find this section hard to understand without being familiar with the cited literature.
Section 5.1
---------------
9. The number of local vicinity samples is unclear. Did the authors use 1k local vicinity samples for synthetic experiments (Table 1) and 5k samples for real-word synthetic experiments?
10. What is the dimensionality (number of words, characters, or pixels) of real world datasets? This is important since it influences the number samples that are required to approximate the vicinity of a particular data point. It is in particular interesting to know how the model accuracy and runtime depends on the dimensionality and the number of local vicinity samples.
11. The authors should define the evaluation metrics (MSE, R-precision) in addition to citing them.
12. How did the authors choose the interaction sets X_I in equation (3) and (4)? How many MLPs (functions g(.)) did the authors fit to learn phi(x)? Is the number of MLPs the same for LIME and Mathe? Otherwise the performance gain of Mathe over LIME can also be attributed the increased number of models and model parameters (ensemble size).
13. What is the average training time of Mathe and baseline models on the different datasets?
Section 5.2.2
-----------------
14. How did the authors choose sigma (0.4, 6, 0.4) for the different datasets?
15. How did Amazon medical turk users evaluate Mathe vs. LIME interactions? Were they given for each sentence the best Mathe and best LIME interaction and asked to decide which one is better? Figure 4 should be more clearly described in the caption. The sentence ‘The result of this experiment is that the majority of preferred explanation …’ is unclear and unjustified by only showing one example in Figure 4.
Section 5.2.3
-----------------
16. The authors should discuss figure 6. The results indicate that neither LIME nor Mathe is able to clearly identify the object of interest, e.g. the water buffalo, and interactions, e.g. between the buffalo and water.
Section 5.3
---------------
17. The sentence ‘... the presence of a French word for “this” or “that”, cet, which …’ is unclear. I suggest to give an example to illustrate which interactions are supposed to be detected. The modification of the Transformer model and the reason why this is necessary is unclear. Overall, I find this evaluation unclear and insufficient since it only applies to a particular interaction.


LEARNING TO SCHEDULE COMMUNICATION IN MULTI-AGENT REINFORCEMENT LEARNING
Many real-world reinforcement learning tasks require multiple agents to make sequential decisions under the agents' interaction, where well-coordinated actions among the agents are crucial to achieve the target goal better at these tasks. One way to accelerate the coordination effect is to enable multiple agents to communicate with each other in a distributed manner and behave as a group. In this paper, we study a practical scenario when (i) the communication bandwidth is limited and (ii) the agents share the communication medium so that only a restricted number of agents are able to simultaneously use the medium, as in the state-of-the-art wireless networking standards. This calls for a certain form of communication scheduling. In that regard, we propose a multi-agent deep reinforcement learning framework, called SchedNet, in which agents learn how to schedule themselves, how to encode the messages, and how to select actions based on received messages. SchedNet is capable of deciding which agents should be entitled to broadcasting their (encoded) messages, by learning the importance of each agent's partially observed information. We evaluate SchedNet against multiple baselines under two different applications, namely, cooperative communication and navigation, and predator-prey. Our experiments show a non-negligible performance gap between SchedNet and other mechanisms such as the ones without communication and with vanilla scheduling methods, e.g., round robin, ranging from 32% to 43%.

# overview
This paper focuses on multi-agent reinforcement learning tasks that require communication between the agents, and further presupposes that the communication protocol is bandwidth constrained and contentious so that a scheduling mechanism is necessary.  To address this they introduce a new learned weighting scheme based scheduler and distributed actor, centralized critic based architecture which is evaluated on a couple of communication driven multi-agent tasks.
The two evaluation tasks had their bandwidth artificially constrained, and SchedNet time to convergence was shown to fall somewhere between having no communication and full communication, and somewhat better than a purely round-robin based scheduling scheme, which doesn't seem particularly informative.  From this it is difficult to assess the significance of the contributions.
# pros
* communication in multi-agent scenarios is an important aspect to consider, and this work shines a spotlight on scenarios in which bandwidth is constrained.
* general presentation fairly clear and easy to read
# cons
* Would have been more impactful to focus experiments on real-world scenarios in which bandwidth is constrained and naturally contentious
# other comments
* pg. 2 related work, suspect you meant to call out Foerster et al 2017b in second reference not Foerster et al 2017a twice.


Computations for the softmax function in neural network models are expensive when the number of output classes is large. This can become a significant issue in both training and inference for such models. In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse Experts, to improve the efficiency for softmax inference. During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts. Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts. During inference, our method quickly locates the most probable expert to compute small-scale softmax. Our method is learning-based and requires no knowledge of the output class partition space a priori. We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

The present paper proposes a fast approximation to the softmax computation when the number of classes is very large. This is typically a bottleneck in deep learning architectures. The approximation is a sparse two-layer mixture of experts.
The paper lacks rigor and the writing is of low quality, both in its clarity and its grammar. See a list of typos below.
An example of lack of mathematical rigor is equation 4 in which the same variable name is used to describe the weights before and after pruning, as if it was computer code instead of an equation. Also pervasive is the use of the asterisk to denote multiplication, again as if it was code and not math.
Algorithm 1 does not include mitosis, which may have an effect on the resulting approximation.
How are the lambda and threshold parameters tuned? The authors mention a validation set, are they just exhaustively explored on a 3D grid on the validation set?
The results only compare with Shim et al. Why only this method? Why would it be expected to be faster than all the other alternatives? Wouldn't similar alternatives like the sparsely gated MoE, D-softmax and adaptive-softmax have chances of being faster?
The column "FLOPS" in the result seems to measure the speedup, whereas the actual FLOPS should be less when the speed increases. Also, a "1x" label seems to be missing in for the full softmax, so that the reference is clearly specified.
All in all, the results show that the proposed method provides a significant speedup with respect to Shim et al., but it lacks comparison with other methods in the literature.
A brief list of typos:
"Sparse Mixture of Sparse of Sparse Experts"
"if we only search right answer"
"it might also like appear"
"which is to design to choose the right"
sparsly
"will only consists partial"
"with γ is a lasso threshold"
"an arbitrarily distance function"
"each 10 sub classes are belonged to one"
"is also needed to tune to achieve"


Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters . It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of trajectories which reflect the differences in various possible environments, and then selecting some subset of these to learn robust policies, such as the ones that result in the worst performance. We propose an active learning based framework, EffAcTS, to selectively choose model parameters for this purpose so as to collect only as much data as necessary to select such a subset. We apply this framework to an existing method, namely EPOpt, and experimentally validate the gains in sample efficiency and the performance of our approach on standard continuous control tasks. We also present a Multi-Task Learning perspective to the problem of Robust Policy Search, and draw connections from our proposed framework to existing work on Multi-Task Learning.

This paper targets at a particular type of robust policy search, where a simulation environment exists with explicit tuning parameters, which is referred to as the model parameters of a Markov decision process. The task of robust policy search is to learn a policy robust to all the parameters of the simulator, so that it can potentially give robust performance in real environment. The previous work handles this problem by sampling many trajectories and only learning from the trajectories, in which the current policy produces the worst performance. This approach effectively focus the policy search on the worst case performance, but is highly inefficient as most of the sampled trajectories are discarded. This paper proposes to improve the sampling efficiency by building a surrogate model predicting the return of the current policy given a MDP parameter. The surrogate model is used to select the MDP parameters leading to the worsts performance, so that the policy search can directly sample and learn from the selected MDP parameters without discarding any trajectories.
This paper tries to tackle the sampling efficiency of RL with building a probabilistic surrogate model. This is a promising direction. The biggest concern is that this paper tackles this problem with a combination with existing techniques, leaving many questions unanswered. Presenting the paper in a more theoretical-grounded perspective would make the paper much stronger.
This paper uses a linear stochastic bandits (LSB) method to build a surrogate model of the return of the current policy and fits the surrogate model into the EPOpt framework by sampling from the worst performing parameters according to the surrogate model. As the Thompson sampling algorithm of LSB draw samples from the distribution of MDP parameters that leads to the wost performance, why not directly use it for policy search?
The surrogate model is expected not to give accurate prediction everywhere due to the limited number of data but produces uncertainty of its prediction as an dictator. However, the uncertainty of prediction is not used by the proposed algorithm.
The presentation of the experiment section needs to be improved. The performance of the baseline needs to be explicitly presented, otherwise it is hard to compare. The proposed method will not outperform if the number of trajectories used for updating policy is the same, as the surrogate model can never be as good as the real model. It would be nice to explicitly demonstrate the runtime and performance trade-off.
Minor issues:
1. What does TRPO stand for?
2. When referring to the paper instead of the authors, the citation format needs to be (authors year) instead of authors (year).


Generative Adversarial Networks are powerful generative models that can model the manifold of natural images. We leverage this property to perform manifold regularization by approximating a variant of the Laplacian norm using a Monte Carlo approximation that is easily computed with the GAN. When incorporated into the semi-supervised feature-matching GAN we achieve state-of-the-art results for semi-supervised learning on CIFAR-10 benchmarks when few labels are used, with a method that is significantly easier to implement than competing methods. We find that manifold regularization improves the quality of generated images, and is affected by the quality of the GAN used to approximate the regularizer.

This paper builds upon the assumption that GANs successfully approximate the data manifold, and uses this assumption to regularize semi-supervised learning process.
The proposed regularization strategy enforces that a discriminator or a given classifier should be invariant to small perturbations on the data manifold z. It is empirically shown that naively enforcing such a constraint by randomly adding noise to z could lead to under-smoothing or over-smoothing in some cases which can harm the final classification performance. Consequently, the proposed regularization technique takes a step of tunable size in the direction of the manifold gradient, which has the effect of smoothing along the direction of the gradient while ignoring its norm.

Extensive experiments have been conducted, showing that the proposed approach
outperforms or is comparable with recent state-of-the-art approaches on cifar 10, especially in presence of fewer labelled data points. On SVHN however, the proposed approach fails in comparison with (Kumar et al 2017) but performs better than other approaches.
Furthermore, it has been shown that adding the proposed manifold regularization technique to the training of GAN greatly improves the image quality of generated images (in terms of FID scores and inception scores). Also, by combining the proposed regularizer with a classical supervised classifier (via pre-training a GAN and using it for regularization) decreases classification error by 2 to 3%.

Finally, it has also been shown that after training a GAN using the manifold regularization, the algorithm is able to produce similar images giving a low enough perturbation of the data manifold z.

Overall, this paper is well written and show significant improvements especially for image generation. However, the novelty is rather limited as similar ideas have been undertaken (e.g., Mescheder et al 2018), but in different contexts. The paper would be improved if the following points are taken into account:

A comparison with Graph Convolutional Network based techniques seems appropriate (e.g. Kipf and Welling 2017).
How do the FID/Inception improvements compare to (Mescheder et al 2018)?
It would be interesting to discuss why the FID score for SVHN gets worse in presence of 1000 labels.
Although there is a clear improvement in FID scores for Cifar10. It would be informative to show the generated images w/ and w/o manifold regularization.
More analysis should be provided on why (Kumar et al 2017) perform so well on SVHN.
It should be stated that bold values in tables do not represent best results (as it is usually the case) but rather results for the proposed approach.