Meta-learning has recently achieved success in many optimization problems. In general, a meta learner g(.) could be learned for a base model f (.) on a variety of tasks, such that it can be more efficient on a new task. In this paper, we make some key modifications to enhance the performance of meta-learning models. (1) we leverage different meta-strategies for different modules to optimize them separately: we use conservative "slow learners" on low-level basic feature representation layers and "fast learners" on high-level task-specific layers; (2) Furthermore, we provide theoretical analysis on why the proposed approach works, based on a case study on a two-layer MLP. We evaluate our model on synthetic MLP regression, as well as low-shot learning tasks on Omniglot and ImageNet benchmarks. We demonstrate that our approach is able to achieve state-of-the-art performance.

[Summary]
The paper presents a novel learning framework for meta-learning that is motivated by neural learning process of human over long periods. Specifically, the process of meta-learning is divided into a slow and a fast learning modules, where the slowly-learnt component accounts for low-level representation that is progressively optimized over all data seen so far to achieve generalization power, and the fastly-learnt component is supposed to pick up the target in a new task for quick adaptation. It is proposed that meta-learning should focus on capturing the meta-information for the fast learning module, and leave the slow module being updated steadily without task-specific adaptation. Theoretical analysis is presented on a linear MLP examples to shed some light on the properties of the proposed algorithm. Results on both synthetic dataset and benchmarks justify the theoretical observation and advantages.
Pros
Novel treatment and formulation of meta-learning from the perspective of fast and slow  learning process
Cons
Some interesting cases not tested
Presentation could be improved
[Originality]
The paper approaches the recently popular meta-learning from a novel perspective by decomposing the learning process into slow and fast ones.
[Quality]
Overall,  the paper is well motivated and implemented with both theoretical study and empirical justification. There are a few questions / areas for further improvements, though:
- It seems that to initialize the slow module, another set of data is needed to pretrain it before the actual meta-learning takes place to learn to optimize the fast learner (as opposed to other meta-learning methods where all parameters in a base model were meta-learnt over the meta training set). How does this affect the performance? E.g., what if the slow module is only updated over the meta-training set (still without reinitialization across different batches) without pre-training?
- In the current formulation, the base model is decomposed into two distinct (slow and fast) modules. What is the rule to decide which layers should belong to slow or fast modules? How does different choice affect the performance? Can we decompose the base model into finer granularities for different learning behaviors? E.g., a third module module in-between the fast and slow ones that follows medium learning pace.
- The theoretical study can be better organized. The proofs can be left in appendix to make room for more discussion on conclusions, non-linear and / or non-Gaussian cases.
- The write-up can be improved too at some places: proper reference at line 4 of section 1 is missing; \phi in (1) is not well defined, as well as “SOA” in section 2;
[Clarity]
The paper is generally clearly written, with a few places to improve (see comments above).
[Significance]
The paper brings in an interesting perspective to meta-learning. It can also inspire more follow-up work to better understand the problem.


Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for many common machine learning algorithms, including logistic regression, elastic-net regression, and support vector machines. Leveraging this property of differentiability, we propose a cross-validation gradient method (CVGM) for hyperparameter optimization. Our method enables efficient optimization in high-dimensional hyperparameter spaces of the cross-validation risk, the best surrogate of the true generalization ability of our learning algorithm.

The paper considers the problem of automatic tuning of hyperparameters in machine learning models. To address this problem the authors propose to use the so called cross-validation gradients, which optimize a validation objective with respect to the hyperparameters of a model. This approach and the investigated setting falls into a class of optimization problems known as bilevel optimization. The main characteristic of this class of optimization problems is the nested structure, with an outer and inner optimization objectives/problems. The outer problem corresponds to the validation objective and it is defined via an optimal solution to the inner problem which corresponds to a training objective. The paper, however, fails to make a reference to a rather rich literature on bilevel optimization (e.g., see [3-5] and references therein).
The approach, presented as Algorithm 1, does not seem different from [1] and [2] where hyperparameter optimization was considered for (kernel) support vector machines and (kernel) ridge regression. The data is initially split into k-folds (not necessarily of identical size) and each fold is used exactly once to define a validation objective whereas the complementary folds act as training data. The validation gradient is obtained by averaging the gradients of the k validation folds. Essentially, the same algorithm with k-fold cross-validation was considered in [1]. Thus, for me there does not seem to be any novelty in this approach and the paper itself.
The experiments involve synthetic regression and classification datasets but there are no novel insights that advance what is already known about the hyperparameter optimization (e.g., see [3]). For example, there is no intuition on the geometry of the optimization problem and the optimality of the outer optimization problem which is non-convex (e.g., see [7]), or dependence of the outer solution on the accuracy of the inner solution.
References:
[1] S. Keerthi, V. Sindhwani, and O. Chapelle. An Efficient Method for Gradient-Based Adaptation of Hyperparameters. NIPS 2007.
[2] O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing Multiple Parameters for Support Vector Machines. Machine Learning, 2002.
[-3] L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil. Bilevel Programming for Hyperparameter Optimization and Meta-Learning. ICML 2018.
[-4] G. Kunapuli, K.P. Bennet, J. Hu, and J-S. Pang. Bilevel Model Selection for SVMs. American Mathematical Society, 2008.
[-5] E.S.H. Neto and A.R. de Pierro. On Perturbed Steepest Descent Methods with Inexact Line Search for Bilevel Convex Optimization. Journal of Mathematical Programming and Operations Research, 2011.
[-6] B. Colson, P. Marcotte, and G. Savard. A Trust-Region Method for Nonlinear Bilevel Programming: Algorithms and Computational Experience. Computational Optimization and Applications, 2005.
[-7] M. Janzamin, H. Sedghi, and A. Anandkumar. Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Method. arXiv preprint arXiv:1506.08473v3, 2016.


We study the problem of designing provably optimal adversarial noise algorithms that induce misclassification in settings where a learner aggregates decisions from multiple classifiers. Given the demonstrated vulnerability of state-of-the-art models to adversarial examples, recent efforts within the field of robust machine learning have focused on the use of ensemble classifiers as a way of boosting the robustness of individual models. In this paper, we design provably optimal attacks against a set of classifiers. We demonstrate how this problem can be framed as finding strategies at equilibrium in a two player, zero sum game between a learner and an adversary and consequently illustrate the need for randomization in adversarial attacks. The main technical challenge we consider is the design of best response oracles that can be implemented in a Multiplicative Weight Updates framework to find equilibrium strategies in the zero-sum game. We develop a series of scalable noise generation algorithms for deep neural networks, and show that it outperforms state-of-the-art attacks on various image classification tasks. Although there are generally no guarantees for deep learning, we show this is a well-principled approach in that it is provably optimal for linear classifiers. The main insight is a geometric characterization of the decision space that reduces the problem of designing best response oracles to minimizing a quadratic function over a set of convex polytopes.

The paper studies the problem of adversarial examples generation. The authors phrase the following problem: given a set of models  C, we want to find an adversarial perturbation that maximizes the loss on an ensemble of models. However, the ensemble weights are chosen by the learner. In the case that we have one example, this is equivalent to asking that the same adversarial perturbation (or distribution over perturbations) fools all the individual models in my collection. This is a reasonable phrasing of the problem, though it seems different from versions studied in literature. In particular, previous works used uniform ensembles.
More generally, the authors consider a set of m examples, and the adversarial player now looks for a (distribution over) perturbations for each of the n examples. The learner player selects mixing weights to minimize the error rate. This is an interesting formulation of the problem: in particular, tying the mixing weights used for all examples is a non-intuitive change and does not have the clean interpretation above any more.
This notion of allowing mixing weights on the learner is a change from previous work. The authors would do well to explain why this formulation is chosen and what the interpretation is. It corresponds to a specific attack model where the learner and the adversary make choices in a very specific order, and could use further explanation on when this a reasonable attack model. Note that previous work looked at the setting of all weights being equal, and one natural variant is to allow a set of mixing weights per example, which would correspond to finding a perturbation (or distribution over perturbations) for this example that fool all models in the set C. The version studied here is left unexplained in the current work.
The authors then argue that we can solve this game by playing MW vs. best response. They propose using best response on the adversarial player. This player is then trying to find the perturbation that maximizes the p-weighted sum of the 0-1 (or rather surrogate) losses, where p represents the mixing weights on C. The authors show that in the convex case, if there is a pure NE, then the best response can be found: in this case we get a convex problem. They study the convex case a bit more, showing that there is at most an exponential number of values for the 0-1 loss, since a {0,1} vector defining which side of each classifier in C x falls in fully defines the loss at x.
Finally, the authors move to the non-convex case where the experiments are done. The authors report interesting results on imagenet and for mnist for the convex case. I had some trouble understanding the imagenet results. For one, it seems fishy that their Madry et al. attack is worse than the FGM for many of the models and suggests strongly that the parameters for the Madry attack were not properly tuned. It is hard to know since the paper does not report on various parameters for these attacks. Second, these attacks are designed for l_infty and modifying them for l_2 would be necessary for a fair comparison. Finally, I am not sure why the authors do not compare to the Carlini and Wagner attacks on Imagenet, which is actually an l2 attack and makes the accuracy 0 at a slightly larger perturbation radius. Also, the authors would do well to emphasize that for larger perturbation radii, there are attacks which make the accuracy zero, and the contribution here seems to be look at smaller radii.
My primary concern with the work is that it is not clear to me how the specific two player game is motivated. The authors do not justify why it makes sense to allow weights on the ensemble, and also why these weights need to be tied together across examples. For a paper that makes strong claims about its approach being principled, this is a serious shortcoming in my view. Secondarily, the experiments section leaves me worried that the comparison is with improperly tuned versions of previous work. I would therefore not be in favor of accepting this paper.
Comments:
pg 1 : "One of the most pressing ..." : that is perhaps an unnecessary exaggeration.
pg 2: The name "NSFW" is an unfortunate choice, is completely non-informative about the contribution and I strongly recommend the authors reconsider it.
- As far as I can tell, Tramer et al. do not build an ensemble model at all; the ensemble word there refers to an ensemble of adversarial perturbations.
- The hinge loss is actually not smooth. However, I don't quite see why you need smoothness there.


While momentum-based methods, in conjunction with the stochastic gradient descent, are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In practice, the momentum parameter is often chosen in a heuristic fashion with little theoretical guidance. In this work we use the framework of algorithmic stability to provide an upper-bound on the generalization error for the class of strongly convex loss functions, under mild technical assumptions. Our bound decays to zero inversely with the size of the training set, and increases as the momentum parameter is increased. We also develop an upper-bound on the expected true risk, in terms of the number of training steps, the size of the training set, and the momentum parameter.

This paper presents an analysis of generalization error of SGD with multiple passes for strongly convex objectives using the framework of algorithmic stability [Bousquet and Elisseef, 2002] and its recent use to analyze generalization error of SGD based methods [Hardt, Recht and Singer 2016].
The problem considered by this work is interesting and raises the possibility of understanding generalization related questions of SGD style methods when augmented with momentum, which is common practice in Deep Learning [Sutskever et al. 2013]. That said, there are some concerns about the results as presented in this paper, which I will elaborate below:
- Consider the stability bound admitted by theorem 2: The special case (similar to theorem 3.9 of Hardt et al 2016) when the learning rate alpha = 1/beta (which is the typical learning rate that theory advocates), and setting kappa = beta/gamma where kappa is the condition number of the problem, leads to the following bound on momentum allowed by theorem 2, which is:
(something non-positive) <= mu < 1/(3*kappa).
This is basically the regime where momentum does not make any difference towards accelerating optimization. Referring to the standard value of momentum for strongly convex functions, we see that the momentum is set as mu = (sqrt(kappa)-1)/(sqrt(kappa)+1) [Nesterov, 1983], or, mu = ( (sqrt(kappa)-1)/(sqrt(kappa)+1) )^2  [Polyak,1964]. Upon simplification of this standard momentum values, we see that mu \approx 1 - 1/sqrt(kappa) which grows close to one as kappa grows large. On the other hand, the momentum values admitted by the paper for their bound is super tiny (which gets to zero as the condition number kappa grows large). This essentially implies there is not much about momentum that is captured by the bound of theorem 2 since there is no characterization of the provided bound for theoretically advocated and practically used parameters for momentum.
- In proposition 1, there is no quantitative description of what "sufficiently small" mu (momentum parameter) is - this statement is imprecise. As mentioned in the previous point, sufficiently small mu really is not descriptive of momentum parameters employed in practice (mu in practice typically is >= 0.9). For strongly convex objectives, this should be closer to 1- (1/sqrt(kappa)). Sufficiently small mu parameter essentially does not yield quantitatively different behaviors compared to standard SGD.
In summary, while this paper attempts to make progress on an interesting question, but falls short and doesn't really capture the behavior of these methods that is even mildly reflective of practice (even in terms of the parameter regimes admitted by the bounds proven in the theorems).
- This paper does not perform a thorough literature survey of published results. Furthermore, this paper does not present a precise treatment of assumptions (and implications) amongst other works cited in the paper (see for e.g. [4] below).
[1] Polyak (1987) presents (generalization) behavior of Heavy Ball momentum with noisy (inexact) gradients.
[2] Several efforts in Signal Processing literature do consider the similar setting as one considered by this paper, which is that of Heavy Ball (called accelerated LMS) method with noisy gradients: refer to Proakis (1974), Roy and Shynk (1990), Sharma et al. (1998).
[3] Kidambi et al (2018) estimate the "optimization" power (which is a part of characterization of generalization error [Bach and Moulines 2011], since this dominates at the start of optimization) of HB method with Stochastic Gradients and prove that HB+stochastic gradients does not offer any speedup over vanilla SGD.
[4] Loizou and Richtarik provide an analysis of stochastic heavy ball with super large batch sizes (so they end up showing accelerated rates) under similar assumptions as considered by this paper, such as assuming the function is smooth and strongly convex. However, the paper dismisses the work of Loizou and Richtarik to be working with a different set of assumptions - this is not really true.


A STATISTICAL APPROACH TO ASSESSING NEURAL NETWORK ROBUSTNESS
We present a new approach to assessing the robustness of neural networks based on estimating the proportion of inputs for which a property is violated. Specifically, we estimate the probability of the event that the property is violated under an input model. Our approach critically varies from the formal verification framework in that when the property can be violated, it provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not verifiable. Furthermore, it provides an ability to scale to larger networks than formal verification approaches. Though the framework still provides a formal guarantee of satisfiability whenever it successfully finds one or more violations, these advantages do come at the cost of only providing a statistical estimate of unsatisfiability whenever no violation is found. Key to the practical success of our approach is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical robustness framework. We demonstrate that our approach is able to emulate formal verification procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability.

Verifying the properties of neural networks can be very difficult.  Instead of
finding a formal proof for a property that gives a True/False answer, this
paper proposes to take a sufficiently large number of samples around the input
point point and estimate the probability that a violation can be found.  Naive
Monte-Carlo (MC) sampling is not effective especially when the dimension is
high, so the author proposes to use adaptive multi-level splitting (AMLS) as a
sampling scheme. This is a good application of AMLS method.
Experiments show that AMLS can make a good estimate (similar quality as naive
MC with a large number of samples) while using much less samples than MC, on
both small and relatively larger models.  Additionally, the authors conduct
sensitivity analysis and run the proposed algorithm with many different
parameters (M, N, pho, etc), which is good to see.
I have some concerns on this paper:
I have doubts on applying the proposed method to higher dimensional inputs. In
section 6.3, the authors show an experiments in this case, but only on a dense
ReLU network with 2 hidden layers, and it is unknown if it works in general.
How does the number of required samples increases when the dimension of input
(x) increases?
Formally, if there exists a violation (counter-example) for a certain property,
and given a failure probability p, what is the upper bound of number of samples
(in terms of input dimension, and other factors) required so that the
probability we cannot detect this violation with probability less than p?
Without such a guarantee, the proposed method is not very useful because we
have no idea how confident the sampling based result is. Verification needs
something that is either deterministic, or a probabilistic result with a small
and bounded failure rate, otherwise it is not really a verification method.
The experiments of this paper lack comparisons to certified verification
methods. There are some scalable property verification methods that can give a
lower bound on the input perturbation (see [1][2][3]).  These methods can
guarantee that when epsilon is smaller than a threshold, no violations can be
found.  On the other hand, adversarial attacks give an upper bound of input
perturbation by providing a counter-example (violation). The authors should
compare the sampling based method with these lower and upper bounds. For
example, what is log(I) for epsilon larger than upper bound?
Additionally, in section 6.4, the results in Figure 2 also does not look very
positive - it unlikely to be true that an undefended network is predominantly
robust to perturbation of size epsilon = 0.1. Without any adversarial training,
adversarial examples (or counter-examples for property verification) with L_inf
distortion less than 0.1 (at least on some images) should be able to find. It
is better to conduct strong adversarial attacks after each epoch and see what
are the epsilons of adversarial examples.
Ideas on further improvement:
The proposed method can become more useful if it is not a point-wise method.
If given a point, current formal verification method can tell if a property is
hold or not.  However, most formal verification method cannot deal with a input
drawn from a distribution randomly (for example, an unseen test example). This
is the place where we really need a probabilistic verification method. The
setting in the current paper is not ideal because a probabilistic estimate of
violation of a single point is not very useful, especially without a guarantee
of failure rates.
For finding counter-examples for a property, using gradient based methods might
be a better way. The authors can consider adding Hamiltonian Monte Carlo to
this framework (See [4]).
References:
There are some papers from the same group of authors, and I merged them to one.
Some of these papers are very recent, and should be helpful for the authors
to further improve their work.
[1] "AI2: Safety and Robustness Certification of Neural Networks with Abstract
Interpretation", IEEE S&P 2018 by Timon Gehr, Matthew Mirman, Dana
Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, Martin Vechev
(see also "Differentiable Abstract Interpretation for Provably Robust Neural
Networks", ICML 2018. by Matthew Mirman, Timon Gehr, Martin Vechev.  They also
have a new NIPS 2018 paper "Fast and Effective Robustness Certification" but is
not on arxiv yet)
[2] "Efficient Neural Network Robustness Certification with General Activation
Functions", NIPS 2018. by Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui
Hsieh, Luca Daniel.
(see also "Towards Fast Computation of Certified Robustness for ReLU Networks",
ICML 2018 by Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh,
Duane Boning, Inderjit S. Dhillon, Luca Danie.)
[3] Provable defenses against adversarial examples via the convex outer
adversarial polytope, NIPS 2018. by Eric Wong, J. Zico Kolter.
(see also "Scaling provable adversarial defenses", NIPS 2018 by the same authors)
[4] "Stochastic gradient hamiltonian monte carlo." ICML 2014. by Tianqi Chen,
Emily Fox, and Carlos Guestrin.
============================================
After discussions with the authors, they agree to revise the paper according to our discussions and my primary concerns of this paper have been resolved. Thus I increased my rating.


We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset. Experiments on convolutional and capsules neural networks show that both models have poor performance in dealing with global translation-invariance; however, the performance improved by using data augmentation. Although the capsule network is better on the MNIST testing dataset, the convolutional neural network generally has better performance on the translation-invariance.

The authors of this paper compare the robustness of CNN and CapsNet to global translation on the MNIST dataset. Both models were trained on the standard training set of MNIST, and then tested on a set with digits shifting from the upper left corner to the lower right corner. The results of both models were poor. To improve it, the authors add some shifted digits to the training set, and the performances of both models were significantly enhanced. Moreover, the performance of CNN was better than that of CapsNet in the experiments. Generally speaking, the work presented in this paper is clear and straightforward. However, the work is not significant enough to publish as a ICLR paper. Below is my major comments.
1. There are lots of typos and grammatical errors everywhere in the paper. Thus, the manuscript was not well prepared.
2. It is unclear which CapsNet and what settings were used in the experiment.
3. It is well-known that convolutional networks are good at capturing local patterns from the images, while capsule networks enhance it to consider global configurations of the local patterns, and robust to affine transformation. Obviously, the experiments presented in this manuscript is too simple. Lots of work should be done in the investigation. For example,o on the training set and shifted test set, the authors can enlarge the background and keep the digits in the original size to make it as a local pattern in the image. Will it be detected by CNNs with larger receptive fields for the images? How is it compared with CapsNets?
4. How are both models compared on other (perhaps more complicated and larger) datasets?
In summary, the work presented here is interesting, but lots of work should be done in order to make it publishable.


Generative adversarial networks (GANs) have achieved outstanding success in generating the high-quality data. Focusing on the generation process, existing GANs learn a unidirectional mapping from the latent vector to the data. Later, various studies point out that the latent space of GANs is semantically meaningful and can be utilized for advanced data analysis and manipulation. In order to analyze the real data in the latent space of GANs, it is necessary to investigate the inverse generation mapping from the data to the latent vector. To tackle this problem, the bidirectional generative models introduce an encoder to establish the inverse path of the generation process. Unfortunately, their inference mapping does not accurately predict the latent vector from the data because the imperfect generator rather interferes the encoder training. In this paper, we propose an effective algorithm to accurately infer the latent vector based on existing unidirectional GANs. It is important to note that we focus on increasing the accuracy and efficiency of the inference mapping but not influencing the GAN performance toward image quality or diversity. Instead, utilizing the proposed inference mapping algorithm, we suggest a new metric for evaluating the GAN models by measuring the reconstruction error of unseen real data. The experimental analysis demonstrates that the proposed algorithm achieves more accurate inference mapping than existing methods and provides the robust metric for evaluating GAN performance.

Paper Summary:
This paper proposes to reconstruct the generated images to the their corresponding latent code. As claimed, the goal is to improve the accuracy and efficiency of inference mapping better than other inference mapping techniques, while maintaining their generation quality.
 Instead of using an independent encoder, the authors propose to share the encoder parameters with the discriminator: a Connection Network (CN) is built on top of the features extracted by the discriminator. The weight-sharing machisme shows better performance in Figure 1.
The proposed method has two benefits: : a) manipulating the image by disentangling the latent space and b) suggesting a new metric for assessing the GAN model by measuring reconstruction errors of real data.
General Comments:
In term of algorithm, the paper essentially adds the conscontruction term (CN) to the standard GAN loss, and partially shares the weights of the “encoder” and discriminator. However, it is almost identical to the existing works, which are NOT cited, and the connections are not discussed.
Connection to InfoGAN: To relate the generated images to the latent code,  the proposed method employs the reconstruction loss, InfoGAN employs the mutual information. Note that reconstruction loss = negative log likelihood, and effectively is equivalent to Mutual Information and Conditional Entropy in the case. Please see the discussion in Lemma 3 and Appendix A of [3] for detailed discussion.  Further, InfoGAN has proposed to to sharing weights of the encoder and discriminator, exactly the same with this submission. The claimed advantage is to disentangle the latent space. It is not surprise at all, once the authors see the connection to InfoGAN, which was originally proposed to disentangle the latent codes.
Connection to CycleGAN: CycleGAN consists of four losses: two reconstruction losses and two standard GAN losses. As shown in Section 4 of [3] “Connecting ALI and CycleGAN”, one reconstruction loss and  one standard GAN loss is sufficient to achieve CycleGAN’s objective, the other two losses would only help to accelerate. In another word, the proposed method is exactly half of the CycleGAN losses.
The author mention in Abstract that “the bidirectional generative models introduce an encoder to establish the inverse path of the generation process. Unfortunately, their inference mapping does not accurately predict the latent vector from the data because the imperfect generator rather interferes the encoder training.” This is the non-identifiable issue of ALI/BiGAN discovered in [3]. Please clarify.
The proposed method should compare with [1] and [2] in great detail, to demonstrate its own advantages. Given the missing literature, the current experimental comparisons seem not that meaningful, because the baseline methods are not really the competitors.
One interesting contribution of the submission is to consider the reconstruction errors to measure the quality of GANs. To my best knowledge, it is original.
References:
[1] InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, NIPS 2016
[2] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV 2017
[3] ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching, NIPS 2017


DIFFUSION SCATTERING TRANSFORMS ON GRAPHS
Stability is a key aspect of data analysis. In many applications, the natural notion of stability is geometric, as illustrated for example in computer vision. Scattering transforms construct deep convolutional representations which are certified stable to input deformations. This stability to deformations can be interpreted as stability with respect to changes in the metric structure of the domain. In this work, we show that scattering transforms can be generalized to nonEuclidean domains using diffusion wavelets, while preserving a notion of stability with respect to metric changes in the domain, measured with diffusion maps. The resulting representation is stable to metric perturbations of the domain while being able to capture "high-frequency" information, akin to the Euclidean Scattering.

The paper introduces an adaptation of the Scattering transform to signals defined on graphs
by relying on multi-scale diffusion wavelets, and studies a notion of stability of this representation
with respect to changes in the graph structure with an appropriate diffusion metric.
The notion of stability in convolutional networks is an important one, and the proposed notion of stability
with respect to diffusion distances seems like an interesting and relevant way to extend this to signals on graphs.
With this goal in mind, the authors introduce a scattering transform on graphs by relying on diffusion wavelets,
and provide an appropriate study of stability, which seems to highlight relevant properties of the graphs.
The proposed representation seems to provide benefits compared to the previous work of Zou & Lerman,
particularly regarding computational efficiency, as well as stability with respect to a metric that is perhaps more
useful, though there is a dependence on the graph topology through the spectral gaps.
In addition, the experiments on author attribution and source localization suggest that the
resulting representation remains discriminative, in addition to providing stability to changes in graph structure.
I find that these contributions provide an interesting advance in theoretical understanding of graph convolutional networks
from a stability perspective, in addition to introducing a useful non-learned representation,
and am thus in favor of acceptance.
Nevertheless, some parts of paper would benefit from further discussions and more clarity:
- other than empirically, one aspect that's missing compared to the original study of the scattering transform is energy preservation. The authors could at least provide a discussion of whether such a property can be obtained here as well (does it depend on the spectral gap through C(beta)?)
- what is the role of the spectral gap in the stability bounds? is this a drawback of the diffusion metric / choice of wavelets?
- Section 3.2 suggests that metric stability is a good way to characterize stability, by seeing deformations in Euclidian domains as a change to the ground metric. Yet, in Euclidian scattering, the same representation is applied to a deformed signal and the original signal, and stability is measured with the Euclidian metric.
Can the link be made more precise, by explaining what a deformation of a signal would be on a graph, or by applying arguments from the proposed construction to the Euclidian case?
- the paper is heavy on terminology from wavelets and harmonic analysis, a more detailed presentation of diffusion wavelets and related concepts such as localization would be beneficial. Also, it seems like the chosen wavelets in the construction favor spatial over frequential localization - is this due to a trade-off? if so, can it be avoided?
Some more detailed comments:
- Section 2, 'generally far weaker': what is meant by 'weaker'?
- Section 3.3:
  * 'calculus on T': T is used before being defined
  * clarify what norm is used (I assume operator norm?)
  * 'defines a distance', 'stronger than .. GH': this should probably be justified
- Section 4:
  * 'optimal spatial localization', 'temporal difference', 'favoring spatial over frequential localization': these could be clarified
  * 'amplify the signal': what does this mean?
  * the sentence about the choice of the appropriate J is not clear and should be further clarified
- Section 5.1:
  * the sentence about the choice pi/pi* = 1 should be clarified. Also, where is this assumption used?
  * epsilon_psi, epsilon_U should be defined
  * 'given that [..] by definition': this doesn't seem to be defined elsewhere
  * (16): isn't a factor m missing in the first term?


The recently proposed information bottleneck (IB) theory of deep nets suggests that during training, each layer attempts to maximize its mutual information (MI) with the target labels (so as to allow good prediction accuracy), while minimizing its MI with the input (leading to effective compression and thus good generalization). To date, evidence of this phenomenon has been indirect and aroused controversy due to theoretical and practical complications. In particular, it has been pointed out that the MI with the input is theoretically infinite in many cases of interest, and that the MI with the target is fundamentally difficult to estimate in high dimensions. As a consequence, the validity of this theory has been questioned. In this paper, we overcome these obstacles by two means. First, as previously suggested, we replace the MI with the input by a noise-regularized version, which ensures it is finite. As we show, this modified penalty in fact acts as a form of weight-decay regularization. Second, to obtain accurate (noise regularized) MI estimates between an intermediate representation and the input, we incorporate the strong prior-knowledge we have about their relation, into the recently proposed MI estimator of Belghazi et al. (2018). With this scheme, we are able to stably train each layer independently to explicitly optimize the IB functional. Surprisingly, this leads to enhanced prediction accuracy, thus directly validating the IB theory of deep nets for the first time.

While overall the writing quality of the paper is high, the paper itself is a strong rejection.  I believe the analysis of the paper is at points flawed, and the experiments are minimal.
This work attempts to study the degree to which a layer by layer information bottleneck inspired objective can improve performance, as well as generally attempt to clarify some of the discussion surrounding Shwartz-Ziv & Tishby 2017.  Here, the authors study a deterministic neural network, for which the mutual information estimation is difficult (I(X,L)) and error prone.  To combat this they use the noise-regularized mutual information estimator (I(X; L+eps)).  To actually estimate the mutual information the authors use the MINE estimator of Belghazi (2018).  Here they suggest using the neural network itself as a structural element in the form of the discriminator to take advantage of the specific circumstances in this case.  Doing this ensured that their estimator diverged in the zero noise limit as expected.  From here they show some experimental results of the effect of their objective on an MNIST / CIFAR10 classification task.
This paper fits into what is an increasingly large discussion in the literature, surrounding Information Bottleneck.  The paper itself does a very good job of citing recent relevant work.  Technically however I take issue with the framing of previous work in the last paragraph of the "Deep neural nets" subsection of Section 2.  Technically Achille & Soatto explicitly formed a variational approximation to the posterior over the weights of the neural network and so was not a "single bottleneck layer" as stated in the paper.  More generally at the end of that paragraph it is implied that the single bottleneck layer scheme "deviates from the original theory".  This is a misleading characterization of the original information bottleneck (Tishby et al 1999) in which there was a single random variable, a representation of the data (Z) satisfying the Markov conditions Z <- X -> Y.   I believe the authors instead meant to say that the cited works deviate from the information bottleneck theory of learning suggested in (Shwartz-Ziv & Tishby 2017).  In general the paper does a poor job of distinguishing between the Shwartz-Ziv & Tishby paper and the rest, but this is a distinction that should be maintained.  The original information bottleneck may and has demonstrated utility regardless of whether the information bottleneck generally can help explain why ordinary deterministic feed forward networks trained with cross entropy and sgd generalize well.
This also raises one of the main problems with the current work. The title, abstract and especially the conclusion ("This provides, for the first time, strong and direct emperical evidence for the validity of the IB theory of deep learning") seem to present the paper as somehow offering some clarity and further support for the assertions of the Shwartz-Ziv & Tishby 2017 paper, but that paper hoped to establish that information bottleneck can explain the workings of ordinary networks.  Here the authors modify the ordinary cross entropy objective, and so their networks are necessarily not ordinary and so they cannot claim they have helped clarify our understanding of the vast majority of neural networks currently being trained.  Again, this is distinct and should be kept distinct from the utility of their proposed objective, itself inspired by the information bottleneck.  Here too the paper falls flat.  If instead of attempting to comment on networks as they are designed today they aim to proposed a new information bottleneck inspired objective they really ought to directly compare other attempts along those lines (such as the ones they themselves cite  Alemi et al. 2018, Kolchinsky et al. 2017, Chalk et al. 2016, Achile & Soatto 2018, Belghazi et al. 2018) but there are no comparative studies.
The experiments are extremely lacking, not only are any of their cited alternatives compared, they don't compare to what would be an equivalent network to their but where they did utilize the noise at every layer and actually made the network stochastic.  Their reported numbers are not very impressive with their top MNIST number at 98.09 and their baseline at 97.73. These numbers are worse than many of the papers they themselves cite.  Only a single comparative results for both a limited training set run and the full one are shown, as well as only a single choice of beta.  The CIFAR10 numbers are not very good either.  There is some discussion of the text suggesting they believe their method acts like an approximate weight decay, but there are no results showing the effect of weight decay just on the baseline classification accuracies they compare against.
Technically a deterministic function need not have infinite mutual information, if it is non-invertible, i.e. the sign function, or just floating point discretization.
Their own results in Figure 2 and the main body of the text highlight that the authors believe the true mutual information between the activations of the intermediate layers and the input is infinite.  If the true mutual information is infinite and the noise regularized estimator is only meant for comparative purposes, why then are the results of the training trajectories interpreted so literally as estimates of the true mutual information?
Just plugging in the Discriminator for the objective (equation (7)) is flawed.  The discriminator, if optimal would learn to approximate the density ratio 1 + log p(x,y)/(p(x) p(y)) .   ( see f-GAN, Norowin et al. 2016).  How does this justify using the individual elements of the discriminator in the functional form of the IB objective?
At the bottom of page 6 they rightfully say that mutual information is invariant to reparameterizations, but their noise regularized mutual information estimator is not (by their own reference (Saxe et al 2017).
The discussion at the center of page 8 is confusing.   They claim that Figure 5 (a) is more 'quantized' than (b) and "has reduced entropy".  I think it should be the other way.  More clusters should translate to a higher KL divergence, or higher entropy.  If you need only identify which cluster an activation is in, that should require log K nats where K is the number of clusters.  (a) shows more clusters and so seems like it should cost more and have a higher entropy not a lower one.
Despite a recurring focus of the text that this paper applies and information theoretic objective at each layer of the network, and hence is novel, the final sentence of the paper suggests it might not actually be needed and single layer IB objectives can work as well.


Currently the only techniques for sharing governance of a deep learning model are homomorphic encryption and secure multiparty computation. Unfortunately neither of these techniques is applicable to the training of large neural networks due to their large computational and communication overheads. As a scalable technique for shared model governance, we propose splitting deep learning model between multiple parties. This paper empirically investigates the security guarantee of this technique, which is introduced as the problem of model completion: Given the entire training data set or an environment simulator, and a subset of the parameters of a trained deep learning model, how much training is required to recover the model's original performance? We define a metric for evaluating the hardness of the model completion problem and study it empirically in both supervised learning on ImageNet and reinforcement learning on Atari and DeepMind Lab. Our experiments show that (1) the model completion problem is harder in reinforcement learning than in supervised learning because of the unavailability of the trained agent's trajectories, and (2) its hardness depends not primarily on the number of parameters of the missing part, but more so on their type and location. Our results suggest that model splitting might be a feasible technique for shared model governance in some settings where training is very expensive.

This paper proposes the interesting idea of analyzing how difficult
it is to re-initialize and re-train layers in neural networks.
They study these techniques in the context of ImageNet classification
and reinforcement learning in the Atari and DeepMind lab domains.
While these are interesting ideas and domains to study, I have
concerns with the positioning and execution of the paper.
[Positioning, execution and motivation]
On the positioning of the paper, a significant part of the introduction
and related work section is spent arguing that this approach can be used
for shared model governance in contexts where homomorphic encryption
or secure multi-party computation would instead be used.
Comparing the approaches studied in this paper to these
sophisticated cryptographically-motivated techniques seems
like too much of a stretch, as the methods serve very different
purposes and in most cases cannot even be directly compared.
The first and second paragraph discuss the vision of distributing
the training of models between multiple parties.
I agree that this is a useful area to study and direction
for the community to go, but as the introduction of this paper
states, this is the most interesting when the parties have
control over logically separate components of the modeling pipeline
and also when joint training of the components is being done,
potentially on disjoint and private datasets.
The empirical results of this paper do none of this,
as they only look at the case when a single layer is being
replaced.
Furthermore the motivation and positioning of the paper is
not carried through in the empirical setup, where they
investigate approaches that do training over all
of the parameters of the model, breaking the assumption
that the parties should be independent and should
not share information.
[Metrics for measuring model completeness]
Section 3.1 defines the metric of completion hardness that is
used throughout the rest of the paper. The metric looks at the
number of iterations that re-training the model takes to
reach the same performance as the original model.
It's not clear why this is an important metric and I am
not convinced it is the right one to use as it:
1) does not give a notion of how nicely the missing portion
was recovered, just that the accuracy reached the
same accuracy as the original network, and
2) methods with a very long per-iteration runtime such as
second-order and sampling-based methods could be used to
reach a good performance in a small number of iterations,
making these methods appear to be very "good" at
completing models. I don't think it is nice that this
metric relies on the same optimizer being used for the
original model and the completed model.
I think it's more interesting to study *how much* data is
required to recover missing portions of the model instead
of how many iterations are needed to recover the same performance.
The supervised learning experiments appear to be done
using the entire dataset while the RL experiments do
present a setting where the data is not the same.
[Empirical results]
I am also surprised by the empirical finding in Section 5.1
that T1 outperforms T2, since it seems like only optimizing
the parameters of the missing layer would be the best
approach. I think that if a similarity metric was used
instead, T2 would be significantly better at finding the
layer that is the most similar to the layer that was removed.
Some smaller comments:
1. In Section 3.1, the definition of C_T does not use T explicitly
   inside of it.
2. In the last paragraph of Section 3.1 and first paragraph of
   Section 3.2, N should be defined as an iteration that
   reaches the best loss.
3. The description of T3 does not say what method is used to
   optimize the over-parameterized layer, is it T1 or T2?
4. Why does T4 use T1 instead of T2?
5. In the experimental setup, why is T2 applied with a different
   learning rate schedule than the original training procedure?
6. Why is T2 not shown in the AlexNet results for Figure 2?
7. The dissimilar axes between the plots in Figure 2 and
   Figure 3 make them difficult to compare and interpret.
8. It's surprising that in Figure 3, the hardness of \alpha=1.0
   for T2 is 1.0 for everything.


A wide range of defenses have been proposed to harden neural networks against adversarial attacks. However, a pattern has emerged in which the majority of adversarial defenses are quickly broken by new attacks. Given the lack of success at generating robust defenses, we are led to ask a fundamental question: Are adversarial attacks inevitable? This paper analyzes adversarial examples from a theoretical perspective, and identifies fundamental bounds on the susceptibility of a classifier to adversarial attacks. We show that, for certain classes of problems, adversarial examples are inescapable. Using experiments, we explore the implications of theoretical guarantees for real-world problems and discuss how factors such as dimensionality and image complexity limit a classifier's robustness against adversarial examples.

This paper explores the inevitability of adversarial examples with concentration inequalities. It is motivated by the difficulties of achieving adversarial robustness in literature. It derives isoperimetric inequalities on a cube, and then discuss the adversarial robustness of data distributed inside the cube, with the assumption that the data has bounded density. These inequalities are established on different norms. The authors then discuss limitation of the proposed bounds when analyzing practical data distribution and discussed the influence of dimensionality on adversarial robustness.
Novelty of the idea:
The idea of using concentration inequalities to explain vulnerability is novel in the field of adversarial examples and is a relevant/meaningful angle on understanding this phenomenon. (Although there are concurrent works also relating concentration inequalities to adversarial robustness, they don't diminish the novelty of this work.)
On technical contributions:
In summary, this paper applies / adapts previous results in concentration inequalities to develop bounds related to adversarial examples. The bounds in Lemma 3 are on any p>0, this seems to be new to my knowledge, but the technical contribution in the proof is limited.
Here are some detailed comments.
The authors claim that
"This question is complicated by the fact that simple, geometric isoperimetric inequalities fail to exist for the cube, and the shapes that achieve minimal \eps-expansion (if they exist) depend on the volume they enclose and the choice of \eps."
This statement is at least misleading, if not wrong. It is well known that geometric isoperimetric inequality does exist for cube for the L2 case (see Ledoux, M., 2001. Proposition 2.8.), and the proof procedure the author used is also very similar to the proofs in Ledoux, M., 2001.
Theorem 5's proof is confusing, if not wrong.
This is my brief recap on the first part of Thm 5,
If there exists eps and p such that, for all classifiers on MNIST, a random image has eps-adv with probability at least p, then for all classifiers on b-MNIST, a random image has b*eps-adv with probability at least p.
The proof in Appendix E says b-MNIST images can be classified by first downsampling. These downsampled classifiers do not cover "all classifiers on b-MNIST", so I don't see how the proof stands.
Likewise, the proof of the second part has the similar problem.
Therefore, I'm not yet convinced that Thm 5 is correct.
Also I suggest the authors use more rigorous language to present Theorem 5, in a similar fashion to previous theorems.
Re: Lemma 4, my understanding is that it is from previous literature. The authors should point out exactly where is it from (with section# and theorem#), so that readers and reviewers can more easily check the correctness of it.
The authors mention that "Intuitively, the concentration limit Uc can be interpreted as a measure of image complexity."
I think this statement is problematic. It is, at best, oversimplifying the the problem. If we assume the data lies in low-dimensional space, the volume of the support will be 0, no matter how complex the shape of the manifold is. This lead to unbounded density in the ambient dimension.
Even when considering "expanded dataset" like the authors discussed in Section 7, it is not obvious that Uc can be interpreted as image complexity. To make such a claim, more assumptions need to made and more analyses need to be done.
Similar comments applies to the "correlations between pixels" and concentration.
On the significance:
As the author themselves have already mentioned, the bounds described in the paper all depends on the bounded density of the data distribution. In practice, the density of data distribution is difficult to understand, if not impossible. Therefore it is still inconclusive whether the "inevitability" exists. But to be fair, I believe this is mostly due to the difficulty of the problem being studied.
Clarity and writing:
The skeleton of the paper is well written and easy to follow. I've pointed out some problems in my previous comments.
I also appreciate that the authors made efforts to not overclaim.
here are a few more comments:
- I personally feel Section 3 as an "warm-up" section is redundant, and the authors can consider move them to the appendix.
- In Section 6 and 7, the authors talk about when is the bound "meaningful" and "active". This part is confusing/misleading. eps=sqrt(n) is actually the maximum possible perturbation and not falls into the common "adversarial perturbation" where the perturbation does not change the semantic meaning of the image. There should be a least an additional numerical examples on small eps, so the readers have better ideas on the tightness/looseness of the bound.
References:
Ledoux, M., 2001. The concentration of measure phenomenon (No. 89). American Mathematical Soc..
==========================
I change my rating on this paper to be 6, after the authors' response.


For the challenging semantic image segmentation task the best performing models have traditionally combined the structured modelling capabilities of Conditional Random Fields (CRFs) with the feature extraction power of CNNs. In more recent works however, CRF post-processing has fallen out of favour. We argue that this is mainly due to the slow training and inference speeds of CRFs, as well as the difficulty of learning the internal CRF parameters. To overcome both issues we propose to add the assumption of conditional independence to the framework of fully-connected CRFs. This allows us to reformulate the inference in terms of convolutions, which can be implemented highly efficiently on GPUs .Doing so speeds up inference and training by two orders of magnitude. All parameters of the convolutional CRFs can easily be optimized using backpropagation. Towards the goal of facilitating further CRF research we have made our implementations publicly available.

The authors propose an efficient method to perform message passing on a truncated Gaussian kernel CRF. The main contributions are the definition of a specific form of truncated Gaussian kernel that allows for fast message passing via convolutions, and the implementation of such parallelized message passing on GPU.
In my opinion, the paper fails to convey the main idea in a clear and precise manner, the notation is mixed and often confusing, furthermore there are a number of sentences that should be rephrased to be less sensationalist, or removed. The experiments seem to show performance in par with the FullCRF on decoupled training, which seem in contrast with the much bigger performance gain of the first experiment on syntetic data. No discussion has been provided as to the possible reasons of this performance gap, although the experimental settings appear to be similar. Finally, in the last experiments with end-to-end training the authors report a performance improvement over CRFasRNN, a 3 years old paper that is far away in terms of performance with the current SOTA on Pascal VOC. The authors base on a different network than that of the CRFasRNN baseline (i.e., the difference is not only in the CRF implementation, but rather the whole network before the CRF in the proposed method), it is therefore difficult to say whether the performance improvement is due to the ResNet101 + FCN unary potentials, which is not a contribution of this manuscript, or to the proposed CRF. In general, I believe that the considerable speed gain of the proposed method might be enough to justify a publication, but the paper should be phrased in that sense if that was the intention of the authors. It is unclear to me whether the main contribution they claim is segmentation performance (IoU) or speed or both. The main contributions of this work should be stated clearly, and the modelling differences w.r.t. the FullCRF model that they aim to improve should be more explicit in the text rather than let to the reader to infer comparing the formulas.
On these grounds, I suggest a major revision of the paper and I don't recommend publication at this stage.
MAJOR
1) I firmly advocate against making strong claims, unless supported by solid proofs. I strongly recommend to rephrase, if not remove, exaggerate claims such as:
a) "[deep networks] lack the capability to utilize context information and cannot model interactions between predictions directly". This is simply not true. Any CNN with enough layers will exploit contextual information. Furthermore, any autoregressive model will model the interaction between predictions directly. See e.g., "RiFCN: Recurrent Network in Fully Convolutional Network for Semantic Segmentation of High Resolution Remote Sensing Images" by Mou et Al., "ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation" by Visin et Al., or "Predicting Deeper into the Future of Semantic Segmentation" by Luc et Al. for video semantic segmentation.
b) "CRF inference is two orders of magnitude slower than CNN inference": this, of course, depends on the kind of CRF.
c) "The long training times of the current generation of CRFs also make more in-depth research and experiments with such structured models impractical": again, not true. While it's true that CRFs tend to be slow, research with such models is not impractical and indeed there are papers that focus exactly on that (among the others, some of the ones cited in this manuscript).
d) "we propose to add the strong and valid assumption of conditional independence": as with every assumption, this is an approximation. I wouldn't claim it to be valid nor invalid, as it is simply a modeling choice.
e) "Predictions are pixel-wise and conditionally independendent (given the common feature base of nearby pixels). Structured knowledge and background context is ignored in these models.". It's unclear what is meant by "structured knowledge", but to my best understanding this sentence is misleading or wrong. Background context is considered by CNN-based models, as well as the general structure in the image.
f) "In the context of semantic segmentation most CRF based approaches are based on the Fully Connected CRF". CRFs have been used much earlier than 2011, before Fully Connected CRF was published.
g) "This makes the theoretical foundation of ConvCRF very promising, strong and valid assumptions are the powerhouse of machine learning modelling." The authors here propose a logical, but quite obvious, approximation, i.e., to constrain the CRF to model dependencies in a local neighborhood. This is the same implicit assumption of many other Gaussian kernels and algorithms for approximated inference. For how it's surely valid, and possibly strong, I don't see how it can make a "theoretical foundation". The self-congratulatory closure is unnecessary, and inappropriate.
2) While CRFs have been used for a long time on visual data, the citations in this work focus mostly on the last few years. I suggest to add at least one of the following:
* “Discriminative fields for modeling spatial dependencies in natural images" 2003
* "Multiscale conditional random fields for image labelling" 2004
* “Textonboost: Joint appearance, shape and context modeling for mulit-class object recognition and segmentation,” 2006
It could also be beneficial to the reader to add to the references broad overview works, such as "An Introduction to Conditional Random Fields" by Charles Sutton and Andrew McCallum, 2011, and/or "Structured prediction and learning in computer vision" by Nowozin and Lampert, 2011.
3) Line 5 of the algorithm adds the unary potentials at each iteration of message passing. Can you elaborate on the motivation behind this choice? The algorithm is already initialized with such potentials, and to my best knowledge unary potentials are not usually added in the mean field message passing loop. It would be interesting to compare the performance of the algorithm with and without this addition.
4) Sec4.1 reports that the filters are constant over the channel dimension c and that, in other words, this can be seen as applying the convolution over the dimension c. I fail to understand this sentence. My understanding from the formula is that the same kernel is applied to all the channels of the input, i.e., the channels of the input fed to the CRF are all processed in the same way. This should be explained clearly and the reasoning behind this choice should be explained as well. Furthermore, if I am not mistaken the authors learn a different filter in each position *of the input* rather than reusing the same filters at every position. This choice should be clarified and discussed.
5) Regarding the implementation, is there a reason not to apply a convolution with a flipped kernel to compute the cross-correlation? Also, IIRC if the kernel is symmetric (as should be for a Gaussian kernel) convolution and cross-correlation are the same.
6) The notation is often ambiguous and at times unnecessarely heavy. I strongly recommend to go over the manuscript and use a consistent notation, making sure that every element of the notation is introduced before or right after it's used. In particular,
a) Sec3: in the text, a segmentation instance is referred to as X, while in the formula as \hat x. \tilde I is never introduced. k_{\alpha} is defined but I believe never used (I suggest to drop the name if there is no reference to it). Is there a reason to drop the subscript G in the FullCRF pairwise potential? Or conversely, is there a reason to have it everywhere else? Furthermore, there doesn't seem to be a difference between k_G, k_g and g, I suggest to use only one consistent notation to refer to the Gaussian kernels throughout the paper. It's also unclear if the I superscript is needed for the feature vectors.
b) Sec4.1, The shape of the Gaussian kernels is the same as that of the input. I believe that the input in this context refers to a patch and not to the whole image. If so, this should be specified, otherwise the dimensions of the kernel should be referred to with a different letter than those of the input.
c) Sec4.1, I believe dx and dy refer to the in-kernel displacement. Their semantic is not clear from the text and should be defined properly.
d) Sec4.1, the feature vectors are defined in the text as f_1..f_d. The formula of the kernel uses f_i^{(d)} instead. It's unclear what the superscripts stands for and whether it is actually useful or redundant.
e) Sec4.1, x and y are not defined, I suspect they refer to the position of the pixel, which was previously encoded as p_i and p_j. Once again, the notation should be consistent across the manuscript.
f) In the definition of the convCRFs, w is used for the width of the input, w_i for the weights of the kernels. In the FullCRF, w^{(1)} and w^{(2)} for the potentials, w^{(m)} for the sum over the kernels. k is used for the kernel dimension, k_G to refer to the kernel itself, as well as g. The notation could be made less ambiguous and consistent (superscript vs subscript semantics).
g) Sec3, the number of pixel is defined as n but in Formula 2 N is used instead.
g) Vectors and matrices should be bold-face. The use of capital letters for constants might also improve the readability of the manuscript.
7) The experiments with the Conv (ConvCRF?) variants of Table 2 are not discussed in the text.
8) Although Sec5.2 concludes with "The experiments also confirm the observation of Sec5.1, that ConvCRF performs slightly better than FullCRF", Sec5.1 reported that "it can be seen that ConvCRFs outperform FullCRFs significantly". The authors should decide whether the results are slightly better or outperform the baseline. In general a in-depth discussion on the performance of the algorithm is missing.
9) Sec5.3, it's unclear what this sentence means "introduce an auxiliary unary loss to counterbalance the vanishing gradient problem". If such a term has been added, it should be reported in a formula and it's effectiveness should be supported by experimental data.
MINOR
m1) In the related work, the sentence "transposed convolution layers are applied at the end of the prediction pipeline ot produce high-resolution output" seems to suggest these are always applied, while many recent methods rely on bilinear upsampling to recover the original resolution. Please rephrase it accordingly.
m2) In Parameter learning in CRF: "the idea utilizes, that for the message passing the identity .. is valid." This sentence doesn't make any sense to me. Is it possible it is a leftover?
m4) In Sec3, the features vectors [...] may depend on the input image I. I am confused as to when they might be independent of the image. Can you elaborate on that?
m5) In Sec3, it's unclear to me what the vertical bar in the Pot model stands for. I believe the correct formula should be 1_{[xi != xj]}.
m6) In mean field inference, Algorithm 1 does not refer to FullCRFs.
m7) End of page 6, "Note that this gives FullCRFs a natural advantage. The performance of CRFs however is very robust [...]". Why is this an advantage for FullCRFs? How does that relate to the following sentence?
m8) Sec4.1, the authors claim that one of the key contribution of the paper is that exact message passing is efficient. Given the locality assumption, message passing is approximate - which is also why it's efficient. The authors could instead argue that using convolutions is faster and possibly leads to better final performance than using the permutohedral lattice approximation (although it's unclear whether this is the case from the experiments), with proper reference to compelling results in this direction.
Finally, a few typos:
* Abstract, space missing after GPUs
* Introduction, Convolutional Neuronal -> Convolutional Neural
* Introduction, order of magnitude slower then -> than
* Introduction, to slow -> too slow
* Parameter learning in CRF: missing space before proposed to use gradient descent
* Parameter learning in CRF: gradient decent -> descent
* Parameter learning in CRF: extra comma after "another advantage of this method is"
* Sec 3: "weighted sum of Gaussian kernels", the apex of the second should be "M" I believe.
* Sec 3: "can be chosen arbitrary" -> arbitrarily
* Sec 5.2, beginning of page 8: then -> than


Determinantal Point Processes (DPPs) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items. For this reason, they have gained prominence in many machine learning applications that rely on subset selection. However, sampling from a DPP over a ground set of size N is a costly operation, requiring in general an O(N 3 ) preprocessing cost and an O(N k 3 ) sampling cost for subsets of size k. We approach this problem by introducing DPPNETs: generative deep models that produce DPP-like samples for arbitrary ground sets. We develop an inhibitive attention mechanism based on transformer networks that captures a notion of dissimilarity between feature vectors. We show theoretically that such an approximation is sensible as it maintains the guarantees of inhibition or dissimilarity that makes DPPs so powerful and unique. Empirically, we demonstrate that samples from our model receive high likelihood under the more expensive DPP alternative.

This paper proposes a scaleable algorithm for sampling from DppNets, a proposed model which approximates the distribution of a DPP. The approach builds upon a proposed inhibitive attention mechanism and transformer networks.
The proposed approach and focus on sampling is original as far as I can tell. The problem is also important to parts of the community as DPPs (or similar distributions) are used more and more frequently. However, the applicability of the proposed approach is limited as it is unclear how to deal with varying ground set sizes — the authors briefly discuss this issue in their conclusion referring to circumvent this problem by subsampling (this can however be problematic either requiring to sample from a DPP or incurring high probability of missing „important“ items).
Furthermore the used evaluation method is „biased“ in favor of DppNets as numerical results evaluate the likelihood of samples under the DPP which the DppNet is trained to approximate for. This makes it difficult to draw conclusions from the presented results. I understand that this evaluation is used as there is no standard way of measuring diversity of a subset of items, but it is also clear that „no“ baseline can be competitive. One possibility to overcome this bias would be to consider a downstream task and evaluate performance on that task.
Furthermore, I suggest to make certain aspects of the paper more explicit and provide additional details. For instance, I would suggest to spell out a training algorithm, provide equations for the training criterion and the evaluation criterion. Please comment on the cost of training (constantly computing the marginal probabilities for training should be quite expensive) and the convergence of the training (maybe show a training curve; this would be interesting in the light of Theorem 1 and Corollary 1).
Certain parts of the paper are unclear or details are missing:
* Table 3: What is „DPP Gao“?
* How are results for k-medoids computed (including the standard error)? Are these results obtained by computing multiple k-medoids solutions with differing initial conditions?
* In the paper you say: „Furthermore, greedily sampling the mode from the DPPNET achieves a better NLL than DPP samples themselves.“ What are the implications of this? What is the NLL of an (approximate) mode of the original DPP? Is the statement that you want to make, that the greedy approximation works well?


SPIGAN: PRIVILEGED ADVERSARIAL LEARNING FROM SIMULATION
Deep Learning for Computer Vision depends mainly on the source of supervision. Photo-realistic simulators can generate large-scale automatically labeled synthetic data, but introduce a domain gap negatively impacting performance. We propose a new unsupervised domain adaptation algorithm, called SPIGAN, relying on Simulator Privileged Information (PI) and Generative Adversarial Networks (GAN). We use internal data from the simulator as PI during the training of a target task network. We experimentally evaluate our approach on semantic segmentation. We train the networks on real-world Cityscapes and Vistas datasets, using only unlabeled real-world images and synthetic labeled data with z-buffer (depth) PI from the SYNTHIA dataset. Our method improves over no adaptation and state-of-theart unsupervised domain adaptation techniques.

This papers presents an unsupervised domain adaptation algorithm for semantic segmentation. A generative adversarial network is envisaged to carry out synthetic-to-real image translation. In doing so, depth information extracted from a simulator is used as privileged information (PI) to boost the transfer on the target domain, regularizing the model and ensuring a better generalization.
*Quality*
The paper addresses a relevant problem, which is the adaptation of methods from simulated data to real ones. The authors devise a convincing method which takes advantage of state-of-the art generative adversarial architectures and privileged information.
*Clarity*
The paper is sufficiently well written. In general, the main idea and proposed method are clear and easy to follow. The only problem is that some background concepts (such as privileged information or unsupervised domain adaptation) are given for granted, compromising the readability for someone not familiar with those topics.
On a more technical side, for reproducibility purposes, the following aspects have to be clarified:
1.	Details about the validation set used for grid search. Is the validation set extracted from the target domain? (In principled labels from the target domain should not be used during learning).
2.	Number of iterations before convergence: is the training of the network numerically stable? Are there issues in convergence of some of the sub-modules? Which one is leading the learning?
3.	Comments about the relative magnitudes of losses. This will maybe give some intuitions about the values used for the hyper-parameters (e.g., the L_PI is only weighted by 0.1).
*Originality*
The way authors take advantage of depth information extracted from a simulator as privileged information is novel in the sense that, with respect to the original student-teacher paradigm of the paper by Vapnik & Vashist, here the idea of privileged information is interpreted as a regularizer to boost the training stage.
*Significance*
The application of semantic segmentation in urban scenes for navigation tasks is relevant. The scored results are on pair with/ superior to state-of-the-art in unsupervised domain adaptation.
However, the ablation study could be more extensive in order to understand the contribution of the several components, besides the PI network. In fact, it would be interesting to analyze the contribution of the perceptual loss (and others). Also, one could include the target-only result (as done in original LSD paper) to provide an upper bound on the best accuracy that is achievable.
*Pros*
1. The applicative setting of semantic segmentation in urban scenes for navigation is relevant.
2. Using privileged information from simulators seems novel and well presented in this paper.
3. Strong experimental results achieved in challenging benchmarks.
*Cons*
1. The regularization effect of the PI network could be supported by a more extensive ablation study of the model, for example by ablating the several losses used (in particular, the perceptual loss).
2. A quite relevant amount of hyper-parameters need to be cross-validated. Is the method robust against different parameters’ configuration?
3. Missing citations [1, 2]: there are works in the literature that can hallucinate a missing modality during testing. Although such works approach a different problem, authors should cite them.
[1] Judy Hoffman, Saurabh Gupta, Trevor Darrell - Learning with Side Information through Modality Hallucination – CVPR 2016
[2] Nuno Garcia, Pietro Morerio, Vittorio Murino - Modality Distillation with Multiple Stream Networks for Action Recognition – ECCV 2018
*Final Evaluation*
The authors face the challenging synthetic-to-real adaptation setup, with an interesting usage of z-buffer from a simulator as privileged information. Overall, the work is fine, apart from the following points.
1.	In addition to a few missing citations [1, 2], an ablation study on the perceptual loss is necessary to dissect the impact of each component of the pipeline.
2.	The clarity of the paper can be improved by adding some background material on unsupervised domain adaptation and learning with privileged information (PI), as to better highlight the technical novelty of using PI within a L1 regularizer.
3.	The training stage of all submodules could have better investigated, for instance, by providing some convergence plots of the loss functions across iterations.
4.	How to do grid search for parameters in a domain adaptation setting is always a delicate aspect and authors seem elusive on that respect.
5.	Again about hyper-parameters. Due to their high number, some sensitivity analysis should have provided.
As it is, the paper’s strengths slightly outperform the weaknesses, leading to an overall borderline-accept. If authors implement the suggested modification, a full acceptance will be feasible.
[COMMENTS AFTER AUTHORS' RESPONSE]
After the rebuttal provided by authors, all raised questions and criticisms have been fully solved. Therefore, I recommend for a full acceptance.


Deep neural networks have demonstrated cutting edge performance on various tasks including classification. However, it is well known that adversarially designed imperceptible perturbation of the input can mislead advanced classifiers. In this paper, Permutation Phase Defense (PPD), is proposed as a novel method to resist adversarial attacks. PPD combines random permutation of the image with phase component of its Fourier transform. The basic idea behind this approach is to turn adversarial defense problems analogously into symmetric cryptography, which relies solely on safekeeping of the keys for security. In PPD, safe keeping of the selected permutation ensures effectiveness against adversarial attacks. Testing PPD on MNIST and CIFAR-10 datasets yielded state-of-the-art robustness against the most powerful adversarial attacks currently available.

This paper explores the idea of utilizing a secret random permutation in the Fourier phase domain to defense against adversarial examples. The idea is drawn from cryptography, where the random permutation is treated as a secret key that the adversarial does not have access to. This setting has practical limitations, but is plausible in theory.
While the defense technique is certainly novel and inspired, its use case seems limited to simple datasets such as MNIST. The permuted phase component does not admit weight sharing and invariances exploited by convolutional networks, which results in severely hindered clean accuracy -- only 96% on MNIST and 45% on CIFAR-10 for a single model. While the security of a model against adversarial attacks is important, a defense should not sacrifice clean accuracy to such an extent. For this weakness, I recommend rejection but encourage the authors to continue exploring in this direction for a more suitable scheme that does not compromise clean accuracy.
Pros:
- Novel defense technique against very challenging white-box attacks.
- Sound threat model drawn from traditional security.
- Clearly written.
Cons:
- Poor clean accuracy makes the technique very impractical.
- Insufficient baselines. While the permutation is kept as a secret, it is plausible that the adversary may attempt to learn the transformation when given enough input-output pairs. Also, the adversary may attack an ensemble of PPD models for different random permutations (i.e. expectation over random permutations). The authors should introduce an appropriate threat model and evaluate this defense against plausible attacks under that threat model.


Conditional generation refers to the process of sampling from an unknown distribution conditioned on semantics of the data. This can be achieved by augmenting the generative model with the desired semantic labels, albeit it is not straightforward in an unsupervised setting where the semantic label of every data sample is unknown. In this paper, we address this issue by proposing a method that can generate samples conditioned on the properties of a latent distribution engineered in accordance with a certain data prior. In particular, a latent space inversion network is trained in tandem with a generative adversarial network such that the modal properties of the latent space distribution are induced in the data generating distribution. We demonstrate that our model, despite being fully unsupervised, is effective in learning meaningful representations through its mode matching property. We validate our method on multiple unsupervised tasks such as conditional generation, dataset attribute discovery and inference using three real world image datasets namely MNIST, CIFAR-10 and CelebA and show that the results are comparable to the state-of-the-art methods.

This paper is concerned with the so-called conditional generation, which was descried as the task of sampling from an unknown distribution conditioned on semantics of the data. This proposed method aims at achieving this by using a latent distribution engineered according to a certain data prior. Experimental results showed that the proposed method seems to produce good results.
I have several questions about the motivation and the method in the paper. First, it is not clear to me how the "semantics" of the data was defined. Is it given by visual inspection? Is it possible to find it with some automated method? Second, the authors seem to advocate the idea that data of a k-mode distribution should be generated from a k-mode latent distribution. It might be useful in certain scenarios; however, it is not clear why the transformation from the latent to the observed does not change the number of modes or why keeping the same number of modes would endow the latent distribution a "semantics" meaning. We know that a k-mode distribution can be obtained by applying a smooth nonlinear transformation to a Gaussian or uniform distribution and, similarly, a k-mode distribution can be transformed to a single-mode distribution with a smooth mapping. So I am not sure why engineering the latent distribution this way can give it a "semantics" meaning. Should we try to enforce a kind of smoothness of the transformation, by, say, penalizing high derivative values? Third, the experimental results seem nice, but the lack of comparisons blurs the advantage of the proposed method. How is the result produced by GAN compared to the reported one? How did the original GAN with the engineering latent distribution work? It would be appreciated if the authors could address these issues more clearly.


The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the 'edge of chaos' can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, φ swish (x) = x · sigmoid(x), used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of φ swish observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.

The authors prove some theoretical results under the mean field regime and support their conclusions with a small number of experiments. Their central argument is that a correlation curve that leads to sub-exponential correlation convergence (edge of chaos) can still lead to rapid convergence if the rate is e.g. quadratic. They show that this is the case for ReLU and argue that we must ensure not only sub-exponential convergence, but also have a correlation curve that is close to the identity everywhere. They suggest activation functions that attain conditions as laid out in propositions 4/5 as an alternative.
The paper has many flaws:
- the value of the theoretical results is unclear
- the paper contains many statements that are either incorrect or overly sweeping
- the experimental setup and results are questionnable
Theoretical results:
**Proposition 1: pretty trivial, not much value in itself
**Proposition 2: Pretty obvious to the experienced reader, but nonetheless a valuable if narrow result.
**Proposition 3: Interesting if narrow result. Unfortunately, it is not clear what the ultimate takeaway is. Is quadratic correlation convergence "fast"? Is it "slow"? Are you implying that we should find activation function where at EOC convergence is slower than quadratic? Do those activation functions exist? It would be good to compare this result against similar results for other activation functions. For example, do swish / SeLU etc. have a convergence rate that is less than quadratic?
**Proposition 4: The conditions of proposition 4 are highly technical. It is not clear how one should go about verifying these conditions for an arbitrary activation function, let alone how one could generate new activation functions that satisfy these conditions. In fact, for an arbitrary nonlinearity, verifying the conditions of proposition 4 seems harder than verifying f(x) - x \approx 0 directly. Hence, proposition 4 has little to no value. Further, it is not even clear whether f(x) - x \approx 0 is actually desirable. For example, the activation function phi(x)=x achieves f(x) = x. But does that mean the identity is a good activation function for deep networks? Clearly not.
**Proposition 5: The conditions of prop 5 are somewhat simpler than those of prop 4, but since we cannot eliminate the complicated condition (ii) from prop 4, it doesn't help much.
**Proposition 6: True, but the fact that we have f(x) - x \approx 0 for swish when q is small is kind of obvious. When q is small, \phi_swish(x) \approx 0.5x, and so swish is approximately linear and so its correlation curve is approximately the identity. We don't need to take a detour via propposition 4 to realize this.
Presentation issues:
- While I understand the point figures 1, 2 and 4b are trying to make, I don't understand what those figures actually depict. They are insufficiently labeled. For example, what does each axis represent?
- You claim that for ReLU, EOC = {(0,\sqrt{2})}. But this is not true. By definition 2, EOC is a subset of D_\phi,var. But {(0,\sqrt{2})} is not in D_\phi,var, because it simply leaves all variances unchanged and does not cause them to converge to a single value. You acknowledge this by saying "For this class of activation functions, we see (Proposition 2) that the variance is unchanged (qal = qa1) on the EOC, so that q does not formally exist in the sense that the limit of qal depends on a. However,this does not impact the analysis of the correlations." Section 2 is full of complicated definitions and technical results. If you expect the reader to plow through them all, then you should really stick to those definitions from then on. Declaring that it's fine to ignore your own definitions at the beginning of the very next section is bad presentation. This problem becomes even worse in section 3.2, where it is not clear which definition is actually used for EOC in your main result (prop 4), making prop 4 even harder to parse than it already is.
Correctness issues:
- "In this chaotic regime, it has been observed in Schoenholz et al. (2017) that the correlations converge to some random value c < 1" Actually, the correlation converges deterministically, so c is not random.
- "This means that very close inputs (in terms of correlation) lead to very different outputs. Therefore, in the chaotic phase, the output function of the neural network is non-continuous everywhere." Actually, the function computed by a plain tanh network is continuous everywhere. I think you mean something like "the output can change drastically under small changes to the input". But this concept is not the same as discontinuity, which has an established formal definition.
- "In unreported experiments, we observed that numerical convergence towards 1 for l ≥ 50 on the EOC." Covergence of a sequence is a property of the limit of the sequence, and not of the 50th element. This statement makes no sense. Also if you give a subjective interpretation of those experimental results, you should present the actual results first.
- "Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions." This statement is very vague and sweeping. Also, one could argue that the fact that ReLU is much more popular and tends to give better results than tanh in practice disproves the statement outright.
- "Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions. However, these functions suffer from the vanishing gradient problem during back-propagation" At the edge of chaos, vanishing gradients are impossible! As Schoenholz showed, at the edge of chaos, \chi_1=1, but \chi_1 is also the rate of growth of the gradient. Pascanu et al (2013) discussed vanishing gradients in RNNs, which is a different story.
- "Other activation functions that have been shown to outperform empirically ReLU such as ELU (Clevert et al. (2016)), SELU (Klambauer et al. (2017)) and Softplus also satisfy the conditions of Proposition 4 (see Supplementary Material for ELU)." Firstly, SeLU does not satisfy proposition 4. f(x) \approx x requires \phi to be close to a linear function in the range where the pre-activations occur. Since SeLU has a kink at 0, it cannot be close to a linear function no matter how small the pre-activations are. Secondly, softplus also doesn't satisfy proposition 4, as \phi(0) = 0 does not hold. Thirdly, this statement is too sweeping. If ELU / SELU / Softplus "outperform" ReLU, why is ReLU still used in practice? At best, those nonlinearities have been shown to outperform in a few scenarios.
- "We proved in Section 3.2 that the Tanh activation guarantees better information propagation through the network when initialized on the EOC." Prop 4 only applies in the limit as \sigma_b converges to 0. So you can't claim that you showed tanh as "better information propagation" in general.
- "However, for deeper networks (L ≥ 40), Tanh is stuck at a very low test accuracy, this is due to the fact that a lot of parameters remain essentially unchanged because the gradient is very small." But in figure 6b the accuracy for tanh is decreasing rapidly, so therefore the parameters are not remaining "essentially unchanged", as this would also cause the accuracy to remain essentially unchanged. Also, if the parameter changes are too small ... why not increase the learning rate?
- "To obtain much richer priors, our results indicate that we need to select not only parameters (σb , σw ) on the EOC but also an activation function satisfying Proposition 4." Prop 4 only applies when \sigma_b is small, so you additionally need to make sure \sigma_b small.
- "In the ordered phase, we know that the output converges exponentially to a fixed value (same value for all Xi), thus a small change in w and b will not change significantly the value of the loss function, therefore the gradient is approximately zero and the gradient descent algorithm will be stuck around the initial value." But you are using Adam, not gradient descent! Adam explicitly corrects for this kind of gradient vanishing, so a small gradient can't be the reason for the lack of training success.
Experimental issues:
- "We use the Adam optimizer with learning rate lr = 0.001." You must tune the learning rate independently for each architecture for an ubiased comparison.
- In figure 6b, why does tanh start with a high accuracy and end up with a low accuracy? I've never seen a training curve like this ... This suggests something is wrong with your setup.
- You should run more experiments with a larger variety of activation functions.
Minor comments:
- "Therefore, it is easy to see that for any (σb , σw ) such that F is increasing and admits at least one fixed point,wehaveKφ,corr(σb,σw) ≥ qwhereqistheminimalfixedpoint;i.e. q := min{x : F(x) = x}." I believe this statement is true, but I also think it requires more justification.
- At the end of page 3, I think \epsilon_r should be \epsilon_q
There are some good ideas here, but they need to be developed/refined/polished much further before publication. The above (non-exhaustive) list of issues will hopefully be helpful for this.
### Addendum ###
After an in-depth discussion with the authors (see below), my opinion on the paper has not changed. All of my major criticisms remain: (1) There are far easier ways of achieving f(x) ~ x than propositions 4/5/7, i.e. we simply have to choose \phi(x) approximately linear. (2) The experiments are too narrow, and learning rates are badly chosen. (3) The authors do not discuss the fact that as f(x) gets too close to x, performance actually degrades as \phi(x) gets too close to a linear function. (Many other criticisms also remain.)
The one criticism that the authors disputed until the end of the discussion is criticism (1). Their argument seems to hinge on the fact that their paper provides a path to construct activation function that avoid "structural vanishing gradients", which they claim 'tanh' suffers from. While they acknowledge that tanh does not necessarily suffer from "regular" vanishing gradients (as shown by "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice" and "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks"), they claim it suffers from structural vanishing gradients. I do not believe that there is such a thing as structural vanishing gradients. However, even if such a concept did exist, it falls on the the authors to provide a clear definition / explanation, which they neither do in the paper nor the rebuttal.