Adaptive moment methods have been remarkably successful for optimization under the presence of high dimensional or sparse gradients, in parallel to this, adaptive sampling probabilities for SGD have allowed optimizers to improve convergence rates by prioritizing examples to learn efficiently. Numerous applications in the past have implicitly combined adaptive moment methods with adaptive probabilities yet the theoretical guarantees of such procedures have not been explored. We formalize double adaptive stochastic gradient methods DASGRAD as an optimization technique and analyze its convergence improvements in a stochastic convex optimization setting, we provide empirical validation of our findings with convex and non convex objectives. We observe that the benefits of the method increase with the model complexity and variability of the gradients, and we explore the resulting utility in extensions to transfer learning.

Summary:
This paper explores the combination of adaptive moment methods and adaptive data sampling procedures for stochastic optimization. The paper presents standard analysis in the stochastic convex setting which raised some question for me (presented below). The empirical results are promising but in my opinion can be further improved through minor changes to experimental procedure.
Overall:
1) There are a few instances of undefined notation. For example, $\beta_1(t)$ in the definition of Adam/AMSGrad or $V_t$ (the latter I assume is the average over all datapoints?).
2) Theorem 1 is stated very informally with some notation left unexplained (e.g. V_t above or E_n[...]). Further, it is not clear to me whether the authors are proving results in the online convex optimization setting or the stochastic convex optimization setting (though, there is some equivalence between them). Finally, I am concerned by the presentation of Theorem 1 and Corollary 1.1 as it has been shown by Reddi et al. that Adam does not have bounded regret in general without algorithmic changes. In particular, bimodal gradient noise distributions can be designed such that the average Regret is unbounded (see Reddi et al. for details). While these issues can be addressed through modifications to the standard Adam algorithm I am unable to find a suitable discussion within the paper.
3) Theorem 2 introduces DASGrad which has not yet been clearly defined (Algorithm 1 depends on averaging functions of the past gradients for flexibility --- which averaging functions are being used here?). Once again, the notation Var_n appears whose definition is not clear to me. Additionally, could you provide a citation for the optimal adaptive probabilities and clearly state under which conditions these adaptive probabilities are optimal?
The crux of Theorem 2 depends on the informal statements present in Section 3.3, in which it is claimed that the variance of the second order moments are sufficiently large to dominate the sum of gradient histories. I am having a hard time seeing how this would be true generally and I believe that further theoretical analysis is necessary to justify this discussion.
4) The introduction of Algorithm 2 felt like a bait-and-switch. This algorithm seems intuitively sensible and uses the same check-pointing ideas that have been introduced in recent variance reduction methods (e.g. [1]). However, this is a different algorithm which is not analyzed in this paper.
Additionally, the fourth line of Algorithm 2 says to compute p_t lying in the simplex --- I believe this is a typo? Further, could you explain the introduction of epsilon in line 4. This seems unnecessary given the propto symbol but I am worried I am missing something.
5) The empirical results introduce an additional deviation from the proposed algorithm by using batch sizes of 32. Could the authors please explain how the adaptive sampling probabilities are used in this setting? (My apologies if this is obvious, I am not overly familiar with such methods).
Overall I felt that the empirical results were fairly promising but some questions remained for me. The toy experiments in section 6.1 were illustrative but seem to only consider adaptive probabilities without the adaptive moments?
Section 6.3 was also an interesting experimental setup which is well suited to DASGrad but I felt that the baseline comparison was unfair. DASGrad was provided with the ground-truth balance statistics in the importance weights while the baselines were trained without any correction. I would recommend that the authors explore combinations of enabling balance correction in the baseline methods (possibly through reweighting the loss according to the balance statistics) or requiring DASGrad to compute the optimal probabilities through the adaptive moments as suggested throughout the rest of the paper.
Minor:
- Pg 3, typo "we first we extend"
References
[1] Accelerating Stochastic Gradient Descent using Predictive Variance Reduction, Rie Johnson and Tong Zhang


Adversarial training is a principled approach for training robust neural networks. Despite of tremendous successes in practice, its theoretical properties still remain largely unexplored. In this paper, we provide new theoretical insights of gradient descent based adversarial training by studying its computational properties, specifically on its implicit bias. We take the binary classification task on linearly separable data as an illustrative example, where the loss asymptotically attains its infimum as the parameter diverges to infinity along certain directions. Specifically, we show that for any fixed iteration T , when the adversarial perturbation during training has proper bounded 2 -norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum 2 -norm margin classifier at the rate of O(1/ √ T ), significantly faster than the rate O (1/ log T ) of training with clean data. In addition, when the adversarial perturbation during training has bounded q -norm with q ≥ 1, the resulting classifier converges in direction to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum 2 -norm margin classifier under worst-case q -norm perturbation to the data. Our findings provide theoretical backups for adversarial training that it indeed promotes robustness against adversarial perturbation.

**Contributions:**
This paper extends results on the implicit bias of gradient descent (GD) Soudry et al. (2018)[3] for the case of gradient descent based adversarial training (GDAT).
**1) Theoretical result for L2 norm:** Convergence of standard risk in GDAT is significantly faster than its counterpart in the standard clean training using GD: For any fixed iteration T, when the adversarial perturbation during training has bounded l2-norm, the classifier learned by GDAT converges to the maximum l2-norm margin classifier at the rate of O(1/√T), exponentially faster than the rate O (1/ log T ) obtained when training with only cleaned data Soudry et al. (2018) [3].
**2) theoretical result for Lq norm:** When the adversarial perturbation during training has bounded l_q-norm with q > 1, with a proper choice of c, the gradient descent based adversarial training is directionally convergent to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum $l_2$-norm margin classifier under worst-case $l_q$-norm perturbation to the data.
The paper is well written and easy to understand. I haven’t checked the proofs, but I focused on readability and motivations.
I have the following questions.
**Theoretical questions:**
- Can you please explain the intuition of lemma 3.2 and 3.3 for convergence?
**Experiments questions:**
Keeping in mind that the core of the paper is about proving that GDAT enjoys the same implicit bias as GD and better convergence performance on clean data, I can’t help myself by asking you how these properties reflect in experiments with deep Neural Networks:
    - Can you please clarify your motivation for the reduction of the performance gap when the with of the hidden layer increases (Figure 2)? It is not clear what do you mean by ...“as network width increases, the margin on the samples outputted by the hidden layer also increases”...
My personal interpretation of the results is that the inner maximization problem is much harder to solve and being approximated, the effect of the GDAT is much less prominent, meaning that PGD is not enough to benefit from adversarial training in this sense. What do you think about it?
    - Take away message of the paper: adversarial training accelerates convergence. Theoretical results are for the training empirical loss.
Is this also the case of the validation loss? You observe this behavior for the case of a single MLP on MNIST binary problem 2 vs 9 (appendix E) claiming that adversarial training improves generalization performance.
This is not observed in the literature, it is actually the opposite [1, 2]: it exists a tension between robustness and performance on the clean test set. Why do you think you are not observing this? My guesses:
    - you are analysing a very simple binary problem
    - your model is very simple and comparable to a linear classifier, so your theoretical results hold.
[1][1805.12152 Robustness May Be at Odds with Accuracy](https://arxiv.org/abs/1805.12152)
[2][1810.12715 On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://arxiv.org/abs/1810.12715)
[3][1710.10345 The Implicit Bias of Gradient Descent on Separable Data](https://arxiv.org/abs/1710.10345)


Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with "t-shaped" adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VXXLXX STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the stateof-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.

Summary:
This paper proposes a deep learning method to produce "pans" of an input image. That is, simulated images of the scene from translated viewpoints. Unlike some previous work that considers only a fixed baseline (such as the 2nd view of a stereo camera), this approach allows generation of a range of views. A specially crafted convolutional architecture is shown to be well-suited to this problem. Results demonstrate visually pleasing image generation and low metric errors on several datasets.
Strengths:
- The justifications for the design choices in this paper, in particular the convolution structure and connection to image geometry, was quite strong compared to recent papers (although, many of the presented ideas are known in more classical, non-learning, techniques).
- All presented empirical results are impressive, and show the method is likely to "really work" and be reproducible, judging from the number of experiments where the method has consistently outperformed.
- The method is clear and straightforward to implement either on its own, or as a module/architecture within a larger pipeline.
Areas for Improvement and Detailed Suggestions:
- The problem of panned view generation is a bit more narrow than some authors are lately considering (generate any viewpoint including off-axis rotations).
- The t-shaped network architecture here is largely presented as only appropriate to handing image panning. Could a more general network be created, perhaps parameterized by the type of rigid motion occurring? Better, could more flexible networks be proposed with sparsity constraints that allow sub-patterns like the t-network to be learned in a data-driven fashion?
- Please try to cite referenced work more consistently. For several methods, such as Deep 3D Zoom Net, you have begun to discuss the method using name only for large stretches. It would be helpful to keep using the citation at least once per paragraph to remind the reader of the source of these ideas.
Decision and Justification:
Weak reject due to the lack of generality in the approach. I suspect the impact of this work will be a bit less than the very competitive bar for ICLR this year. However, I must note that I am the least expert in this area, out of any paper in my stack.


We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. We study the prevalence of local minima in loss landscapes, whether small-norm parameter vectors generalize better (and whether this explains the advantages of weight decay), whether widenetwork theories (like the neural tangent kernel) describe the behaviors of classifiers, and whether the rank of weight matrices can be linked to generalization and robustness in real-world networks.

The authors look at empirical properties of deep neural networks and discuss their connection to past theoretical work on the following issues:
* Local minima: they give an example of setting where bad local minima (far from the global minimum) are obtained. More specifically, they show such minima can be obtained by initializing with large random biases for MLPs with ReLU activation. They also provide a theoretical result that can be used to find a small set of such minima. I believe this is a useful incremental step towards a better understanding of local minima in deep learning, although it is not clear how many practical implications this has. One question that would ideally be answered is: in practical settings, to what degree does bad initialization cause bad performance specifically due to bad minima? (as opposed to, say, slow convergence or bad generalization performance).
* Weight decay: the authors penalize the size of the norm of the weights as it diverges from a constant, as opposed to when it diverges from 0 as is normally done for weight decay. They show that this works as well or better than normal weight decay in a number of settings. This seem to put into question the belief sometimes held that solutions with smaller norms will generalize better.
* Kernel theory: the authors try to reproduce some of the empirical properties predicted in the Neural Tangent Kernel paper (Jacot et al., 2018) in particular by using more realistic architectures. The results, however, do not appear very conclusive. This might be the weakest part of the paper, as it is hard to draw anything conclusive from their empirical results.
* Rank: The authors challenge the common belief that low rank provides better generalization and more robustness towards adversarial attacks. When enforcing a low or high rank weight matrices during training on ResNet-18 trained on CIFAR-10, the two settings have similar performance and are similarly robust to adversarial attacks, showing at least one counter example.
I think overall this is a useful although somewhat incremental paper, that makes progress in the understanding of the behavior of neural networks in practice, and can help guide further theoretical work and the  development of new and improved training techniques and initialization regimes for deep learning.
Other comments/notes:
* minor: the order of the last 2 sub topics covered (rank and NTK) is flipped in the introduction, compared to the abstract and the order of the chapters
* in the table confidence intervals are given, it would be nice to have more details on how they are computed, (e.g. +- 1.96 * std error)
* how is the constant \mu in the norm-bias chosen?


Auto-encoding and generative models have made tremendous successes in image and signal representation learning and generation. These models, however, generally employ the full Euclidean space or a bounded subset (such as [0, 1]   l ) as the latent space, whose flat geometry is often too simplistic to meaningfully reflect the topological structure of the data. This paper aims at exploring a universal geometric structure of the latent space for better data representation. Inspired by differential geometry, we propose a Chart Auto-Encoder (CAE), which captures the manifold structure of the data with multiple charts and transition functions among them. CAE translates the mathematical definition of manifold through parameterizing the entire data set as a collection of overlapping charts, creating local latent representations. These representations are an enhancement of the singlecharted latent space commonly employed in auto-encoding models, as they reflect the intrinsic structure of the manifold. Therefore, CAE achieves a more accurate approximation of data and generates realistic synthetic examples. We demonstrate the efficacy of CAEs through a series experiments with synthetic and real-life data which illustrate that CAEs can out-preform variational auto-encoders on reconstruction tasks while using much smaller latent spaces.

## Summary of the Paper
This paper introduces a new architecture for autoencoders based on the
concept of *charts* from (differential) topology. Instead of learning a
single latent representation, the paper proposes learning charts, which
serve as local latent representations. Experiments demonstrate that the
local representations perform favourably in terms of approximating  the
underlying manifold.
## Summary of the Review
This is an interesting paper with an original idea. I appreciate the use
of concepts from differential topology in deep learning and agree with
the paper that such a perspective is required to increase our
understanding of complicated manifold data sets. However, I find the
following issues with the paper in its current form, which prevent me
from endorsing it for acceptance:
1. I have doubts about the technical correctness of the proposed
   architecture; specifically, the relevance of the *initial* latent
   representation, which employs a Euclidean space, is not analysed.
2. The role of the number of charts, which needs to be specified
   before-hand, is not analysed in an ablation study.
3. The experiments do not showcase the *conceptual* improvements of the
   proposed technique.
I shall briefly comment on each of these points before discussing other
improvements.
I want to point out that I really like the ideas presented in this
paper and I think it has the potential to make a strong contribution but
the issues in their current form require substantial revisions and
additions.
# Concern 1: Technical correctness
The paper claims at multiple places that the geometry of Euclidean space
is 'trivial' or 'too simplistic' to meaningfully reflect the structure
of the data. This claim is double-edged, though: first, there are
many methods that use autoencoders based on these spaces that exhibit
sufficient reconstruction capabilities. Second, the proposed
architecture itself uses a Euclidean latent representation as its
initial encoder. The paper states that 'Ideally, this step preserves the
topology of the data [...]', but this is never analysed.
I fully agree with the idea that charts are a suitable way to describe
complicated manifolds, but the paper needs more precision when terms
such as 'topology' and 'geometry' are being used. Likewise, I disagree
with referring to Euclidean space as 'trivial'. Again, other methods
demonstrate that the space captures high-level phenomena sufficiently
well for reconstruction purposes. At the very least, the paper should
be more precise here.
Moreover, I would recommend experiments in which the dimensionality of
the initial encoder is discussed.
# Concern 2: Number of charts
Selecting the number of charts appears to me as a critical component of
the proposed method. While the appendix contains one experiment for
MNIST with different numbers of charts, this concept needs to be fleshed
out more. How do we know that we have a sufficient number of charts?
Since in differential topology, the choice of chart should not matter,
how does it behave in these cases? Is there a way to detect that the
number of charts must be increased?
I could envision something like a simple 'step size' control procedure:
if a quality measure indicates that there need to be more charts, double
the number of charts and re-run the training; if the number of charts
is too big, halve it and re-run the training.
I get the idea that increasing the number of charts will probably
decrease the reconstruction error, but this comes at the obvious expense
of even more parameters. I thus recommend another set of experiments
that shows the influence of the number of charts, maybe even on the
synthetic data sets used in the paper.
# Concern 3: Conceptual improvements
While I enjoyed the didactic approach of the paper, which first
introduces simple test data sets to illustrate the concepts, my
main question is about the conceptual improvements that the charts
provide in the end.
I see that the reconstruction error for MNIST goes down---but there are
also significantly more (!) parameters than in the comparison
architectures. The ideas of the sampling or interpolation experiments
go in the right direction, but in their present version, they are not
entirely convincing. In fact, they even raised more questions for me:
- Figure 6 depicts individual charts but their *covering* of the space
  is highly non-uniform. The letter '0' is covered more often than the
  letter '1', for example. How can this be compatible with the claim
  that the novel architecture learns a suitable set of charts? I could
  understand some overlaps, but there seems to be a clear difference
  between the charts generated in the synthetic examples---which do
  appear to be cover everything in a uniform manner---and the charts for
  MNIST. This needs to be elucidated some more, in particular since the
  paper writes that the charts 'cover [...] in a balanced and regular
  way'.
- The digit morphing example is not not entirely convincing to me. Is
  this not something that I can do equally well with a VAE or generative
  models in general? I am *not* disputing the claims of the paper here,
  I am merely stating that *if* the new method is beneficial for this
  sort of application, a more in-depth experiment is required.
Thus, while I would like to give the paper the benefit of the doubt, it
does not show just *why* it is relevant to have a chart-based embedding.
Some suggestions for a set of experiments:
- Do charts help in separating the input space? I would hypothesise
  that this is the case---it thus might be worthwhile to study
  low-dimensional embeddings obtained based on each chart and 'stitch'
  them together.
- Do charts tell us something about the properties of a manifold? For
  example, are certain charts 'easier' to embed than others? This could
  be used to indicate different dimensions in a data set.
## Experimental setup
I have one major point of critique here, namely the way results are
presented without any measures of tendency. Instead of showing a bar
plot in Figure 7, I would suggest showing a Table with standard
deviations along multiple repetitions of the experiment. It is not clear
from looking at this to what extent this results can be replicated.
Moreover, a discussion of the number of parameters is required. To some
extent, I find it not surprising that a better reconstruction error is
achieved if more parameters are present.
This makes some of the claims in the paper hard to assess.
## Technical clarity
The papers is generally written well and has a good expository style.
Here are some cases where I find that clarity can be improved:
- To add to what I wrote above: if charts are Euclidean as well, the
  paper should elucidate why Euclidean charts do *not* suffer from being
  too simplistic.
- The discussion of homeomorphisms in the introduction is slightly
  misleading; none of the functions learned later on is a homeomorphism
  because of the latent space dimensions.
- Homeomorphic mappings of manifolds into a Euclidean space are not
  necessarily desirable---this is why the definition of a manifold uses
  the concept of neighbourhoods. I think this should be rephrased in
  a positive manner, as in: manifolds are complex, so we cannot expect
  a *single* map to suffice...
- The leading example of a torus embedding needs more details. Why is
  the structure destroyed?
- The introduction of 'topological features' on p. 2 is slightly abrupt.
  It would be sufficient to explain by means of the figure that the
  mapping obviously does not respect all properties.
- p.2: paths become invariant to _what_ exactly?
- p.2: what is the 'topological class'?
- p.2: would LLE not be a good precursor to the method proposed in this
  paper?
- p.2: is this paper to be seen as an implementation of Chen et al.  (2019)?
  This should be made more clear.
- p.3: the concept of intrinsic dimension slightly varies in literature.
  I would propose mentioning the homeomorphism of every chart to some
  $d$-dimensional space, and state that if this exists, one calls the
  manifold $d$-dimensional.
- p.3: the circle example could be explained in more detail for readers
  unfamiliar with the concepts.
- p.4: the chart prediction module requires a brief explanation at the
  point when it is first introduced (1 sentence is sufficient). The
  method plus architecture is presented but the details come very late;
  I would prefer some intuition here
- p.4: $N$ needs to be defined earlier
- p.4: how is the dimension of latent spaces chosen? Please also refer
  to my comments on the experiments above.
- p.5: Section 4.1 again mixes 'topological' and 'geometrical' concepts;
  suddenly, the concept of curvature crops up---this needs to be
  explained better!
- p.5: Distances can always be measured in connected subsets of
  real-valued spaces; whether the set is open or closed does not change
  the fact that a centre exists. Am I misunderstanding this?
- p.5: I like the 'partition of unity' approach, but to me, this reads
  like a convex combination of predictions. Am I misreading this? If
  not, I would suggest to rephrase this.
- p.5/6: the goals of the new method need to be stated more clearly; the
  paper needs to explain better to what extent *reconstruction error* is
  affected by charts (it does not seem to be, as I outlined above)---and
  this again raises the question of which quality measure the new method
  *can* preserve.
- p.6: the definition of the Lipschitz constant could be more precise;
  please specify the requirements $f$ has to satisfy
- Eq. 4 needs more details for me: it seems as if the weights appear
  twice as a kind of 'decay term' (in the second part, I see the sum
  but the product appears in both terms). This should be stated more
  clearly.
- p.6: the pre-training needs more details; how crucial is this step?
- p.6: what does the 'orientation' imply? It is not defined except in the
  appendix.
- p.6: the jump from the illustrative examples to the non-synthetic ones
  is large; the uniform sampling of the latent space does not scale to
  higher dimensions, for example. The paper should comment on this if
  possible.
- In general, I would recommend giving the employed models more
  'speaking' names. I found it hard to keep track of all of them and had
  to refer to the appendix constantly.
- For Figure 4, please show the full space, together will all charts
- p.7: please give some ideas (see above) for how to use the covering of
  the points in practice; I like that the object can be reconstructed
  with a proper set of charts, but the paper could make the necessity
  of the technique much more obvious by choosing stronger examples.
- p.7: the object arguably *also* has a complex geometry, not only
  complex topology. This should be mentioned.
- p.8: the discussion of MNIST is slightly incorrect; as outlined above,
  many digits appear to be generated by multiple charts, while some,
  such as `1` do not appear on more than one chart.
- The metrics in Section 5.3 should be introduced earlier, maybe at the
  expense of some exposition in the introduction or the simpler
  examples; it is not good style to have to refer to the appendix to
  understand a core experiment of a paper.
- p.8: I do not understand the term 'wholly pyramid'.
- p.11: the decoder should map to $x_i$, if I am not mistaken
- p.11: I would suggest a more consistent terminology to describe the
  models. The prediction function is replicated multiple times, for
  example, so why not introduce a shorthand notation for this?
- p.12: '\cup' and '\cap' need to be switched: the *intersection* of
  domains needs to be empty, not their *union*
- p.13: to what extent are the 'faithfulness' and 'coverage' established
  metrics? It seems that they are developed for this paper, so I would
  explain them in the main text and also make clear why they are
  desirable metrics---else, the metrics could be criticised as being
  fine-tuned for the proposed method.
  For example, if *coverage* can measure the phenomenon of *mode
  collapse*, this needs to be demonstrated.
## Minor comments
Some typos:
- low dimensional --> low-dimensional
- eigen-functions --> eigenfunctions
- considers manifold point --> considers a manifold point
- paring subnetwork --> pairing subnetwork
- paramterized --> parametrized
- preformed --> performed [occurs multiple times]
- chats --> charts
- Lipshitz --> Lipschitz (in Figure 4)
- evalutation --> evaluation
- seciton --> section


The lack of crisp mathematical models that capture the structure of real-world data sets is a major obstacle to the detailed theoretical understanding of deep neural networks. Here, we first demonstrate the effect of structured data sets by experimentally comparing the dynamics and the performance of two-layer networks trained on two different data sets: (i) an unstructured synthetic data set containing random i.i.d. inputs, and (ii) a simple canonical data set containing MNIST images. Our analysis reveals two phenomena related to the dynamics of the networks and their ability to generalise that only appear when training on structured data sets. Second, we introduce a generative model for data sets, where high-dimensional inputs lie on a lower-dimensional manifold and have labels that depend only on their position within this manifold. We call it the hidden manifold model and we experimentally demonstrate that training networks on data sets drawn from this model reproduces both the phenomena seen during training on MNIST.

The paper studies how different settings of data structure affect learning of neural networks and how to mimic the behavior of neural networks seen on real datasets (e.g. MNIST) when learning on a synthetic one.
I would recommend rejecting the paper due to several issues pointed out below.
1. The paper abuses blanket citation making it difficult to identify and verify contribution and conduct a comparison with existing literature. Related work section amounts only to one paragraph in size. It looks questionable, that nobody ever treated the problem of the generalization ability of neural networks from a point of the data manifold properties. After a quick search, for example, [1] provides an in-depth analysis and generalization bounds for two-layered neural networks and provides a data complexity measure that can discriminate between random labels (which are equivalent to the outputs of randomly initialized fixed teacher networks in this work) and true labels on structured datasets like MNIST and CIFAR. The paper fails to cite this work as well.
2. The paper claims to experimentally identify key differences in the training dynamics of neural networks in teacher-student setup and on an MNIST task (binary classification into even and odd numbers).  One of the differences is the presence of plateaus in the learning curves in the vanilla teacher-student setup, however, as the paper states itself - this is a well established and studied characteristic of the setup, not something unexpected and new.
3. Overall, I have yet to see actionable development in this paper as it consists of observations that have been noticed and studied previously and presents no attempt at explanation or rigorous analysis.
As for the experiments:
The setting of the experiment in section 3.1 leaves space for improvement. For example, it would be interesting to see whether two neural networks learned in the vanilla teacher-student setup on the iid random inputs agree on the MNIST inputs (i.e. inverting the experiment in 3.1) as a sanity check for other factors interplay since MNIST inputs would be an example of the out of distribution inputs for the networks learned on iid random examples used in the experiment. As another pointer, [2] shows that even when training input is from a standard normal distribution, the problem can have spurious local minima, implying that even on unstructured training datasets, neural networks from different initializations yield diverse outputs for out of distribution inputs, not agreeing among each other.
I would also like to see concrete examples when and how the hidden manifold model may benefit theoretical understanding or practical knowledge on, for example, how to cook a dataset or check if the dataset admits/affects learning.
References
[1] Arora, Sanjeev, et al. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.
[2] Safran, Itay, and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks.


Reliably transmitting messages despite information loss due to a noisy channel is a core problem of information theory. One of the most important aspects of real world communication is that it may happen at varying levels of information transfer. The bandwidth-limited channel models this phenomenon. In this study we consider learning joint coding with the bandwidth-limited channel. Although, classical results suggest that it is asymptotically optimal to separate the sub-tasks of compression (source coding) and error correction (channel coding), it is well known that for finite block-length problems, and when there are restrictions to the computational complexity of coding, this optimality may not be achieved. Thus, we empirically compare the performance of joint and separate systems, and conclude that joint systems outperform their separate counterparts when coding is performed by flexible learnable function approximators such as neural networks. Specifically, we cast the joint communication problem as a variational learning problem. To facilitate this, we introduce a differentiable and computationally efficient version of the bandwidth-limited channel. We show that our design compensates for the loss of information by two mechanisms: (i) missing information is modelled by a prior model incorporated in the channel model, and (ii) sampling from the joint model is improved by auxiliary latent variables in the decoder. Experimental results justify the validity of our design decisions through improved distortion and FID scores.
to it's ubiquity as a fundamental component in the real world communication systems. The main contributions of this work include:
1. We cast the problem of learning joint communication as a variational learning problem, parallel to other work (Choi et al., 2019). 2. We justify the importance of jointly learned systems by empirically evaluating the gap between neural systems for joint and separate communication. 3. We design standard channels such as the Gaussian and Binary channel as differentiable probabilistic nodes, which serve as base for our design of the bandwidth-limited channel. 4. We investigate two core design choices of our neural joint model and bandwidth-limited channel:
(i) how transmission rate can be improved through learned prior models (ii) how we may improve image reconstructions in the low bandwidth regime through the introduction of auxiliary latent variables in the decoding process.

Paper Summary:
The paper proposes to use ML methods, specifically neural networks, to learn source and/or channel coding systems, either jointly or separately. Specifically, they investigate these systems under the bandwidth-limited channel.  They investigate their models applied to the task of transferring images across the channel.
Pros:
- The paper is in a difficult area, and needs to communicate ideas about communications and information as well as various deep learning models. The initial exposition does this well.
- The paper discusses ML-based communications models applied to the bandwidth limited channel and performs experiments to investigate joint vs. separate coding.
Two main things of concern:
- This area (communications, compression, coding) is highly developed, yet there are no comparisons to practical, state of the art techniques in this paper. The experiments investigate using the authors' models to send images across a noisy channel. So shouldn't we see a comparison to an equivalent non-ML-based system that currently accomplishes this task, if only for perspective?
- The paper contains too much exposition and the structure makes it difficult to read.  The authors' work seems to mostly be focused in sections 4 and 6, but prior work is summarized in section 5.  The authors want to communicate a lot of ideas which the audience might not be familiar with, and this is a difficult task. However, the paper could be reorganized such that the ideas are clearer. Rate distortion is delegated to an appendix, but two pages are spent on basic communications. Overall it could be much more focused.
Some smaller concerns:
- Pg. 3 "hypothesis spaces can be searched increasingly quickly in an automated fashion, allowing researchers to search..."
I think this statement may be over-reaching. The authors also frequently use the term "flexible function approximation". Neural networks do have interesting approximation properties. But this is not a complete picture of deep learning, and the situation is much more nuanced than this. Where is the role of optimization and data? In order to transmit data with the author's algorithms, do we need to go out and collect a large body of examples in the specific domain, like CelebA? Because you don't need to do that to e.g. compress any image with JPEG, code the data using LDPC to send across a channel. Is this comparing apples and oranges?
- Pg. 6 "Note, that a simple additive white Gaussian noise.... LDPC. However, in more general scenarios they do not perform as well and can be beaten by neural network architectures....."
I think the claims in this paragraph need to be toned down a bit. LDPC does not perform well in regards to what? Can be beaten under what conditions? Decode efficiently with regard to what block length? I don't think the picture is as clear as painted here.
- Pg. 4 The statements here regarding the bandwidth-limited channel appear to be the focus of the author's work. This section should be expanded and explained. Reading the first paragraph then going to the second paragraph ("To summarize..."), there's sort of a disconnect. How do we know these two things are equivalent? What exactly is novel that was introduced?
Small typos:
- Pg. 2 final paragraph, some  norm is used for the distortion, but this is not defined (either on pg. 2 or in App. B)
- Pg. 6 white is misspelled "withe"


Despite their popularity and successes, deep neural networks are poorly understood theoretically and treated as 'black box' systems. Using a functional view of these networks gives us a useful new lens with which to understand them. Theoretical (in shallow) and experimentally probing properties of these networks reveals insights into the effect of standard initializations, the value of depth, the underlying loss surface, and the origins of generalization. One key result is that generalization results from smoothness of the functional approximation, combined with a flat initial approximation. This smoothness increases with number of units, explaining why massively overparamaterized networks continue to generalize well.

This paper proposes a functional characterization to understand the empirical success of deep neural networks. In particular, this paper focuses on the case of deep fully connected univariate ReLU networks, and show that the parameters will result in a Continuous Piecewise Linear (CPWL) approximation to the target function. Moreover, the authors derive the induced distributions of the function space parameters and show that increasing width can reduce the roughness of the initial function.
Besides, this paper analyzes the loss surface in the function space and reveals some relationship between the critical points in the function space and original NN parameter space. Furthermore, a type of gradient descent dynamic in the function space has also been derived.
Many experiments have been conducted to reveal how the expressiveness and optimization performance varies with the neural network width and depth.
Overall, the functional characterization is interesting can potentially help explain the generalization/expressiveness of deep neural networks. However, this paper is not well written and organized, and there are some “??'s” appearing on page 4. The authors should pay more attention to improving the writing and organization of this paper.
Here are the detailed comments:
The statements of theorems are not clear. For example, Theorems 3 and 4 convey too much information, I believe the authors should simplify the statements to make them more concise.  Moreover, the purpose of these two lemmas is not clear. Do they imply something related to expressiveness?
There should be some discussion in the surrounding text of Theorem 5. Some notations are also missing. For example, what’s $\hat \epsilon(t)$?, what’s $a_i(t)$? Besides, what’s the purpose of Theorem 5. It seems that this Theorem is used in the main part of this paper.
One major drawback of this paper is that it only provides the theoretical analysis for two-layer networks, but the title and claimed contributions are related to deep ReLU networks. I wonder whether the theory can be extended to deep cases?
In Theorem 2, I think He initialization will give you $\sigma_w = \sqrt{2/(H+1)}$ and $\sigma_v = \sqrt{2/k}$, if the output dimension is k. In this way, I am curious whether Theorem 2 can still hold. Besides, what’s the meaning of the so-called “roughness“?
I don’t get the point in Implications of Corollary 1. For example, what do you mean “$f$ has significant curvature in the boundaries?”, why an initialization that allocates more breakpoints to the area where the curvature of $f$ lies can be faster to train? The authors should elaborate more on this.
After reading the authors' response.
Thanks for your response.  I still think the contribution of this paper is not enough as the theoretical analysis may not be able to be generalized to deep networks. Thus I would like to keep my score.


Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK]  and then train a model to reconstruct the original tokens. While effective when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by methods such as BERT and XLNet given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where we match the performance of RoBERTa while using less than 1/4 of the compute.

The authors propose replaced token detection, a novel self-supervised task, for learning text representations.
The principle advantage of the approach is that, in contrast with the standard masked language model (MLM) objective used by BERT and derivatives, there is a training signal for all tokens of the input (rather than a small fraction, when 10-20% of the input tokens are masked and then reconstructed under the MLM objective).
A smaller MLE-trained BERT-style generator is used to replace masked words with plausible alternatives, which the ELECTRA discriminator (the part that is retained and finetuned on downstream tasks) must detect (unmodified word slots are also in the objective, and must be detected as such).
In general the paper reads well, and the authors present ablations to reveal the source of gains. ELECTRA matches the performance of RobBERTa on the popular GLUE NLP task, with just 1/4 of the training compute.
Strengths:
-Simple but novel self-supervised task for learning text representations, strong results, adequate ablation.
Limitations:
-The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper, as it may be that replaced token detection is more suitable for these tasks, but inferior to MLM (a higher precision self-supervised task) for more involved tasks like question answering. The latter is arguably of much higher importance to the NLP research community at this point, and some consider the GLUE task to be essentially solved for all practical purposes, given inherent noise levels.
-In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work
Overall:
An okay paper. Results on SQUAD or another more elaborate NLP task and/or the release of the ELECTRA models would make the paper much stronger.


This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents. Moreover, they provide a learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space. Existing methods for solving text-based games are limited to games that are either very simple or have an action space restricted to a predetermined set of admissible actions. In this work, we propose to use the exploration approach of Go-Explore (Ecoffet et al., 2019) for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in solving textbased games, and it is more sample efficient in terms of the number of interactions with the environment. Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.

This paper considers the task of training an agent to play text-based computer games. One of the key challenges is the high-dimensional action space in these games, which poses a problem for many current methods. The authors propose to learn an LSTM-based decoder to output the action $a_t$ by greedily prediction one word at a time. They achieve this by training a sequence to sequence model on trajectories collected by running the game using a previously proposed exploration method (Go-Explore). While the results are promising, there might be limited novelty beyond training a sequence to sequence model on pre-collected trajectories. Further, the experiments are missing key elements in terms of proper comparison to baselines.
Pros:
1. Nice idea for tackling the unbounded action space problem in text-based games.
Cons:
1. The method depends on the assumption that we can get a set of trajectories with high rewards. This seems a pretty strong assumption. In fact, the authors use a smaller set of admissible actions in order to collect these trajectories in the first place - this seems to not be in line with the goal of solving the large action space problem. If we assume access to this admissible action function, why not just use it directly?
2. Some of the empirical results may not be fair comparisons (unless I'm missing something). For example, all the baselines for the CookingWorld games use $\epsilon$-greedy exploration. Since the Go-Explore method assumes access to extra trajectories at the start, this doesn't seem fair to the other baselines which may not observe the same high-reward trajectories.
Other comments:
1. Do you use the game rewards to train/finetune the seq2seq model or is it only trained in a supervised fashion on the trajectories? (like an imitation learning setup)
2. How critical is the first step of producing high reward trajectories to the overall performance? Some more analysis or discusssion on this would be helpful to disentangle the contribution of GoExplore from the seq2seq action decoder.


We propose a novel framework for avoiding the misclassification of data by using a framework of learning with rejection and adversarial examples. Recent developments in machine learning have opened new opportunities for industrial innovations such as self-driving cars. However, many machine learning models are vulnerable to adversarial attacks and industrial practitioners are concerned about accidents arising from misclassification. To avoid critical misclassifications, we define a sample that is likely to be mislabeled as a suspicious sample. Our main idea is to apply a framework of learning with rejection and adversarial examples to assist in the decision making for such suspicious samples. We propose two frameworks, learning with rejection under adversarial attacks and learning with protection. Learning with rejection under adversarial attacks is a naive extension of the learning with rejection framework for handling adversarial examples. Learning with protection is a practical application of learning with rejection under adversarial attacks. This algorithm transforms the original multi-class classification problem into a binary classification for a specific class, and we reject suspicious samples to protect a specific label. We demonstrate the effectiveness of the proposed method in experiments.

This paper wants to study the problem of “learning with rejection under adversarial attacks”. It first naively extends the learning with rejection framework for handling adversarial examples. It then considers the classical cost-sensitive learning by transfer the multi-class problem into binary classification problem through one-vs-all and using the technique they proposed to reject predictions on non-important labels, and name such technique as “learning with protection”. Finally, they do some experimental studies.
The paper does not show any connection between “learning with rejection” and “adversarial learning”. The method it proposes is also a naïve extension of existing methods. Both the problem setting and the technique does not have novelty. The paper fails to realize that the motivated application is actually called “cost-sensitive learning” and has been studied long time before. The paper also has problems in writing. Finally, there is no comparison with any baseline. Only empirical results of the proposed methods are shown. Due to all these reasons, there is still a long way to go before the paper can be published. I will rate it a clear rejection.
More specially,
The definition of “suspicious example” in Sec.3.1 has no relationship with adversary examples. Does the paper focus on adversary examples? If the definition has no relationship, it is classical learning with rejection.
In the last equation of Page 3, there is no definition of \tilde L. Actually, according to Figure 1, x’s is more close to the decision boundary, it is an example more hard to classify, which could also be “suspicious”.
In the definition of “suspicious example” at the beginning of Sec.3.1, is both x and x’ defined as suspicious examples in this way?
In the last equation of page 2, there is a rejection function, so minimizing this loss is a “separation-based approach”. However, at the end of Sec.2 the paper states they “follow a confidence-based approach”. Any comment on the inconsistency?
The motivated problem is not new. It is called cost-sensitive learning in machine learning and can date back to 2001:
Charles Elkan. The Foundations of Cost-Sensitive Learning. IJCAI 2001: 973-978.
Where they study the same problem when misclassifying one class of data may cost a lot than misclassifying another class of data. The current paper has not discussed any related work of cost-sensitive learning although they want to study a problem in its field.
The paper should be also improved in writing in the following aspects.
There is a lot of inaccurate statements in the paper. For example,  “In Sections 3 and 4, we propose and describe our algorithm”, what is the difference between propose and describe? “an estimator \hat h might return result that differ greatly from h^* in a case with finite samples”. Actually there are rigorous theoretical results describing how the number of finite samples will impact the estimator \hat h on unseen data. For example,
Peter L. Bartlett, Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. JMLR, 2002.
So inaccurate/unclear statements that will mislead readers should be avoided.
In writing, the paper also lacks the necessary references in many places. For example, “Learning with rejection is a classification scenario where the learner is given the option to reject an instance instead of predicting its label.”, “…classifies adversary attacks to two types of attacks, white-box attack and black-box attack.”, “Methods for protecting against these adversarial examples are also being proposed.”. Necessary references are needed for these places.
The organization is also problematic. For example, in the second half of Sec.2 introducing two kinds of learning with rejection models, it should be included in a “related work” part.
------------------------------------------
Thank you for the rebuttal. I raised my score a little bit. But I still think this paper has not been ready to be published yet.


We present a meta-learning method for learning parametric loss functions that can generalize across different tasks and model architectures. We develop a pipeline for training such loss functions, targeted at maximizing the performance of model learning with them. We observe that the loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore, we show that our meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during meta-test time.

This paper proposes a mete-learning approach to learn a loss function from old tasks which can generalize well to new tasks. The benefits of the proposed approaches are 1) data-driven a loss function and 2) allowing the usage of extra side-information to design the loss function.
Overall, the presentation in this paper is hard for me to understand technical details and see the difference with existing methods. Please see the questions below. I am glad to discuss problems with the authors' reply during the rebuttal.
Q1. "Mφ(y; fθ(x)) that predicts the loss gave the ground truth target y and the predicted target fθ(x)", "the purpose...loss function, or a meta-loss, which generalizes across multiple training contexts or tasks".
- It is better for the authors to visualize all y v.s. f_{\theta}(x) for all experiments. They have done this only for "Section 4.2.1 SHAPING LOSS FOR REGRESSION". It is better to do this for all the experiments. In this way, we can see why meta-loss can better.
- Besides, it is also better to show some example samples where the learning loss is significantly different from the human-designed one. This helps the reader better understand why the meta-loss can better.
- Finally, the authors claim the extra-information used in the meta-training is helpful. How can we see this point? There is not a step-by-step ablation study on this point.
Q2. Except for problem setup, i.e., learning a loss function, what are novelties in using meta-learning techniques?
Q3. The authors present three usages of the proposed framework in Section 3. Could the authors describe one in detail and then briefly mention the other two usages instead of writing them with the same importance? In this way, readers can understand materials and novelties better.
- For example, I do not understand how exactly gradients are updated on meta-level. The description in Section 3.1. is too brief.
Q4. Why the convergence speed of the meta-learner is important? e.g., Figure 4(b-c).
Q5. We have some basic restrictions for "loss function", i.e., loss(x, y) >= 0 for any x, y; loss(x, x) = 0. How such basic requirements are ensured by the learned meta-loss?
Q6. Could the authors add more explanation in the experiments and motivation in the main text? Currently, the authors just describe what they have done in the proposed method and what they have observed in experiments, just a list of facts (see Q1).


Learning control policies in robotic tasks requires a large number of interactions due to small learning rates, bounds on the updates or unknown constraints. In contrast humans can infer protective and safe solutions after a single failure or unexpected observation. In order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive inference and memorization process for avoiding failures in motor control tasks. A Gaussian Process implements the modeling and the sampling of the acquisition function. This enables rapid learning with large learning rates while a mental replay phase ensures that policy regions that led to failures are inhibited during the sampling process. The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task. We quantitatively compare the human learning performance to our learning approach by evaluating the deviations of the center of mass during training. Our results show that we can reproduce the efficient learning of human subjects in postural control tasks which provides a testable model for future physiological motor control tasks. In these postural control tasks, our method outperforms standard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures.

After rebuttal:
Thank you to the authors for responding to my review.
1) The title of the conference is "... on Learning Representations". As I stated in the review ("no, e.g., neural networks are employed"), neural networks are an *example* of, but do not subsume, all representation learning methods. Therefore, I agree that papers that do not cover neural networks are welcome at the conference. However, as I stated in the review, my evaluation of the method proposed in the submission is that it does not concern representation learning ("The employed features in Table 3 are handcrafted"). I believe this evaluation is defensible, but of course the final evaluation is up to the chairs. However, I note that the authors did not respond directly to my evaluation that the method is not engaging in representation learning.
2-7) As the other reviewer notes, the paper lacks clarity in many places, and does not sufficiently discuss prior work, including in postural control (there is one citation in the references that is not mentioned in the main text), hierarchical Bayesian optimization within or without a Gaussian processes framework (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=hierarchical+bayesian+optimization&btnG=), or experience replay (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=replay+machine+learning&btnG=). Therefore, it is difficult to ascertain the research contribution.
As such, I stand by my evaluation that this submission is not ready for publication at ICLR.
===========================
Before rebuttal:
The submission presents a hierarchical Bayesian optimization (HiBO) approach to solving a postural control task expressed as a proportional-derivative (PD) controller.
Strengths:
- The HiBO approach outperforms the non-hierarchical BO approach on the task of postural control.
Weaknesses:
- The paper does not make use of representation learning (no, e.g., neural networks are employed) and is therefore out-of-place at ICLR. The employed features in Table 3 are handcrafted.
- The task (simulating human postural control) is not well-situated in the context of prior work using HiBO for robotics, so the contribution remains unclear.
- It is not clear why this problem should be formulated as contextual policy search (i.e., to what the context variable refers).
- Only one baseline (Bayesian optimization (BO)) is reported. This baseline corresponds to the ablation of the HiBO method (i.e., the omission of the context variable), and so does not represent, more broadly, an alternative approach.
- The concept of "mental replay" is briefly introduced, but no reference is made to prior work in imagined rollouts, and no ablation study on the impact of this component is performed.
Minor points:
- It is unclear why the problem setting should be labeled as "psychological" postural control.
- There are several missing references ("?") in the text.


That machine learning algorithms can demonstrate bias is well-documented by now. This work confronts the challenge of bias mitigation in fully-connected feedforward neural nets through the lens of multiobjective optimisation. Specifically, recognising that fairness and accuracy are competing objectives, the proposed methodology uses techniques from multiobjective optimisation to ascertain the fairness-accuracy landscape of a neural net classifier. Along the way, a new causal notion of fairness is introduced that is particularly suited to giving a nuanced treatment when data is collected under unfair practices. In particular, special attention is paid to subjects whose covariates could appear with substantial probability in either value of the sensitive attribute. Experimental results suggest that the proposed method produces neural net classifiers that distribute evenly across the Pareto front of the fairness-accuracy space and is more efficient at finding non-dominated points than an adversarial approach.

This paper proposes a method to approximate the "fairness-accuracy landscape" of classifiers based on neural networks.
The key idea is to set up a multi-dimensional objective, where one dimension is about prediction accuracy and another
about fairness. The fairness component relies on a  definition of fairness based on causal inference, relying on the
idea that a sensitive attribute should not causally affect model predictions.
I found the causal idea intriguing, since it makes sense that we don't want a sensitive attribute to have a causal effect.
However, there may be several problems with this approach:
1) For a causal estimate to be valid we need several assumptions. For example, we need A (the sensitive attribute)
to be independent of potential outcomes conditional on X --- the so-called "unconfoundedness assumption" in causal inference.
We also need "positivity", i.e., that 0< P(A=1|X) <1.
These assumptions are not discussed in the paper. Furthermore, the particular context of the paper, where the treatment is actually an immutable characteristic, makes such discussion much more subtle.
What will we do, for instance, if there are no A=1 in the sample when X = ...?
2) The authors seem to assume that the propensity score model is well specified. This can be tested, e.g., using [1].
What do we do when this fails?
3) Why do we want U to be small, i.e., why do we want the causal effect of A to be small, is never justified.
In particular, its relation to "fairness" is never fleshed out, but just assumed to be so.
This can be problematic when, say, we are missing certain important X that are important for A.
Then, there will be a measurable causal effect of A on h().
Some other problems:
- What is the reason for focusing on 'neural classifiers'? There is nothing specific in the method or analysis
that relates to neural networks, except for the use of the causal estimand in a 'hidden layer'.
- In the Introduction, the authors could cite the works of Amartya Sea, etc., on fairness.
Certainly the study of fairness problems did not start in 2016.
- What exactly is a "sensitive attribute"? If we don't want to bias our predictions, then why include it in the analysis?
- It is unclear what is new and what is related work in page 3.
- Sec. 4: The claim that "causal inference is about situations where manipulation is impossible" discards
voluminous work in causal inference through randomized experiments. In fact, many scientists would
agree that causal inference is impossible without manipulation.
- As mentioned above, why this particular estimand leads to more "fairness" is never explained.
- Do we need a square or abs value in Eq (5)?
- The experimental section is weak and does not illustrate the claims well.
   It would be important to explain the choice of the particular causal estimand, the choice of the hidden layer to put the estimand in, to explore the choice of the objective, and so on. Currently, none of these choices/design aspects are being investigated.
[1] "A specification test for the propensity score using its distribution conditional
on participation" (Shaikh et al, 2009)


Audio adversarial examples, imperceptible to humans, have been constructed to attack automatic speech recognition (ASR) systems. However, the adversarial examples generated by existing approaches usually involve notable noise, especially during the periods of silence and pauses, which may lead to the detection of such attacks. This paper proposes a new approach to generate adversarial audios, which uses Iterative Proportional Clipping (IPC) to significantly limit human-perceptible noise and simultaneously exploits the temporal dependency to resist the defenses based on temporal structure. Specifically, in every iteration of optimization, we use a backpropagation model to learn the raw perturbation on the original audio waveform to construct our clipping. We then impose a constraint on the perturbation at the positions with lower sound intensity across the time domain to eliminate the perceptible noise during the silent periods or pauses. IPC preserves the temporal closeness and trend in the original audio waveform to maintain the temporal dependency. We show that the proposed approach can successfully attack the latest state-of-the-art ASR model Wav2letter+, and only requires a few minutes to generate an audio adversarial example. Experimental results also demonstrate that our approach succeeds in preserving temporal dependency and can bypass temporal dependency based defense mechanisms.

This paper introduces an adversarial attack mechanism, Iterative Proportional Clipping (IPC), based on a differentiable MFCC feature extractor. In contrast to other audio attacks, the authors claim that the perturbations are well masked and are much less audible.
They also claim to propose the first attack mechanism that takes temporal dependency into account, at least in some intuitive sense as described in Figure 2. Simulations show that state of the art speech recognizers can be attacked using the approach and certain type of defenses, based on temporal structure, can be circumvented. 	The method relies on the idea of using ‘the proportionally clipped gradient’ as a direction estimate for original audio modification. The basic idea is quite sensible and seems to be effective as supported by computational experiments.
The dataset is LibriSpeech and the speech recognizer is based on wav2letter+ -- here the MFCC features are replaced with a differentiable version to allow gradient based adversarial attacks. The experimental section is detailed and the results illustrate the
The fundamental shortcoming with the paper is that while the method is described in detail, there is not much effort in explaining and formalizing what aspects of the proposed method make it perceptually more appealing -- as this seems to be the one of the main motivations of the paper. I would at least expect to see at least some effort to explain the proposed  clipping technique as maintaining the phase properties of the original signal. The authors claim about perceptual properties of the method seem to be somewhat anecdotal and, while I think the method is sensible, just showing the power spectrum is not very informative. Is the proposed method better just because it does good detection of the  silent periods?
I also find the use of technical jargon somewhat ambiguous at occasions (such as the term ‘linear proportionality’)
Minor:
audios -> waveforms  (audio does not have a plural form )
Fig 2 ‘frequency - signal frequency’ labels -- this is probably just an estimate of the power spectrum by DFT magnitude


Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses. As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies. A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images. To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention. By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels. Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods. Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics. To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

This paper applies the SPADE semantic image synthesis technique (with a custom attention mechanism) to MIBI-TOF data to examine hypotheses about cell-to-cell interactions in the context of an immune infiltrated tumor sample. I think that this is potentially an interesting application of GANs -- to generalize beyond specific gathered data and instead start engaging with counterfactual scenarios like "what effect does it have to add cell type X next to cell type Y".
Unfortunately, I think this paper is not yet finished with regard to both (1) conveying sufficient motivation for the use of image synthesis for distilling biological insights and (2) evaluating the success or failure of the technique primarily in terms of generating useful biological insights. Simultaneously, there are enough red flags with regard to knowledge of the data generation and underlying biology that it's unclear if the authors could correctly sanity check any insights they extract from their model.
More specific criticisms & suggestions
1) It would help to provide much clearer motivation for applying image synthesis to MIBI-TOF data. Section 1.2 skips straight from describing the MIBI-TOF instrument to “we made a new kind of GAN”.  Again in the beginning of the Related Work the paper states: "We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods”. But, why are synthetic images interesting given that we can actually look at real MIBI-TOF data. I start getting a sense of why this model might be interesting or useful only in Section 5 — more rationale is required earlier in the paper.
2) The evaluation section is hard to follow. I think it would be helpful to more clearly describe a larger set of biological phenomena that a practitioner would expect to see in the data, choose a single metric for each case, and show that that these phenomena are recapitulated. No one is going to trust a GAN to give them scientific insights unless they're very confident that all known / simple cell-to-cell interactions have a clear signal.
3) "MIBI-TOF bombards a tissue sample with elemental metals tethered to respective antibodies for dozens of distinct cellular proteins and detects each to obtain image channels” — this is not an accurate description of MIBI-TOF, at least not the instrument I'm familiar with. Typically the tissue is first stained with antibodies tagged with heavy metals and the instrument then bombards the tissue sample with simple ions (like O2+), causing the release of metal ions. It seems very unlikely that bombarding anything with antibody/metal conjugates could be informative.
4) "Tumor cells could be identified by markers such as pan-keratin and beta-catenin” — I guess in the context of a tumor sample beta-catenin could be over-expressed but it's present in pretty much every cell type, including lymphocytes (https://www.proteinatlas.org/ENSG00000168036-CTNNB1)
Small nits:
* Repeated use of “antibodies” in "Engineered antibodies for PD-1/PD-L1 antibodies” — maybe better to write “Antibodies which block the interaction of PD-1/PD-L1"
* "While MIBI-TOF is capable of 36 different markers, we discarded uninformative and irrelevant markers1 resulting in M = 24.” — I’m really surprised that someone put 12/36 uninformative markers in a MIBI-TOF panel. Aren’t these tagged antibodies expensive? Can we at least get a list of what got discarded?
* The model interpretability control seems weak in that lymphocytes are more likely to express MHC-I than tumor cells (which have a potential survival advantage from not expressing it) and tumor cells may actually have more dsDNA than lymphocytes due to changes in ploidy (or original differences in ploidy from e.g. liver cells).


Thanks to graph neural networks (GNNs), semi-supervised node classification has shown the state-of-the-art performance in graph data. However, GNNs have not considered different types of uncertainties associated with the class probabilities to minimize risk increasing misclassification under uncertainty in real life. In this work, we propose a Subjective Bayesian deep learning framework reflecting various types of uncertainties for classification predictions by leveraging the powerful modeling and learning capabilities of GNNs. We considered multiple uncertainty types in both deep learning (DL) and belief/evidence theory domains. We treat the predictions of a Subjective Bayesian GNN (S-BGNN) as nodes' multinomial subjective opinions in a graph based on Dirichlet distributions where each belief mass is a belief probability of each class. By collecting evidence from the given labels of training nodes, the S-BGNN model is designed for accurately predicting probabilities of each class and detecting out-of-distribution. We validated the outperformance of the proposed S-BGNN, compared to the state-of-the-art counterparts in terms of the accuracy of node classification prediction and out-ofdistribution detection based on six real network datasets.

The authors proposed a Bayesian graph neural network framework for node classification. The proposed models outperformed the baselines in six node classification tasks. The main contribution is to evaluate various uncertainty measures for the uncertainty analysis of Bayesian graph neural networks. The authors show that vacuity and aleatoric measure are important to detect out-of-distribution and the dissonance uncertainty plays a key role for improving performance.
** Introduction/Conclusion/Contribution
- Applying MC dropout to graph neural networks is a bit old idea [1-5], it cannot be considered as the “contribution” of this study.
- The authors should emphasize more on what’s the advancements over existing studies.
** Methodology
- My understanding is the authors proposed to use multiple uncertainties (vacuity, dissonance, aleatoric, epstemic). Vacuity and dissonance measures can also be implemented with other Bayesian graph neural networks.
- Ablation study is need to demonstrate the usefulness of each component in the objective function (equation 11).
- Can “BGAT-T” be considered as the proposed method? It doesn't use the proposed GNN framework but just an extension of original GAT with MC dropout and knowledge distillation.
** Experiments
- Please provide the details of hyperparameter settings, e.g. optimizer, batch size, learning rates, …
- The authors should perform experiments with varying numbers of labels per category (e.g. 5, 10, 20), because different methods show different behaviors under label scarcity.
- There are a number of recently published papers that address node classification based on Bayesian graph neural networks, e.g., see references. They should be used as baselines if available.
- I think the authors should evaluate BGAT-T with Co.Physics, Ama.Computer, and Ama.Photo datasets, because this method was overall superior to other methods on the first three datasets.
- Experiments are insufficient in the uncertainty analysis (section 5). The authors can evaluate the performance of BGAT as well as other Bayesian graph neural networks in terms of uncertainty quantification performance.
- Also, in order to evaluate uncertainty quantification performance, I would suggest to look at the trade-off between classification accuracy and classification rejection based on the uncertainty, like accuracy-rejection curve in [6]
** misc
- Check the following sentence of subsection 3.6. “our key contribution is that the proposed Bayesian GNN model is capable of estimating various uncertainty types to predict existing GNNs.”
References
[1] Ryu, S., Kwon, Y., & Kim, W. Y. (2019). Uncertainty quantification of molecular property prediction with Bayesian neural networks. arXiv preprint arXiv:1903.08375.
[2] Zhang, Y., & Lee, A. A. (2019). Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. arXiv preprint arXiv:1902.00925.
[3] Pal, S., Regol, F. L. O. R. E. N. C. E., & Coates, M. A. R. K. (2019). Bayesian graph convolutional neural networks using non-parametric graph learning. In Representation Learning on Graphs and Manifolds Workshop, Int. Conf. Learning Representations.
[4] Akita, H., Nakago, K., Komatsu, T., Sugawara, Y., Maeda, S. I., Baba, Y., & Kashima, H. (2018, December). Bayesgrad: Explaining predictions of graph convolutional networks. In International Conference on Neural Information Processing (pp. 81-92). Springer, Cham.
[5] Zhang, Y., Pal, S., Coates, M., & Ustebay, D. (2019, July). Bayesian graph convolutional neural networks for semi-supervised classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 5829-5836).
[6] Nadeem, M. S. A., Zucker, J. D., & Hanczar, B. (2009, March). Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. In Machine Learning in Systems Biology (pp. 65-81).


Federated learning, where a global model is trained by iterative parameter averaging of locally-computed updates, is a promising approach for distributed training of deep networks; it provides high communication-efficiency and privacypreservability, which allows to fit well into decentralized data environments, e.g., mobile-cloud ecosystems. However, despite the advantages, the federated learning-based methods still have a challenge in dealing with non-IID training data of local devices (i.e., learners). In this regard, we study the effects of a variety of hyperparametric conditions under the non-IID environments, to answer important concerns in practical implementations: (i) We first investigate parameter divergence of local updates to explain performance degradation from non-IID data. The origin of the parameter divergence is also found both empirically and theoretically. (ii) We then revisit the effects of optimizers, network depth/width, and regularization techniques; our observations show that the well-known advantages of the hyperparameter optimization strategies could rather yield diminishing returns with non-IID data. (iii) We finally provide the reasons of the failure cases in a categorized way, mainly based on metrics of the parameter divergence.

The paper experimentally studies the reasons for the slow convergence of the Federated Averaging algorithm when the data are non-iid distributed between workers in the multiclass-classification case. Paper performs extensive experimental study and observes that the main reasons for failure are connected to (i) the parameter divergence during the local steps, (ii) steep-fall phenomena when parameters on different nodes are getting close fast, and to the (iii) high training loss.
My score is weak reject. The paper provides extensive but unclear experimental results. Improving presentation would significantly improve the paper. For example, why in experimental and theoretical study different parameter divergence metrics were used, etc (see below), why different networks use different optimizers.
Moreover, provided experimental comparison might be unfair. The learning rate is constant throughout all of the experiments, depending only on the optimizer, but not on the neural network architecture. This can affect the final results.
Concerns and questions that should be addressed:
1. The initial learning rates were not tuned properly. It is set to be the same for different neural network topologies, which might significantly affect the results. What did the choice of initial learning rates is based on?
2. Why the parameter divergence metric in Definition 1 is not the same as in the theoretical study (Appendix B)? What is the intuition behind Definition 1?
3. Why the divergence of parameters is considered only at the last layer? It seems to hide many important interactions in the other layers.
4. Some important experimental details --- should be added:
   - At which moment the parameter divergence is computed in the plots? Is it computed at the end of the local iterations right before synchronization?
   - How the training loss was computed in the plots? before or after synchronization? on the local only or the global data?
   - Which batch size was used?
   - Improve the figure caption to detail the experimental setup. (e.g. in fig 3. the network architecture was mentioned only for one of the figures, include which optimized was used, etc)
5. In experiments on Fig. 2. and Fig.3 (middle) what is the accuracy for IID baseline? Is the observed phenomena connected to the poor network architecture or to the non-iid data?
6. In table 5 of the appendix, why experiments use Adam optimizer, but not Momentum SGD as in the main paper to compare the performance of ResNet14 and ResNet20?
7. Better re-prase the definition of the steep fall phenomena, now it is not very clear: in the IID setting parameter divergence values are also sometimes reducing sharply; in the network width study parameters divergence doesn’t experience sudden drop. Also, how does this phenomena (and parameter divergence too) connects to the training loss?
8. Why for different experiments different baseline models are used? (NetA, NetB, NetC)
Other minor comments:
- Appendix B, first equation on page 13. (d_q)^t -> (d_q)^t_k; The size of gradient \nabla_w [E ...] is different from the size of (d_q)_k. They cannot be added together.
- page 7, last sentence of the first paragraph: what is the accuracy achieved with Batch Renormalization? Why the reason for accuracy gap is “significant parameter divergence”? on fig. 3 “parameter divergence” is smaller than for the baseline.
- Why the name of the section on page 7 is “excessively high training loss of local updates” if later it is stated that it is actually smaller than for the IID case?
- Defenition 1, line 4: “the then” -> “the”
- section 3: “A pleasant level of parameter divergence can help to improve generalization” -> where was it shown?
- section 4.2, paragraph 2: what is meant by “hyperparametric methods”?
- section 4.2, paragraph 3: “quantitative increase in a layer level” -> not clear what does it mean.
- page 4, effect of optimizers: what do you refer to as “all model parameters”?
- page 5, last paragraph: Hinton et al... -> (Hinton et al…). Use \citet(\citep) instead of \cite.
- why Dropout yields bigger parameter divergence if on Fig 2, right it actually helps?
- Last line of the page 5. Where was this observed?


A critical part of the training process of neural networks takes place in the very first gradient steps post-initialization. In this work, we study the connection between the network's architecture and initialization parameters, to the statistical properties of the gradient in random fully connected ReLU networks, through the study of the the per layer Jacobian in finite sized networks. We compare three types of architectures: vanilla networks, ResNets and DenseNets. In our analysis, we show that while the variance of Jacobian squared norm is exponential in depth for ResNets, and polynomial for DenseNets, there exists an initialization strategy for both, such that the norm is preserved through arbitrary depths, preventing exploding or decaying gradients in deep networks.We also show that the statistics of the per layer Jacobian norm is a function of the architecture and the layer's size, but surprisingly, not the layer's depth.

This paper analyzes the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.
The main theoretical results stem from a norm propagation duality that allows computations of certain statistics using simplified architectures. As far as I know, this is a novel way of analyzing multi-pathway architectures, and the method and the implied results will be of interest to deep-learning theorists.
There are two types of empirical results, one to verify the theoretical predictions, and the other to see if those predictions translate into improved generalization performance on some real-world tasks. Figures 2 and 3 show good agreement with theory. I am less convinced by Fig 4. Unless I missed something, I thought the implication from the theoretical analysis was that multi-pathway models should behave better than single-pathway models at large depth, and for single-pathway models concatenated ReLU may have slightly better large-depth behavior than standard ReLU. Doesn't this suggest that for large depth DenseNet ~= ResNet > CR > ReLU? Is the good performance of CR supposed to be understood as supporting evidence for the theoretical analysis? Or is the point more like "the theory suggests CR might be interesting, and indeed it performs well, but we leave the details of generalization performance to a future study"? It would be important to specify the desired interpretation here.
But I think regardless of the interpretation, the empirical results are not strong enough to stand on their own as support for CR, especially since this type of architecture has been introduced and studied previously (c.f. also the looks-linear setup from [1]). Therefore my evaluation is mostly based on the theoretical contributions, which I think are themselves are probably sufficient for publication.
One main area for improvement is in the related work. A significant line of work is omitted that studies higher-order statistics of Jacobian matrices in the large-depth, infinite-width regime [2-8], and while this is at infinite width, it work for any non-linearity, so is an important point for comparison. Other kinds of initialization schemes are also relevant for comparison, e.g. [9-10].
[1] Balduzzi, David, et al. "The shattered gradients problem: If resnets are the answer, then what is the question?." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[2] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems. 2017.
[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "The emergence of spectral universality in deep networks." International Conference on Artificial Intelligence and Statistics. 2018.
[4] Hayase, Tomohiro. "Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network." arXiv preprint arXiv:1908.03901 (2019).
[5] Tarnowski, Wojciech, et al. "Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
[6] Burkholz, Rebekka, and Alina Dubatovka. "Initialization of ReLUs for Dynamical Isometry." arXiv preprint arXiv:1806.06362 (2018).
[7] Xiao, Lechao, et al. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks." International Conference on Machine Learning. 2018.
[8] Chen, Minmin, Jeffrey Pennington, and Samuel Schoenholz. "Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks." International Conference on Machine Learning. 2018.
[9]Sutskever, Ilya, et al. "On the importance of initialization and momentum in deep learning." International conference on machine learning. 2013.
[10] Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015).