Learning to cooperate is crucially important in multi-agent environments. The key is to understand the mutual interplay between agents. However, multi-agent environments are highly dynamic, where agents keep moving and their neighbors change quickly. This makes it hard to learn abstract representations of mutual interplay between agents. To tackle these difficulties, we propose graph convolutional reinforcement learning, where graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment, and relation kernels capture the interplay between agents by their relation representations. Latent features produced by convolutional layers from gradually increased receptive fields are exploited to learn cooperation, and cooperation is further improved by temporal relation regularization for consistency. Empirically, we show that our method substantially outperforms existing methods in a variety of cooperative scenarios.

The paper addresses the problem of coordination in the multi-agent reinforcement learning setting. It proposes the value function factorization similar to independent Q-learning conditioning on the output of the graph convolutional neural network, where the graph topology is based on the agents’ nearest neighbours. The paper is interesting and has some great ideas, for example, KL term to ensure temporal cooperation consistency. However, the paper has its drawbacks and I feel obliged to point them out below. I vote for the weak acceptance of this paper.
One of the main drawbacks of the paper is that it is extremely hard to grasp. Even the Abstract and Introduction are hard to understand without having a pass over the whole paper. The authors often use vague terms such as 'highly dynamic environments' or 'dynamics of the graph' which make it hard to understand what they mean. The paper would benefit from a more precise language. Some of the important notions of the paper are used before they are introduced, which make the general picture very hard to understand and to relate the work to the existing research.
'Related Work' section seems to be missing some recent work applying graph neural networks to multi-agent learning settings:
• Malysheva, Aleksandra, Tegg Taekyong Sung, Chae-Bong Sohn, Daniel Kudenko, and Aleksei Shpilman. "Deep Multi-Agent Reinforcement Learning with Relevance Graphs." arXiv preprint arXiv:1811.12557 (2018).
• Agarwal, Akshat, Sumit Kumar, and Katia Sycara. "Learning Transferable Cooperative Behavior in Multi-Agent Teams." arXiv preprint arXiv:1906.01202 (2019).
My questions to the authors:
• In section 3 you mention 'a set of neighbours ..., which is determined by distance or other metrics'. Can you elaborate on that? What are these metrics in your case?
• Just before the Section 3.1, you say 'In addition, in many multi-agent environments, it may be costly and less helpful to take all other agents into consideration.' Have you run any experiments on that? In the appendix, you show, that making the neighbourhood smaller negatively affects the performance, but what if you make it bigger? Ideally, I would like to see an extended version of Figure 8 and 9 in the main part of the paper since they are very interesting and important for the claims the paper makes.
• At the end of Section 3.1, you mention the soft update of the target network. Later, in 3.3, you say that that you do not use the target network. Can you elaborate more on that?
• In Equation 4, is it a separate KL for each of the attention heads? If yes, this is not clear from the formula.
• It will be useful to see the ablation experiments for all of the testbeds, not only for Battle.
• Why do you think the DQN performance drops in the second half of the training in Figure 4 for all of the runs?
• Have you tried summation instead of the mean aggregation step?
I will put comments for particular parts of the paper below.
ABSTRACT
>>> ...environments are highly dynamic
What do you mean precisely here?
>>> ...graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment
What is the 'dynamics of the underlying graph'? What is the 'graph of the multi-agent environment'?
>>> 'coordination is further boosted'
Not sure that 'boosted' is the right word here.
INTRODUCTION
>>> '...where mutual interplay between humans is abstracted by their relations'
Not sure what it means.
>>> we consider the underlying graph of agents...
The agent graph has not been introduced yet.
>>> DGN shares weights among all agent(s) making it easy to scale
What do you mean precisely by 'easy to scale'? Can you support this claim?
>>> We empirically show the learning effectiveness of DGN in jungle
Needs a reference to the testbed.
>>>  ... interplay between agents and abstract relation representation
What is 'abstract relation representation?
>>> We consider partially observable environments.
What do you mean precisely by that? What is the MDP formalism most suitable for your problem statement? What is objective under your formalism?
>>> However, more convolutional layers will not increase the local region of node i.
What do you mean by that?
>>> As the number and position of agents vary over time, the underlying graph continuously changes, which brings difficulties to graph convolution.
What kind of difficulties?
>>> As the action of agent can change the graph at next timestep which makes it hard to learn Q function.
Why does it make it hard?
>>> DGN can also be seen as a factorization of a centralized policy that outputs actions for all the agents to optimize the average expected return.
It would be useful for the reader to compare your approach with all the others type of the value function factorization. To me, your approach looks like a more sophisticated version of independent Q-learning, is that true?
Minor comments:
* In 3.2 it would be very helpful to put the dimensions for all of the variables for easier understanding.
* The brackets in the equation 3 are not very clear (what are you concatenating across?)
* In section 4, when describing an environment you say 'local observation that contains a square view with 11x11 grids'. What is the total size of the environment?
* The performance plots for Battle include ablations before the ablation subsection is introduced. This is a bit confusing.
* All figures/tables captions should be more detailed and descriptive.
* ‘However, causal influence is not directly related to the reward of environment.’ Should be ‘of the environment’.


Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

--- Overall ---
This paper proposes an algorithm for learning from data with noisy labels which alternates between updating the model and removing samples that look like they have noisy labels, thereby allowing the training procedure to focus on clean samples. Overall, I found the paper very well-written, the proposed approach reasonable, and the experiments convincing. I have some questions about what assumptions are required for such a procedure to work, but in general, I think this is a strong paper.
--- Major comments ---
1. I found it somewhat unclear how large the methodological contribution was. In particular, has the approach of filtering out samples based on disagreement with predictions from the model been tried before (i.e. the primary contribution is self-ensembling)? When the proposed method includes multiple pieces (Mean teacher + iteratively creating a filtered dataset + self-ensembling), I recommend being *very* explicit about which parts are new contributions. With that said, I greatly appreciated the ablation experiments which really highlight the importance of each piece.
2. I don't feel like I have a good sense for what assumptions need to be satisfied for the proposed method to work. For example, in section 2.2 the authors say "If the noise is sufficiently random, the set of correct labels will be representative to achieve high model performance". What is meant by "sufficiently random" here? Is there a formal version of this assumption? Do any independence or positivity assumptions need to be satisfied? Most importantly, what happens when the label noise does not look like what you expect? I would love to see some experiments examining the failure modes of the algorithm. For example, what happens when label errors are concentrated in a particular region of the feature space (or just generally depend on the features)? In this case, even if the filtering procedure work perfectly, the filtered dataset will have a different feature distribution than the data distribution leading to potential covariate shift problems. If I understood the experiments correctly, the method was only tested on label-depended noise models.
3. Along the same lines: what are the necessary conditions to guarantee that this procedure converges? While the authors suggest that self-ensembling prevents samples from oscillating in and out of training set, is this a guarantee or an empirical observation? More broadly, it is not totally clear what the filtering does to the objective function or whether this procedure is even formally optimizing a well specified objective function (potentially some temperature limit of a soft-weighted objective?).
--- Minor comments ---
1. Figures 1 and 4 are not readable in black and grey-scale.
2. I would front-load the justification for using self-ensembling. In particular, I think the two sentences starting with "When learning under label noise,..." on page 2 could be moved much earlier.
3. I'll be interested to see what the other reviewers say, but I found Figure 2 hard to follow.
4. The formatting of Section 4.2.4 makes it a bit hard to figure out where the text starts (as opposed to the table captions).


We tackle the problem of inpainting occluded area in spatiotemporal sequences, such as cloud occluded satellite observations, in an unsupervised manner. We place ourselves in the setting where there is neither access to paired nor unpaired training data. We consider several cases in which the underlying information of the observed sequence in certain areas is lost through an observation operator. In this case, the only available information is provided by the observation of the sequence, the nature of the measurement process and its associated statistics. We propose an unsupervised-learning framework to retrieve the most probable sequence using a generative adversarial network. We demonstrate the capacity of our model to exhibit strong reconstruction capacity on several video datasets such as satellite sequences or natural videos.

Summary
The paper presents an approach to perform video inpainting from corrupted input only . Their approach proposed a method that uses GANs for denoising images to now handle full sequences of images by building on top work that handles inpainting in single images.
Strengths
1) The authors present extensive experiments on many datasets.
2) The presented approach is simple and general enough to be applied to many video problems.
3) The provided code is well structured and easy to read and atrached webpage showcases their results well.
4) The paper is well-written.
Weakness
1) From the code and the description in paper, it seems that a different corruption (https://github.com/anon-ustdi/ustdi/blob/7a81db4972ef9d4eabbd8fe354a8984a7771ae5d/src/datasets/corrupted.py) is applied for each step. Can the authors confirm if that is the case?
If this is the case, I am afraid the method cannot be called unsupervised as across many steps the model would have seen different corruptions of the same video and across many such corruptions the model can learn what an uncorrupted video looks like. It is okay if that is the case but the claim of unsupervised would not hold. If that is the case, the authors need to make comparison with other supervised inpainting methods as well [1,2]
2) Is there a dataset for which these corruptions exist naturally? May be [3, 4].  It would be nice to have experiments on a dataset where the corruptions are present naturally.
3) While the experiments are incomplete for Table 1,  [4] outperforms the proposed approach for the one dataset the numbers have been reported. Performance comparison with [4] are not fair as [4]'s implementation is in MATLAB.
Decision
While the presented approach is good, further experiments are required to further validate the effectiveness of their approach in an unsupervised setting.
References
[1] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details.
[2] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting.
[3] https://github.com/stayhungry1/Video-Rain-Removal
[4] https://github.com/nnUyi/DerainZoo
[5] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes.


Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets, but they can be prohibitively expensive to apply in deep learning. Unlike in other areas of machine learning, the feature representations that these techniques depend on are learned in deep learning rather than given, requiring substantial training times. In this work, we show that we can greatly improve the computational efficiency of data selection in deep learning by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, or training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signal for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error. For core-set selection, proxies that are over 10× faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, making end-to-end training time savings possible.

What is the paper about ?
*The paper proposes a simple method to speed up active learning/core-set selection in deep learning framework. The idea is that instead of using full scale deep model for data selection/uncertainty sampling, use a smaller/faster proxy model and this proxy model’s uncertainty estimates are good enough for choosing data points even though proxy model’s accuracy may not be good.
*The paper experiments with 5 datasets with 2 active learning and 3 core set selection strategies and shows promising speed-up with little to no regression in performance in most settings.
What I like about this paper ?
*The idea is simple and the paper is well written and easy to follow with ample references.
*Unlike a lot of active learning papers, this paper considers both image and text domains.
*Results in both Table 1 and Figure  2 have been reported after multiple runs 3 and 5 respectively. Authors also report standard deviation which is important because the results can vary a lot from run to run in active learning.
*Plenty of empirical results do make the paper attractive and this definitely makes for a nice contribution to the active learning community.
What needs improvement ?
*Novelty is lacking. While the authors may be the first ones to apply this idea in the context of deep learning, they themselves note that the idea of using a smaller proxy like naive bayes for larger models like decision trees is not new. Although I agree this should not be a criteria for rejection especially given the extensive experimentation performed in the paper and has not impacted my score.
*The choice of datasets is not justified in the paper and could have been better. I don’t see the point of showing that method works well for both Amazon Review Polarity and Amazon Review Full. Same can be said for CIFAR10 and CIFAR100. Would have appreciated a more diverse set of datasets. I would have been happy to see results on WMT given data selection is an open challenge in machine translation.
*Paper initially proposes two methods for proxy: one by scaling down the model and two by reducing the number of epochs. fastext falls under neither of those categories and the giant speed up claims made by the paper like 41.7x are only with fastext. It is also important to note that while the performance drop using the fastext or smaller version of VDCNN is not much as compare to VDCNN, the performance gain is also not a lot compared to Random. Due to this reason I am not convinced if I should use SVP if I want speedup or just randomly select points especially in text domain.
*Would have liked to see the comparison of #parameters in your different proxy models instead of just names like VDCNN29 and VDCNN9.
*A lot of experimental details are missing which would make it difficult to reproduce the results unless the code is released.
*Why fasttext worked as a proxy for VDCNN and AlexNet did not work with ResNet is unclear from the paper.
Questions for Authors ?
*Do you plan to release the full code that can reproduce the results in the paper ? Based on my experience, I have found that a lot of active learning results are hard to reproduce and therefore it is difficult to draw any conclusions based on them.
*How sensitive are the results to hyper-parameter tuning and small things like choice of pre-trained word embeddings etc ? How did you select hyper-parameters at each stage during active learning ?


The output of a neural network depends on its parameters in a highly nonlinear way, and it is widely assumed that a network's parameters cannot be identified from its outputs. Here, we show that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability to query the network. ReLU networks are piecewise linear and the boundaries between pieces correspond to inputs for which one of the ReLUs switches between inactive and active states. Thus, first-layer ReLUs can be identified (up to sign and scaling) based on the orientation of their associated hyperplanes. Later-layer ReLU boundaries bend when they cross earlier-layer boundaries and the extent of bending reveals the weights between them. Our algorithm uses this to identify the units in the network and weights connecting them (up to isomorphism). The fact that considerable parts of deep networks can be identified from their outputs has implications for security, neuroscience, and our understanding of neural networks.

This paper introduces an approach to recover weights of ReLU neural networks by querying the network with specifically constructed inputs. The authors notice that the decision regions of such networks are piece-wise linear corresponding to activations of individual neurons. This allows to identify hyperplanes that constitute the decision boundary and find intersection points of the decision boundaries corresponding to neurons at different layers of the network. However, weights can be recovered only up to permutations of neurons in each layer and up to a constant scaling factor for each layer.
The algorithm consists of two parts: Identifying parameters of the first layer and subsequent layers. First they sample a lot of line segments and find their intersections with the decision boundary, i.e. where pre-activations equal 0. Then they sample some points around the intersections and estimate the hyperplanes up to a scaling factor and a sign. For the first layer they check whether some of the hyperplanes belong to it. For the consecutive layers they proceed by moving from the intersection points along the hyperplanes until the decision boundary bends. Again, by identifying which of the bends correspond to the intersection of the current layer's hyperplanes with the previous layers' ones, they are able to recover parameters of the current layer by computing the angles of these intersections.
This paper tackles a very interesting and important problem that might have huge implications for security and many other aspects. However, I'm leaning towards Reject for the following reasons:
1. The algorithm's description is either incomplete or unclear. There are such core functions as PointsOnLine and TestHyperplane, whose pseudo-code would be very helpful for understanding. For example, the authors say that PointsOnLine performs a "binary search," but then this function can find only one (arbitrary) intersection of a line segment with a decision boundary, while each sampled line can intersect multiple ones. If it is not binary search, then the asymptotic analysis given in the end of Sec. 4.2 is incorrect. Even more mysterious is TestHyperplane, from the provided intuition I do not understand how it is possible to distinguish hyperplanes corresponding to the first layer vs. the other layers. In Sec. 4.3, second paragraph, the choice of R is unclear. How to chose it to make sure that the closest boundary intersects it?
The authors consider a very limited setting of only fully-connected (linear) layers with ReLU activations. In this case it is easy to see that the resulting decision boundary is indeed piece-wise linear with a lot of bending. Authors themselves notice, that "the algorithm does not account for weight sharing." For CNN this will lead to each virtual neuron in each channel to have its own kernel weights, although there must be one kernel per channel. Also the authors admit, that pooling layers affect partitioning of the activation regions, making the proposed approach inapplicable. The authors did not discuss whether the proposed approach can handle batchnorm layers. Such non-linear transformations could pose serious problems. All this rules out applications to, for example, all CNN-based architectures, that prevail in computer vision. The authors mention, that their "algorithm holds with slight modification" for ResNets, but as mentioned earlier convolutional, pooling and batchnorm layers make it not so trivial (if at all possible).
2. Experimental evaluation is extremely limited: It is all contained in just one paragraph. Although it is mentioned that "it is often possible to recover parameters of deep ReLU networks," they evaluated their approach on very shallow and narrow networks (only 2 layers, 10 to 50 neurons in each). The immediate question here is why this algorithm is not applied to sufficiently deep NN? At least a network that could classify MNIST reasonably well. Actually, this would be a better proof-of-concept: Given a pre-trained MNIST classifier, apply the proposed method, recover the weights and check if you get the same output as from the original network. Whereas here the evaluation is given as a normalized relative error of the estimated vs. the true weights. Which raises the question of how the scaling factor was chosen? Recall, that the proposed method estimates network's parameters only up to an arbitrary scaling factor. My guess, is that for the Figures 3 and 4 (both right) the estimated weights were re-scaled optimally to minimize the relative error. But in the end, one is interested in recovering the original weights of the network, not relative ones.
I am very confused by Fig. 3 left: Why is the number of queries going down as the number of neurons increases? Should it not be that with more neurons the ambiguity also increases, requiring more queries? Again, this analysis is very limited, it would be very interesting to see, how many more queries one needs for deeper layers of the network. But for this experiments with deeper than 2 layers networks are necessary.
3. The choice of parameters is unclear and not discussed. How long should the line segments be, how many of them. How many points are sampled and within which radius to identify hyperplanes, how to choose 'R'. And how all these choices affect accuracy and performance.
Overall, the paper looks rather incomplete to me and requires a major revision. It will definitely benefit if the "slight modification" for the case of ResNets is included. Also, experimental evaluation should be completely re-done and extended.


We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging unlabeled information in the query set to learn a more powerful meta-model. To develop our framework we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior of each task. We derive a novel amortized variational inference that couples all the variational posteriors into a meta-model, which consists of a synthetic gradient network and an initialization network. The combination of local KL divergences and synthetic gradient network allows for backpropagating information from unlabeled data, thereby enabling transduction. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods.

The authors argue for the importance of transduction in few-shot learning approaches, and augment the empirical Bayes objective established in previous work (estimating the hyperpior $\psi$ in a frequentist fashion) so as to take advantage of the unlabelled test data.
Since the label is, by definition, unknown, a synthetic gradient is instead learned to parameterize the gradient of the variational posterior and tailor it for the transductive setting. The authors provide an analysis of the generalization ability of EB and term their method _synthetic information bottleneck_ (SIB) owing to parallels between its objective of that of the information bottleneck. SIB is tested on two standard few-shot image benchmarks in CIFAR-FS and MiniImageNet, exhibiting impressive performance and outperforming, in some cases by some margin, the baseline methods, in the 1- and 5-shot settings alike, in addition to a synthetic dataset.
The paper is technically sound and, for the most part, well-written, with the authors' motivations and explanation of the method conceived quite straightforwardly. The basic premise of using an estimated gradient to fashion an inductive few-shot learning algorithm into a transductive one is a natural and evidently potent one. The paper does, however, at times feel to be disjointed and, to an extent, lacking in focus. The respective advantages of EB and the repurposing of synthetic gradients to enable the transductive approach are clear to me, yet while they might indeed be obviously complementary, what is not obvious is the necessity of the pairing: it seems there is nothing prohibiting the substitution of the gradient for a learned surrogated just as well under a deterministic meta-initialization framework. As such, despite sporting impressive results on the image datasets, I am not convinced about how truly novel the method is when viewed as a whole.
On a similar note, while the theoretical analysis provided in section 4 was not unappreciated, and indeed it was interesting to see such a connection between EB with information theory rigorously made, it does feel a little out of place within the main text, especially since it is not specific to the transductive setting considered, nor even to the meta-learning setting more broadly. Rather, more experiments, per Appendix C, highlighting the importance of transduction and therein the synthetic gradients and its formulation would be welcome. Indeed, it is stated that an additional loss for training the synthetic gradient network to mimic the true gradient is unnecessary; while I agree with this conclusion, I likewise do not think it would hurt to explore use of the more explicit formulation.
Considering the authors argue specifically for the importance of transduction in the zero-shot learning regime, I think it would be reasonable to expect experiments substantiating this, and the strength of their method in this regard, on non-synthetic datasets. As far as the toy problem is concerned, I am slightly confused as to the choice of baseline, both in the regard to its training procedure and as to why this was deemed more suitable than one purposed for few-shot learning, so that we might go beyond simple verification to getting some initial sense for the performance of SIB. Moreover, it is not clear from the description as to how $\lambda$ is implemented here. As it stands, Section 5, for me, offers little in the way of valuable insights. The experiments section on the whole, results aside, feels somewhat rushed; the synthetic gradients being a potential limiting factor for instance feels "tacked on" and seems to warrant more than just a passing comment.
Minor errors
- Page 7: the "to" in "let $p_\psi(w)$ to be a Gaussian" is extraneous
- Page 8: "split" not "splitted".
- Further down on the same page, "scale" in "scale each feature dimension" should be singular and Drop" is misspelled as "dropp".
- Page 9: "We report results **with using** learning rate..."
- _Equation 17_ includes the indicator function $k \neq i$ but $i$ is not defined within the context.
EDIT: changed score


We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks. Instead of using popular activation functions (such as ReLU), we advocate the use of k-Winners-Take-All (k-WTA) activation, a C 0 discontinuous function that purposely invalidates the neural network model's gradient at densely distributed input data points. The proposed k-WTA activation can be readily used in nearly all existing networks and training methods with no significant overhead. Our proposal is theoretically rationalized. We analyze why the discontinuities in k-WTA networks can largely prevent gradient-based search of adversarial examples and why they at the same time remain innocuous to the network training. This understanding is also empirically backed. We test k-WTA activation on various network structures optimized by a training method, be it adversarial training or not. In all cases, the robustness of k-WTA networks outperforms that of traditional networks under white-box attacks.

This paper addresses the important question of improving the robustness of deep neural networks against adversarial attacks. The authors propose a surprisingly simple measure to improve adversarial robustness, namely replacing typical activation functions such as ReLU with a k-winners-take-all (k-WTA) functions, whereby the k largest values of the input vector are copied, and all other elements of the output vector are set to zero. Since the size of input and output maps varies drastically within networks, the authors instead use a sparsity ratio \gamma that calculates k as a function of input size. k-WTA networks can be trained without special treatment, but for low \gamma values the authors propose a training schedule, whereby \gamma is slowly reduced, then re-training takes place, until the desired value of \gamma is reached. The presented effect is backed up by extensive theoretical investigations that relate the increased robustness to the dense introduction of discontinuities, which makes gradient-based adversarial attacks harder. A small change in an input signal can change the identity of the "winning" inputs, and thus in a sub-sequent matrix multiplication make use of other rows or columns, thus allowing arbitrarily large effects due to small input variations. Empirical evaluations in CIFAR and SVHN for a variety of attacks and defense mechanisms demonstrate the desired effects, and illustrate the loss landscapes due to using k-WTA.
I think the paper is a valuable and novel contribution to an important topic, and is definitely suitable for publication at ICLR. In principle there is just one novel idea, namely using k-WTA activations to improve adversarial robustness, but this claim is investigated thoroughly, in theory, and demonstrated convincingly in experiments. The paper is well written and tries to address all potential questions one might have surrounding the basic idea. There is code available, and the idea should be simple to implement in practice, so I would expect this paper to have large impact on the study of adversarial robustness.
I appreciate the thorough proofs of the claims in section C of the appendix, but I did not review all proofs in  detail.
Potential weaknesses that should be addressed:
1. Section 4: I would propose to fully define A_rob here or at least provide a reference.
2. From Table 1 it seems that using k-WTA leads to a quite noticeable drop in standard accuracy, especially for sparse \gamma, which leads to the best adversarial robustness. Can you please comment on whether the full ReLU accuracy in the natural case can always be recovered by k-WTA networks, e.g. with larger \gamma?
3. Since small changes have a big effect in k-WTA, it should be investigated how robust the k-WTA networks are with respect to more natural perturbations, e.g. noisy input, blurring, translations, rotations, occlusions, etc. as introduced in (Hendrycks and Dietterich, 2019). It would be critical if such perturbations have a stronger effect on k-WTA.
4. Is there any intuition about whether k-WTA should be used everywhere in a deep network, or whether it makes sense to mix k-WTA and ReLU functions?
Minor comments:
5. WTA networks are very popular in computational neuroscience and are even hypothesized to represent canonical microcircuit functions (see e.g. Douglas, Martin, Whitteridge, 1989, Neural Computation, and many follow-up articles). You cite the work of Maass, 2000a,b already, it could be interesting to link your work to other papers in that field who motivate WTA from a biological perspective.


Short-and-sparse deconvolution (SaSD) is the problem of extracting localized, recurring motifs in signals with spatial or temporal structure. Variants of this problem arise in applications such as image deblurring, microscopy, neural spike sorting, and more. The problem is challenging in both theory and practice, as natural optimization formulations are nonconvex. Moreover, practical deconvolution problems involve smooth motifs (kernels) whose spectra decay rapidly, resulting in poor conditioning and numerical challenges. This paper is motivated by recent theoretical advances (Zhang et al., 2017;Kuo et al., 2019), which characterize the optimization landscape of a particular nonconvex formulation of SaSD and give a provable algorithm which exactly solves certain non-practical instances of the SaSD problem. We leverage the key ideas from this theory (sphere constraints, datadriven initialization) to develop a practical algorithm, which performs well on data arising from a range of application areas. We highlight key additional challenges posed by the ill-conditioning of real SaSD problems, and suggest heuristics (acceleration, continuation, reweighting) to mitigate them. Experiments demonstrate the performance and generality of the proposed method.

This paper analyzes some of the optimization challenges presented by a particular formulation of "short-and-sparse deconvolution" and proposes a new general purpose algorithm to try to alleviate them. The paper is reasonably clearly written and the experimental results seem impressive (as a non-specialist in this area). However the experimental investigation on real data does not compare to a baseline, which seems like an essential part of investigating the proposed method. If this were corrected I would recommend an accept.
I would like to make it clear that I have little experience in this area and am not familiar with relevant previous literature, so I was only able to roughly judge novelty of the ideas and how the experimental results might compare with existing approaches.
Major comments:
For the experimental investigations on real data there was no baseline presented. Are there no task-specific algorithms that have previously been developed for deconvolution of calcium signals, fluoresence microscopy and calcium imaging? At the very least it would be helpful to compare to one of the other general purpose methods used for figure 6.
It seemed like a lot of the paper is taken up with a recap of the results presented in Kuo et al. (2019), and it didn't always seem clear exactly what the relevance of these results were to the present paper. In particular it seemed strange to me to devote so much space in section 2 to the ABL objective given that it is not used (as far as I can tell) for the proposed algorithm.
Minor comments:
The abstract says "We leverage... sphere constraints, data-driven initialization". Is the effect of these actually investigated experimentally?
In the abstract, "This is used to derive a provable algorithm" makes it sounds to me like this will be done in the current paper.
In the abstract, a reference for the "due to the spectral decay of the kernel a_0" claim would be helpful.
In the notation section, the definition of the Riemannian gradient seems a little sloppy mathematically: strictly f needs to be defined on $\reals^p$ (or an open subset of $\reals^p$ containing $S^{p - 1}$) in order for the right side of the gradient expression to be defined.
At the start of section 2, it would be helpful to give a one-sentence description of the unifying theme of the section.
It might be helpful to explicitly state that "coherence" is related to the strength of temporal / spatial correlations to aid developing the reader's intuition for the meaning of $\mu(a)$.
Throughout the paper the problems that multiple equivalent local optima (due to shift symmetry) cause come up repeatedly. What happens if this symmetry is removed straightforwardly by adding a term to the cost function to encourage a particular value of $a$ to be the largest?
In section 2.1, for "It analyzes an ABL...", it's not completely clear what "It" refers to (I presume Kuo et. al (2019) from context).
Marginalization in my experience refers to a "partial summing" operation, whereas just above (4) it is used to refer to a "partial maximization" operation. This seems non-standard to me, but is this usage standard in this field?
I didn't understand the relevance of the sentence "Under its marginalization... smaller dimension p << m." to the present paper. The sentence also seemed vague and difficult to understand if you were not already familiar with this result. It should also have a reference to justify this claim.
It wasn't clear to me whether figure 1 were schematic "rough intuition" diagrams intended merely to be suggestive, or provably showing the type of behavior that happens in all cases. Also, what are the axes, generic "parameter space", I guess? I also didn't follow why (a) appears projected on to a plane while (b) and (c) appear projected on to a sphere(?)
In section 3, under "momentum acceleration", it would be helpful to justify the claim that "In shift-coherent settings, the Hessian... ill-conditioned...".
In figure 4, the axis labels are much too small to read. In figure 5 (b), the red poorer results obscure the green better results.
In figure 4 (d), is it fair to say that the main convergence speed improvement for homotopy-iADM is in going faster from "quite near" the optimum to "really near" it, rather than getting "quite near" it in the first place? If so, isn't the latter more often what's relevant in practical applications?
In figure 5, it's a little confusing to switch color meanings between (a) and (b).
Reweighting seems to have a dramatic positive effect in figure 5. Is it worth investigating its effect in the other experiments as well, for example in figure 4?
In figure 6 (b), is there any reason not to compare against standard black box optimizers like vanilla SGD, ADAM, etc (possibly with projection to satisfy the sphere constraint if necessary)? Would they perform very badly?
Typo "Whilst $a_0$ nor $x_0$" should be "Whilst $a_0$ and $x_0$".
What does the square-boxed convolution operator in (8) mean?


In environments with continuous state and action spaces, state-of-the-art actor-critic reinforcement learning algorithms can solve very complex problems, yet can also fail in environments that seem trivial, but the reason for such failures is still poorly understood. In this paper, we contribute a formal explanation of these failures in the particular case of sparse reward and deterministic environments. First, using a very elementary control problem, we illustrate that the learning process can get stuck into a fixed point corresponding to a poor solution. Then, generalizing from the studied example, we provide a detailed analysis of the underlying mechanisms which results in a new understanding of one of the convergence regimes of these algorithms. The resulting perspective casts a new light on already existing solutions to the issues we have highlighted, and suggests other potential approaches.

The paper investigates why DDPG can sometimes fail in environments with sparse rewards. It presents a simple environment that helps the reader build intuition and supports the paper's empirical investigation. First, the paper shows that DDPG fails on the simple environment in ~6% of cases, despite the solution being trivial—go left from the start state. The paper then augments DDPG with epsilon-greedy-style exploration to see if the cause of these failures is simply inadequate exploration. Surprisingly, in 1% of cases DDPG still fails. The paper shows that even in these failure cases there were still rewarded transitions that could have been learned from, and investigates relationships between properties of individual runs and the likelihood of failure. The paper then explains how these failures occur: the policy drifts to always going right, and the critic converges to a piecewise constant function whose gradient goes to zero and prevents further updates to the policy. The paper then generalizes this deadlock mechanism to other continuous-action actor-critic algorithms like TD3 and discusses how function approximation helps mitigate this issue. Finally, the paper gives a brief overview of some existing potential solutions.
Currently, I recommend rejecting this paper; while it is very well-written and rigorously investigates a problem with a popular algorithm, the paper does not actually present a novel solution method. It does a great job of clearly defining and investigating a problem and shows how others have attempted to solve it in the past, but stops there. It doesn't propose a new method and/or compare the existing methods empirically, nor does it recommend a specific solution method.
A much smaller concern is that the problem investigated is somewhat niche; it happens in a very small percentage of runs and is mitigated by the type of function approximation commonly used. This lowers its potential impact a little.
The introduction does a great job of motivating the paper. I would've liked the related work section to elaborate more on the relationships between the insights in this paper and those of Fujimoto et al. (2018a), since the paper says the insights are related. Section 3 gave a very clear overview of DDPG. The simple environment described in section 4 was intuitive and explained well, and the empirical studies were convincing.
However, after section 4 established the existence of the deadlock problem in DDPG, I was expecting the rest of the paper to present a solution method and empirically validate it. Instead, section 5 generalizes the findings from section 4 to include other environments and algorithms. I felt that section 4 and 5 could have been combined and shortened, and the extra space used to explain and empirically test a novel solution method. For example, figures 6 and 7 seem to be conveying similar information, but figure 7 takes up almost half a page.
Currently this paper seems like a good candidate for a workshop. It convincingly establishes the existence of a problem and shows what causes the problem to occur, but doesn't contribute a solution to the problem. Submitting it to a workshop could be a good opportunity for discussion, feedback, and possible collaboration, all of which might help inspire a solution method.


Learning theory tells us that more data is better when minimizing the generalization error of identically distributed training and test sets. However, when training and test distribution differ, this distribution shift can have a significant effect. With a novel perspective on function transfer learning, we are able to lower bound the change of performance when transferring from training to test set with the Wasserstein distance between the embedded training and test set distribution. We find that there is a trade-off affecting performance between how invariant a function is to changes in training and test distribution and how large this shift in distribution is.

The authors consider the relation between Frechet distance of training and test distribution and the generalization gap. The authors derive the lower bound for the difference of loss function w.r.t. training and test set by the Wasserstein distance between embedding training and test set distribution. Empirically, the authors illustrate a strong correlation between test performance and the distance in distributions between training and test set.
The motivation to find the relation between generalization gap and the Frechet distance of training and test distribution is sound. However, I am not sure that the lower bound as in Equation (1) is enough. I am curious that one can derive the upper bound for the relation or not. The finding about choosing a training data distribution should be close to the test data distribution seems quite trivial in some sense. I am not clear about its important since it is quite popular that the distribution shift affects the performance and many learning approach assumes same distribution for training and test data. Overall I feel that the contribution may be quite weak, and I lean on the negative side.
Below are some of my concerns:
1) About the lower-bound in Equation (1), it seems unclear to me that when the W_2(p1, p2) = 0, we can inference any information about the test performance (It seems quite trivial for this case, the left hand side time is greater than or equal 0?) In my opinion, the upper-bound is more important which one can inference much information about the difference of generalization gap.
2) In the proof of Theorem 1, it is quite hard to follow with the current notation, for the integral in (i), (ii) as well as in the proof using the intermediate value theorem, which variables are used? I am confused which one is variable, which one is constants in those integrals.
3) In page 5, at the interpretation (1), for W2(p1, p2) = 0, the learned function fits training distribution perfectly and is not ill-conditioned ==> why one can deduce that the test distribution is fit perfectly? What we have in Theorem 1 is the lower-bound only?


Generative adversarial networks (GANs) are a widely used framework for learning generative models. Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax optimization problem to global optimality, but are in practice successfully trained using stochastic gradient descent-ascent. In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution with polynomial time and sample complexity.

I have read the authors response. In the response the authors clarified the contributions of this paper. I agree with the authors that the analysis of gradient descent-ascent is a difficult problem, and the optimization results given in this paper is a contribution of importance. Because of this I have improved my score.
However, I do not agree with the authors that studying quadratic discriminators instead of more complicated ones should be considered as a contribution instead of drawback. In my opinion, as long as the focus is on WGAN, results involving standard neural networks are still more desired compared with the results in this submission. For example, similar results for a neural network discriminator might be even more impactful, because the optimization problem is even more difficult. Therefore I still consider the simple discriminator and generator as a weak point of this paper.
======================================================================================================
This paper studies the training of WGANs with stochastic gradient descent. The authors show that for one-layer generator network and quadratic discriminator, if the target distribution is modeled by a teacher network same as the generator, then stochastic gradient descent-ascent can learn this target distribution in polynomial time. The authors also provide sample complexity results.
The paper is well-written and the theoretical analysis seems to be valid and complete. However, I think the WGANs studied in this paper are simplified too much that the analysis can no longer capture the true nature of WGAN training.
First, the paper only studies linear and quadratic discriminators. This is not very consistent with the original intuition of WGAN, which is to use the worst Lipschitz continuous neural network to approximate the worst function in the set of all Lipschitz continuous functions in the definition of Wasserstein distance. When the discriminator is as simple as linear or quadratic functions, there is pretty much no “Wasserstein” in the optimization problem.
Moreover, the claim that SGD learns one-layer networks can be very misleading. In fact what is a “one-layer” neural network?
- if the authors meant “two-layer network” or “single hidden layer network”, then this is not true. Because as far as I can tell, the model $x = B \phi(A z)$ is much more difficult than the model $x = \phi(A z)$. The former is a standard single hidden layer network which is non-convex, while the latter is essentially a linear model especially when \phi is known.
- if the authors meant “a linear model with elementwise monotonic transform”, then I would like to suggest that a more appropriate name should be used to avoid unnecessary confusion.
As previously mentioned, the discriminators are too simple to approximate the Wasserstein distance, and therefore in general it should not be possible to guarantee recovery of the true data distribution. However, in this paper it is still shown that certain true distributions can be learned. This is due to the extremely simplified true model. In fact, even if the activation function $\phi$ is unknown, it seems that one can still learn $A^* (A^*)^\top$ well (for example, by Kendall’s tau).


To deflect adversarial attacks, a range of "certified" classifiers have been proposed. In addition to labeling an image, certified classifiers produce (when possible) a certificate guaranteeing that the input image is not an p -bounded adversarial example. We present a new attack that exploits not only the labelling function of a classifier, but also the certificate generator. The proposed method applies large perturbations that place images far from a class boundary while maintaining the imperceptibility property of adversarial examples. The proposed "Shadow Attack" causes certifiably robust networks to mislabel an image and simultaneously produce a "spoofed" certificate of robustness.

The paper proposes a new way to generate adversarial images that are perturbed based on natural images called Shadow Attach. The generated adversarial images are imperceptible and have a large norm to escape the certification regions. The proposed method incorporates the quantities of total variation of the perturbation, change in the mean of each color channel, and dissimilarity between channels, into the loss function, to make sure the generate adversarial images are smooth and natural. Quantitative studies on CIFAR-10 and ImageNet shows that the new attack method can generate adversarial images that have larger certified radii than natural images. To further improve the paper, it would be great if the authors can address the following questions:
- In Table 1, for ImageNet, Shadow Attach does not always generate adversarial examples that have on average larger certified radii than the natural parallel, at least for sigma=0.5 and 1.0. Could the authors explain the reason?
- In Table 2, it is not clear to me what is the point for comparing errors of the natural images (which measures the misclassification rate of a natural image) and that of the adversarial images (which measures successful attacks rate), and why this comparison helps to support the claim that the attack results in a stronger certificates. In my opinion, to support the above claim, shouldn’t the authors provide a similar table as Table 1, directly comparing the certified radii of the natural images and adversarial images?
- From Figure 9, we see the certificate radii of the natural have at least two peaks. Though on average the certificate radii of the adversarial attacks is higher than that of the natural images, it is smaller than the right peak. Could the authors elaborate more of the results?
- Sim(delta) should be Dissim(delta) which measures the dissimilarity between channels. A smaller dissimilarity suggests a greater similarity between channels.
- Lambda sim and lambda s are used interchangeably. Please make it consistent.
- The caption of Table 1 is a little vague. Please clearly state the meaning of the numbers in the table.


Biological and artificial agents must learn to act optimally in spite of a limited capacity for processing, storing, and attending to information. We formalize this type of bounded rationality in terms of an information-theoretic constraint on the complexity of policies that agents seek to learn. We present the Capacity-Limited Reinforcement Learning (CLRL) objective which defines an optimal policy subject to an information capacity constraint. This objective is optimized by drawing from methods used in rate distortion theory and information theory, and applied to the reinforcement learning setting. Using this objective we implement a novel Capacity-Limited Actor-Critic (CLAC) algorithm and situate it within a broader family of RL algorithms such as the Soft Actor Critic (SAC) and discuss their similarities and differences. Our experiments show that compared to alternative approaches, CLAC offers improvements in generalization between training and modified test environments. This is achieved in the CLAC model while displaying high sample efficiency and minimal requirements for hyper-parameter tuning.

The authors propose capacity-limited reinforcement learning and apply an actor-critic method (CLAC) in some continuous control domains. The authors claim that CLAC gives improvements in generalization from training to modified test environments, and that it shows high sample efficiency and requires minimal hyper-parameter tuning.
The introduction started off making me think about this area in a new way, but as the paper continued I started to find some issues. To begin with, I think the motivation in the introduction could be improved. Why would I choose to limit capacity? This is not sufficiently motivated. I suspect that the author(s) want to argue that it *should* give better generalization, but this argument is not made very clearly in the introduction. Perhaps this is because it would be difficult to make this argument formally, and so it is merely suggested at?
Are there connections between this and things like variational intrinsic control (VIC, Gregor et al. 2016) and diversity is all you need (DIAYN, Eysenbach et al., 2019)? These works aim to maximize the mutual information between latent variable policies and states/trajectories, whereas this work is really doing the opposite. I would be interested in understanding the author’s take on how the two are related conceptually.
Moving to the connections with past work, this paper seriously abuses notation in a way that actually hinders comprehension. Some of the parts that really bothered me, and should be fixed to be correct:
Mutual information is a function of two random variables, whereas it is repeatedly expressed as a function of the policy. Being explicit about the random variables / distribution here is pretty important.
In Equation 2 (and subsequent paragraph) the marginal distributions p_a(a) and p_s(s) are not well defined, marginalizing over what, what are these distributions? I might guess that p_s(s) is the steady state distribution under a policy pi, and that p_a(a) is marginalizing over the same distribution, essentially capturing the prior probability of each action under the policy. But these sort of things need to be said explicitly.
In KL-RL section there is a sentence with “This allows us to define KL-RL to be the case where p_0(a, s) = \pi_0(a_t | s_t).” What does this actually mean? One of these is a joint probability for state and action, and one is an action probability conditional on a state.
What does \pi_\mu(a_t) \sim \mathcal{D} mean?
In the block just before Algorithm 1, many of these symbols are never defined. This needs a significant amount of care (by the authors) and right now relies on the reader to simply make a best guess at what the authors probably intend.
Overall in the first three sections the message I would like the authors to understand is that, in striving for a concise explanation they have significantly overshot. These sections require some significant work to be considered publishable.
The experiment in section 4.1 is intended to give a clean intuitive understanding of the method, but falls a bit short here. It is clean, but I needed more explanation to really drive the intuition home. I see that CLAC finds a solution more sensitive to the beta distribution, but help me understand why this is the right solution in this particular case.
I really disagree with the conclusions around the experiments in section 4.2. I do not think these results show that for the CLAC model increasing the mutual information coefficient increases performance on the perturbed environments. First, the obvious, how many seeds and where are the standard deviations? Second, the trend is extremely small and the gap between CLAC and SAC is just as minor. Finally, CLAC has better performance on the training distribution which means that it actually lost *more* performance than SAC when transferring to the testing and extreme testing distributions.
The results for section 4.3 are just not significant enough to draw any real conclusions. The massive temporal variability makes me very suspicious of those super tight error bands, but even without that question, the gap is just not very large.
Finally, in section 4.4 we see the first somewhat convincing experimental results. These look reasonable, but even here I have a fairly pointed question: compared with the results in Packer et al (2018) the amount of regression from training to testing is extremely large (whereas they found vanilla algorithms transfer surprisingly well). Can you explain why there is such a big discrepancy between those results and these? But again, this section’s results are in my opinion the most convincing that something interesting is happening here.
Lastly, in section 8.1 the range of hyper-parameters for the mutual information coefficient is very broad, which really makes it hard to buy the claim of requiring minimal hyper-parameter tuning.
All in all there is something truly interesting in this work, but in the present state I am unable to recommend acceptance, and the amount of work required along with questions raised lead me to be fairly confident in this assessment.


We present BlockBERT, a lightweight and efficient BERT model that is designed to better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training time, which also enables attention heads to capture either short-or long-range contextual information. We conduct experiments on several benchmark question answering datasets with various paragraph lengths. Results show that BlockBERT uses 18.7-36.1% less memory and reduces the training time by 12.0-25.1%, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

The authors propose BlockBERT, a model that makes the attention matrix of Transformer models sparse by introducing block structure. This has the dual benefit of reducing memory and reducing training time. The authors show on various question answering tasks that their model is competitive with RoBERTa.
1. Can the authors add additional details about the training of their model in Section 4.1? It is not clear for example the vocabulary size - is that the RoBERTa vocabulary size of around 50k or the BERT vocabulary size that is smaller? I believe this will affect the memory and training speed.
2. For the datasets such as TriviaQA and SearchQA, how is RoBERTa finetuned on these tasks? By doing the window approach?
3. The authors compare to RoBERTa and Sparse BERT as baselines for the section on performance. However, can the authors also include metrics on training time and memory in Table 2 for Sparse BERT as well as other sparse attention transformer architectures proposed (for example the Correia paper or the Sukhbaatar paper)? It is not clear the savings from this architecture compared to sparse Transformers in general.
4. The analysis in Section 4.3 is quite unclear due to how compressed the description is and how tiny the graphs are.
5. The authors mention that attention heads can be sparsified due to the memory usage and quadratic time complexity. Other work has also shown that the attention heads are quite redundant and can be pruned away, so attention head dropout is effective. For example https://arxiv.org/abs/1905.09418


Many defenses for Convolutional Neural Networks are based on a simple observation that the adversarial examples are not robust and small perturbations to the attacking input often recover the desired prediction. Intuitively, the adversarial examples occupy a very small cone in the decision space which is surrounded by a large area corresponding to the correct class. While the intuition is simple, a detailed understanding of this phenomenon is missing from the research literature. We identify a family of defense techniques that are based on the instability assumption. The defenses include deterministic lossy compression algorithms and randomized perturbations to the input that all lead to similar gains in robustness. We present a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

The paper studies the robustness of adversarial attacks to transformations of their output. Specifically, for standard methods for crafting adversarial examples, the authors evaluate whether the crafted examples remain adversarial under (stochastic or deterministic) transformations such as Gaussian noise and SVD compression. The authors argue that different transformations have a similar impact on the perturbed inputs. Finally, they argue that the reason why these transformations can sometimes recover the correct label is due to the loss being more unstable at these points (from a first- and second-order pespective).
From a conceptual point of view, I did not find the paper particularly impactful. In particular, I can identify three claimed contributions:
a) The L2 distortion introduced by these transformation might have more impact than the specific transformation used (figure 1).
Firstly, I would argue that this effect is most prominent for small L2 distortions (distortions larger than 5/40 do exhibit noticeable differences). Secondly, I am not sure why one would expect these transformations to have fundamentally different impact. The adversarial attacks considered manipulate the input using first-order methods in input space. Since the transformations are model- and data-agnostic it seems expected that the primary mechanism behind their effect is the pixel-wise distortion of the image.
b) Adversarial perturbations that are robust to one type of transformation tend to also be robust to other transformations (table 2).
This is probably the most interesting observation of the paper, indicating that attackers can bypass several of these transformation defenses by only aiming to be robust to a subset of them. At the same time, I am not sure what the impact of this observation is given that we already know how to bypass most of these defenses anyway.
c) The instability of adversarial perturbations can be explained by a larger gradient norm and Hessian spectrum (figure 2, 3).
First of all, I do not understand how this is considered a potential explanation of empirical behavior. If gradient norm is indicative of instability, then _natural_ images would be unstable (since the gradient wrt the wrong class is large). Furthermore, instability does not explain why adding noise leads to the _correct_ class as opposed to a random class. Perhaps most importantly, from what I understand (also skimming the code), Figure 2 plots the gradient of the _cross-entropy loss_ with respect to each class. However, the norm of the gradient is directly affected by the softmax probability of each class. Hence, for natural images, the probability of the correct class is high leading to a small norm while for the adversarial class it is low leading to large norm. Based on this reasoning, this plot does convey information about the classifier's stability but rather about the softmax probabilites assigned to each class.
In summary, after reading the paper, I am not sure how our understanding of transformation robustness has changed or what insight we have gained that will help us design future attacks and defenses (especially given prior work on how most of these defenses can be bypassed). I thus recommend rejection.
Comments to authors:
-- Many of the attack details are missing (what is the epsilon allowed for PGD? what does the "c" parameter correspond to for CW attacks?) which prevented me from fully comprehending the experimental results.
-- LBFGS is not a particular attack, it is a general optimization algorithm, https://en.wikipedia.org/wiki/Limited-memory_BFGS
-- The fact that adversarial points are close to correctly classified points does not mean that a small amount of random noise will change the classifier prediction. In high dimension, the distance to a hyperplane can be epsilon but moving in a random direction requires movement of epsilon * sqrt(number of dimensions) .
==============
UPDATE: I appreciate the authors' response. As stated in my original review, I do recognize the performance of a non-adaptive adversary as a contribution. However, I still don't think that the impact of this finding is significant enough for publication at ICLR. I hence keep my original score.
Response to specific points:
-- Unfortunately, the authors did not address my concerns about the gradient of the loss. The experiments and reasoning of the paper should be sufficient without citing other work in the author response. Moreover, it is unclear if the papers referenced in the response are showing fundamental properties of adversarial examples or are specific to the attacks considered.  For instance, there has been work challenging these papers (https://arxiv.org/abs/1907.12138).
-- As mentioned in my original review, measuring the norm of the gradient of the softmax loss will inevitably take into account the predicted probability of each class. Hence in order to truly measure sensitivity it would be important to work with the logits instead of the softmax probabilities.


We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept "disentangled representations" used in supervised and unsupervised methods along three dimensionsinformativeness, separability and interpretability-which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability, and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.

Summary:
The paper presents a new set of metrics for evaluating disentangled representations in both supervised and unsupervised settings. Disentangled representations are evaluated along three dimensions: informativeness, separability, and interpretability. While previous work offers metrics for similar dimensions (e.g., (Eastwood & Williams, 2018)), the paper suggests that the metrics of the submission are superior to (Eastwood & Williams, 2018), based on a comparison between FactorVAE and Beta-VAE.
Strengths:
(1) The metrics of the paper are well motivated from an information-theoretic standpoint. While, in some cases, the metrics themselves are straight applications of information theory (e.g., informativeness metric == mutual information), the authors came up with new metrics when existing information-theoretic definitions could be fooled by, e.g., introducing noisy and independent latent representations z.
(2) Experimental results show that these metrics are better than previous metrics at discriminating between FactorVAE and Beta-VAE (in an experimental setting where the former is clearly superior to the latter).
(3) As opposed to much of prior work, these metrics do not require any training, which can be considered a plus as they do not require adaptation for each (sub)domain.
(4) The paper is overall well written and clear. I very much enjoyed reading it.
Weaknesses:
(1) My main concern with the paper, which makes me vote for “weak accept” instead of “accept”: The metrics of the paper are compared against previous metrics (Eastwood & Williams, 2018) on *only two* types of disentangled representations, namely FactorVAE and Beta-VAE, and in one experimental condition. (There is plenty of other experimental material in the appendix which also introduces AAE, but none of it seems to compare against previous *metrics*). I think comparing metrics on only two systems is somewhat problematic, and we don’t know how general the results are. I would have preferred to see FactorVAE and Beta-VAE evaluated with fewer hyperparameter choices to allow introducing more variational approaches. Could you (the authors) at least evaluate AAE against Eastwood & Williams too? This concern is not just mere quibbling, as the authors have shown (e.g., in Section 3.2) that specific edge cases can fool naïve metrics (e.g., by introducing noisy and independent latent variables), and there may be other edge cases that the authors have not considered. I think evaluating new metrics against previous work (i.e., metrics) on only two underlying systems is not very convincing. I acknowledge that the paper offers *lots* of experiments – the full pdf is 30 pages (!) – but I think some of the existing ones could go to leave space for evaluations using more underlying VAE/baselines/edge-case systems.
(2) The paper presents six metrics (MI, MISJED, WSEPIN, WINDIN, RMIG, JEMMIG) along three dimensions. While each individual metric makes sense, I feel the paper lacks a discussion section that ties these pieces together and suggests a way of using these metrics conjointly across the three dimensions towards building better-disentangled representations (the paper has a “discussion” section, but it is very short and actually more of a conclusion). The more metrics we have, the more chances each underlying VAE model can “win” on one of the metrics, which is not particularly enlightening.
(3) The paper is not self-contained, and some parts are almost impossible to understand without familiarity with (Kim & Mnih, 2018) and (Higgins et al., 2017a). It uses technical terms of these papers without explanations (e.g., TC is not even spelled out it seems). Hyperparameters of these papers (e.g., beta, gamma) are used without explanation.
(4) There is no related work section. Such a section could, e.g., make what is borrowed from (Kim & Mnih, 2018; Higgins et al., 2017a) more understandable.
Overall, I think it is a nice paper that makes significant contributions to the problem of building better disentangles representations thanks to better evaluation metrics. The empirical support for some of the claims (e.g., superiority to (Eastwood & Williams, 2018)) is a bit weak, but other strengths mentioned above largely make up for that.
Minor comments:
The definitions of SEPIN@k and INDIN@k don’t seem quite right. The summation iterates over z_0, …, z_k-1, leaving off z_k, …, z_L-1, but the latter variables might contain some z’s with lowest mutual information with x. The way the ‘sorted’ function is written only has the effect of reordering z_0, …, z_{k-1} among themselves, never considering any of the z_k and above whatever their MI’s are, which is probably not what the authors meant. Perhaps ‘sorted’ is intended to both sort and rename z’s, but if z’s are indeed renamed I think this should be indicated mathematically otherwise the equation is wrong (e.g., (z’_1, …, z’_{L-1}) = sorted_by_MI(z_1, …, z_{L_1})).
Typos in WSEPIN and WINDIN definitions: “0 = 1” -> “i = 1”?
Figure 1: “Image”? The paper is written in general terms and doesn’t seem to assume otherwise x is an image representation (or does it?).
Section 3.4: “Table. 1” -> “Table 1”
Table 1 vs. title of Section 3.1: The author’s informativeness metric doesn’t have a name, but the section title contains “informativeness,” which could be confusing vs. “informativeness” in Table 1 which is a completely different metric.
Page 8, interpretability: “TC=10”. Do you mean “TC loss” here? I presume 10 is the value of a hyperparameter of the FactorVAE paper (gamma?), and not the actual value of the TC term. Same question for Figure 9 and later figures of the appendix.


Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order recurrent models. However, such a model requires a large number of parameters and operations, making it intractable to learn and prone to overfitting in practice. In this work, we propose Convolutional TensorTrain LSTM (Conv-TT-LSTM), which learns higher-order Convolutional Long Short-Term Memory (ConvLSTM) efficiently using Convolutional Tensor-Train Decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information with low memory and computational requirements by efficient low-rank tensor-train representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and other ConvLSTM-based approaches, but with much fewer parameters.

This paper proposed a convolutional tensor-train (CTT) format based high-order and convolutional LSTM approach for long-term video prediction. This paper is well-motivated. Video data usually have high dimensional input, and the proposed method aims to explicitly take into account more than one hidden representation of previous frames - both lead to a huge number of parameters. Therefore, some sort of parameter reduction is needed. This paper considers two different types of operations - convolution and tensor-train (TT) decomposition - in an interleaved way. The basic model considered in this paper is a high-order variant of convolutional LSTM (convLSTM).

There exist several works using tensor decomposition methods including TT to compress a fully connected layer or a convolutional layer in neural nets, to break the memory bottleneck and accelerate computation. This paper takes a different direction - it further embeds convolution into the TT decomposition and thus defines a new type of tensor decomposition, termed convolutional tensor train (CTT) decomposition. CTT is used to represent the huge weight matrices arisen in the high-order convLSTM. To my best knowledge, this combination of convolution and TT decomposition is new.

The paper is well-written as the literature review is well done. Experimental results demonstrate improved performance over the convolutional LSTM baseline, a fewer number of parameters, and the qualitative results show sharp and clean digits. This improvement could be attributed to multiple causes: the high-order, the tensor decomposition-based compression, or the CTT. The authors also provide an ablation study, but it mainly concerns comparisons with ConvLSTM.

Despite the promising results, this paper is not ready for ICLR yet. Below is a list of suggested points needed to address:
(1) Yang et al 2017 claim that TT-RNN without convolution can also capture spatial and temporal dependence patterns in video modeling. This is an important baseline but missing in the current version of the paper.
(2) The justification of high-order modeling in long-term prediction. The first-order model also implicitly aggregates multiple past steps. It would be good to add more experimental evidence to support the necessity of the high-order.
(3) There exists some unjustified complexity for the CTT approach. How does it compare to TT for high-order ConvLSTM?

Perhaps, a more complete ablation study should include:
(1) LSTM with TT but without high-order and convolution
(2) LSTM with high-order and TT but without convolution
(3) ConvLSTM with TT
(4) ConvLSTM with CTT
(5) ConvLSTM with high-order and TT
(6) ConvLSTM with high-order and CTT

Question:
• How is the backpropagation done for the CTT core tensors?
• What is the error propagation issue of first-order methods and how does the high-order one not prone to it?


The training of stochastic neural network models with binary (±1) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to ±1, for deeper networks to be trainable.

The paper provides an in-depth exploration of stochastic binary networks, continuous surrogates, and their training dynamics with some potentially actionable insights on how to initialize weights for best performance. This topic is relevant and the paper would have more impact if its structure, presentation  and formalism could be improved. Overall it lacks clarity in the presentation of the results, the assumptions made are not always clearly stated and the split between previous work and original derivations should be improved.
In particular in section 2.1, the author should state what exactly the mean-field approximation is and at which step it is required (e.g. independence is assumed to apply the CLT but this is not clearly stated). Section 3 should also clearly state the assumptions made. That section just follows the “background” section where different works treating different cases are mentioned and it is important to restate here which cases this paper specifically considers. Aside from making assumptions clearer, it would be helpful to highlight the specific contributions of the paper so we can easily see the distinctions between straightforward adaptations of previous work and new contributions.
Specific questions:
It might be worth double checking the equation between eq. (2) and eq. (3) , the boundary case (l=0) does not make sense to me, in particular what is S^0 ?.
What does the term hat{p}(x^l) mean in the left hand side of eq.(3)?
In eq. (7) (8) why use the definition symbol := ?
At the beginning of section 3.1, please indicate what “matcal(M)” precisely refers to. Using the term P(mathcal(M) = M_ij) does not make much sense if the intent is to use a continuous distribution for the means.
Just after eq. (9), please explain what Xi_{c*} means.
Small typo: Eq. (10) is introduced as “can be read from the vector equation 31”, what is eq. (31)?
In section 5.2, why reducing the training set size to 25% of MNIST?


Robustness verification that aims to formally certify the prediction behavior of neural networks has become an important tool for understanding the behavior of a given model and for obtaining safety guarantees. However, previous methods are usually limited to relatively simple neural networks. In this paper, we consider the robustness verification problem for Transformers. Transformers have complex self-attention layers that pose many challenges for verification, including cross-nonlinearity and cross-position dependency, which have not been discussed in previous work. We resolve these challenges and develop the first verification algorithm for Transformers. The certified robustness bounds computed by our method are significantly tighter than those by naive Interval Bound Propagation. These bounds also shed light on interpreting Transformers as they consistently reflect the importance of words in sentiment analysis.

Summary:
This paper builds upon the CROWN framework (Zhang et al 2018) to provide robustness verification for transformers. The  CROWN framework is based upon the idea of propagating linear bounds and has been applied to architectures like MLP, CNNs and RNNs. However, in Transformers, the presence of cross-nonlinearities and cross-position dependencies makes the backward propagation of bounds in CROWN computationally intensive. A major contribution of this paper is to use forward propagation of bounds in self attention layers along with the usual back-propagation of bounds in all other layers. The proposed method provides overall reduction in computational complexity by a factor of O(n). Although the fully forward propagation leads to loose bounds, the mixed approach (forward-backward) presented in this work provides bounds which are as tight as fully backward method.
Strength:
Use of forward propagation to reduce computational complexity is non-trivial
Strong results on two text classification datasets:
Lower bounds obtained are significantly tighter than IBP
The proposed method is an order of magnitude faster than fully backward propagation, while still maintaining the bounds tight.
Weakness:
Experiments only cover the task of text classification. Experiments on other tasks utilizing transformers would have made the results stronger.
The paper makes the simplifying assumption that only a single position of an input sentence will be perturbed.  They claim that generalizing to multiple positions is easy in their setup but that is not supported.   The paper needs to declare this assumption early on in the paper (abstract and intro). As far as I could tell, even during the experiments they perturb a single word at a time.
The paper is more technical than insightful.  I am not at all convinced from a practitioners viewpoint that such bounds are useful.  However, given the hotness of this topic, someone or the other got to work out the details.   If the math is correct, then this paper can be it.
The presentation requires improvement.  Some parts, example, the Discussion section cannot be understood.
Questions:
The set-up in the paper assumes only one position in the input sequence is perturbed for simplicity. Does the analysis remain the same when multiple positions are perturbed?
Suggestions:
A diagram to describe the forward and backward process would significantly improve the understanding of the reader.
In Table 3, I am surprised that none of the sentiment bearing words were selected as the top-word by any of the methods.  Among the ‘best’ words,  the words chosen by their method does not seem better than those selected by the grad method.
Several typos in the paper: spelling mistakes in “obtain a safty guarantee”, poor sentence construction in “and independent on embedding distributions”, subject verb disagreement in “Upper bounds are discrete and rely on the distribution of words in the embedding space and thus it cannot well verify”.
I have not verified the math to see if they indeed compute a lower bound.


ML algorithms or models, especially deep neural networks (DNNs), have shown significant promise in several areas. However, recently researchers have demonstrated that ML algorithms, especially DNNs, are vulnerable to adversarial examples (slightly perturbed samples that cause mis-classification). Existence of adversarial examples has hindered deployment of ML algorithms in safety-critical sectors, such as security. Several defenses for adversarial examples exist in the literature. One of the important classes of defenses are manifold-based defenses, where a sample is "pulled back" into the data manifold before classifying. These defenses rely on the manifold assumption (data lie in a manifold of lower dimension than the input space). These defenses use a generative model to approximate the input distribution. This paper asks the following question: do the generative models used in manifold-based defenses need to be topology-aware? Our paper suggests the answer is yes. We provide theoretical and empirical evidence to support our claim.

I. Summary of the Paper
This paper studies robustness to adversarial examples from the
perspective of having 'topology-aware' generative models. Next
to some experiments on data sets with a manifold structure, the
main contribution of the paper is a tandem of theorems that state
the conditions under which models can recover the topology---or
the number of connected components---of a data set correctly,
thereby making them more robust to adversarial examples.
II. Summary of the Review
This paper provides a novel perspective on adversarial examples through
the lens of Riemannian Geometry and topology. I appreciate novel
research that employs topology-based methods, but at present, I cannot
fully endorse accepting the paper. Specifically, I see the following
issues:
- Missing clarity: while the appendix is very comprehensive, which
  I appreciate, the main text could be improved; some statements appear
  redundant, while others need to be re-formulated to build intuition
- This paper appears to span both theory and applications. I appreciate
  this attempt, knowing full well that this is no easy feat to
  accomplish. However, the main theoretical result on the number of
  connected components only applies to mixtures of Gaussian
  distributions, but the purported scope of the paper is the analysis of
  manifold-based defences in general. I would expect a more in-depth
  discussion of the limitations of the theorem. Can we expect this to
  generalise? Moreover, 'topology' is reduced to 'connected components'
  in this paper. While this is perfectly adequate in the sense of
  connected components being a particular concept from topology, I would
  expect this to be clarified much earlier in the paper. In addition,
  connected components are a very basic and coarse concept, so I am
  wondering to what extent it is sufficient to describe models purely
  based on that information.
- As a sort of corollary to the previous point, the experiments could be
  improved. I like the idea of employing known data sets with a simple
  manifold structure, but the setup is somewhat preliminary; I would
  prefer to see an analysis of border cases or limit cases in which the
  theorem _almost_ applies (or not); plus, a more in-depth analysis of
  stochastic effects during training: do _all_ models end up being
  robust if their number of connected components is sufficiently large?
  Is there a dependency between the number of connected components and
  vulnerabilities---are models with a very small number of connected
  components more vulnerable than models with a very larger number of
  connected components? The present experimental section is lacking this
  depth.
- The same statements apply to the INC example. I found this super
  instructive, but it is only _one_ case on _one_ manifold---I would
  like to see more details here; maybe some of the experiments in the
  appendix could be moved to the front? I have some suggestions for
  shortening the paper (see below).
Despite these issues, I think this paper can be a strong contribution if
properly revised; since I am positive that at least some of these
suggestions could be performed within a revision cycle, I want to be
upfront and state that I will definitely consider raising my score,
provided that my concerns are addressed appropriately!
I have to state that I am _not_ an expert in adversarial examples, but
an expert in topology-based methods; I consider this paper to belong to
the latter field given its theoretical contributions about 'recovering'
the correct density distribution.
III. Clarity
The paper provides an extensive background to Riemannian geometry, which
I appreciated as a reference. Nevertheless, there are improvements to
the main text that I would suggest:
- Please consider changing the title to 'On the need...'
- The manifold assumption is that data lie _on_ a manifold or _close to_
  a manifold whose intrinsic dimension is much lower than that of the
  ambient space. This is not stated in sufficient precision in the
  paper; please correct the usage on p. 1 and p. 2
- In terms of notation, why use $p_M$ to denote the density on the whole
  of $\mathds{R}^n$? I would expect $p_M$ to refer to the density on $M$
  rather than the density of the whole space.
- If $M$ is a disjoint union of manifolds, please consider using
  a '\cupdot' operator to make this more clear.
- If the pairwise manifolds are disjoint, how can the resulting data
  distribution still contain any ambiguities? I find this hard to
  harmonise with the statement in Section 3.4 about the existence of
  a classifier that separates the manifolds. Please clarify the meaning
  behind the term 'ambiguities' here.
- The '(R0)' requirement definition and the discussion in Section 3.2
  strike me as needlessly complex. Would it be possible to shorten this
  or move some content to the appendix? I think it would be sufficient
  to have Eq. 1 and mention how it could be solved.
- Section 3.3 could be shortened as well, if I am not mistaken; while it
  is good to know how such models look, the 'change of variable formula'
  is not used directly any more in the paper; I think it might be easier
  to write down a generic form of the density for each model.
- The salient points of Section 3.4 seem to be the projection point;
  maybe this could also be shortened somewhat in the interest of having
  more space for experiments. The relevant information of this section
  was to learn how projections work for different models, but it would
  be sufficient to keep Eq. 3 and discuss Eq. 4 in the appendix
- The 'topological differences' mentioned in Section 4 could be
  clarified: the paper talks about differences in connected components.
- The $\lambda$-density set appears to be a superlevel set, if I am not
  mistaken: the level set would be defined for a single threshold only,
  while this paper introduces the pre-image of an interval.
- I would expect the pre-image to be defined as $p^{-1}$, not $p_{-1}$;
  the latter strikes me as somewhat non-standard usage
- The statements preceding Definition 1 require some more intuition;
  what is the purpose of these assumptions?
- The notation for the Euclidean balls should be briefly introduced
  before Definition 1. I recognise this as a standard notation, but
  since this is the first appearance of the symbol, it should be
  mentioned at least briefly.
- Definition 1 could also be phrased more intuitively; additional
  sentence behind each definition would be useful, such as:
  $\delta_\lambda$ is the largest $\delta$ such that the full
  superlevel set is contained in a ball of radius $\delta_\lambda$.
  Figure 1 is already helpful in that regard; it should ideally precede
  the definition and/or be made larger to be more illustrative
- Maybe the results of Theorem 2 could already be stated earlier; it
  could probably be explained reasonably well when describing the number
  of connected components as a sort of 'baseline' topological complexity
  that a model has to satisfy.
  On a more abstract level, could it also be summarised as 'the
  inclusion of prior knowledge is a necessary condition for robustness'?
- I would not state that the main goals of the experiments are to
  'check the correctness of Theorem 2'---the proofs should be
  responsible for this! I would rather say that the main goals are to
  provide empirical evidence for the *relevance* or *applicability* of
  the Theorem.
- The paragraph on 'Latent vector distributions' contains the most
  relevant information, viz. the knowledge of what the paper considers
  to be a 'topology-aware' and 'topology-ignorant' model; this should be
  highlighted more; maybe the figures could be extended to contain
  information about $n_X$?
Overall, I like the idea of having one overarching question in a paper
that is subsequently answered or discussed under different aspects. I
very much commend the authors of the paper for choosing this sort of
writing style!
IV. Experimental setup
As mentioned above, the experiments require more depth. I would propose
adding more repetitions of the training process for different data sets
and analysing the impact of the 'parameters' used in the theorems.
In particular, the 'INC' experiments show great promise for multiple
repetitions. Why not choose more data sets and more staring positions
and visualise the trajectories of _multiple_ draws, as shown for
a single draw in Figure 4 a,b,c?
Also, please consider moving the additional experiments from the
appendix to the main paper.
More compelling examples would also be helpful. Why not generate data
sets consisting of more than two manifolds? At present, the largest
issue I see in this section is that the conceptual 'leap of faith'
between the theory and the applications is simply too large. Would it
not be possible to perform the same experiments on a simple digits data
set, say MNIST?
V. Minor issues
The paper is well-written overall. There are only a few typos and minor
style issues that I would recommend fixing:
- please check the usage of quotes; it should be ``pulled back'', not ''pulled back'' in LaTeX
- please check the usage of citations; if '\citet' is used, citations
  can be used as nouns directly (for example: 'in Pennec (1999)' instead
  of 'in (Pennec, 1999)'.
- etc.. --> etc.
- it approximates posterior distribution --> it approximates a posterior distribution
- for compound distribution --> for a compound distribution
- a relatively simpler distribution --> a simpler distribution
- near manifold --> near a/the manifold
- we simply the minimum --> we denote (?) the minimum
- satisfies the followings --> satisfies the following properties
- number of connected component --> number of connected components
- Experimental result --> Experimental results
VI. Update after the rebuttal
The authors managed to address the majority of my comments. Overall, I still would like to see a more detailed/in-depth experimental setup, but I realise that this not directly possible within the timeframe allotted during the rebuttal period. I am thus raising my score.