In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

The authors introduce a new regularization technique for the specific task of finetuning models. It's inspired by dropout and stochastically mixes source and target weights in order to avoid moving the parameters towards 0.
The authors provide a theoretical justification as to why mixout would do useful things in the convex case and then demonstrate empirically that using it achieves good accuracies on some downstream, finetuned, non-convex-loss-utilizing tasks. Their experiments incorporate both small models with good analysis (ie, sec 4) as well as larger, real-world models (sect 5). The paper is in general well-written.
I have a few concerns that I would like to see addressed:
1a. For starters, all the theoretical motivation describes a particular way in which mixout is supposed to aid in downstream tasks for the case of convex functions, but there are no experiments that match their assumptions and which demonstrate this is the actual behavior we see. It would be nice to see results which demonstrate the theory.
1b. In few of the prsented empirical experiments is it the case that the use of mixout by itself is useful.  Why does mixout have to be coupled with other regularization techniques? There is little analysis given here, either empirical or theoretical.
2. Why are there only 4 GLUE tasks reported? Devlin 2018 reports on all but WNLI.
3. The choice of hyperparameters for GLUE in sect 5 is a bit misleading. Devlin 2018 chose those parameters to get the maximum scores on downstream tasks; the metric they use is max score. However, the authors want to instead discuss the average score against a set of random restarts, perhaps because the max scores using their method aren't terribly different from the baselines.
Therefore, a more extensive hyperparameter sweep should have been run for the baselines: they should have been re-tuned for the average score if that is the metric the authors wish to use. Instead, the authors only used one task, RTE, to find baseline hyperparameters.


Almost all current adversarial attacks of CNN classifiers rely on information derived from the output layer of the network. This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions. We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models. Further, we place a priority on explainability and interpretability of the attacking process. Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement. We also conceptualize a transition from task/data-specific to model-specific features within a CNN architecture that directly impacts the transferability of adversarial examples.

The paper proposes a new adversarial attack for the targeted blackbox model Unlike previous approaches which use the output layer possibly with some additional terms and regularization, the proposed approaches only rely on intermediate features. In fact, the adversarial example is based on a single intermediate layer. The adversarial examples are built by training, for each target class, a binary classifier for the class based only on the features of that layer.
- There are multiple similar prior works that use intermediate layers in some way, and the paper does a good job to explain the differences between the proposed approach and these.
The paper proposes three variants. The simplest one appears rather weak in numerical experiments. The two other methods seem to achieve a different tradeoff, between focusing on the general error rate or on the targeted success rate.
In particular, FDA+fd is similar in some ways to Zhou et al. (2018) which uses an additional loss term to maximize the distance of the features at various layers. A comparison with this method (which uses the output layer) would be interesting, and may constitute a relevant additional baseline, at least for non-targeted metrics like error (which is where FDA+ms shines).
Similarly, a comparison to AA is provided in Figure 2 in the case of 10 classes, but not in Table 1 for the 1000 classes experiment. This casts some doubt on the significance of the result in the 1000 classes setting. A major issue with the proposed method is that training binary classifiers in large multi-class settings is quite costly, which is bound to hinder one's ability to identify 'high performing attack settings' In particular, the paper mentions that the 10 classes are used for that, but no evidence is provided that these settings are actually good for all classes.
The conclusions are rather consistent accross different pairs of source/target network.
The paper does not mention adversarial training. As this has become a common defence practice in the literature, it would be interesting to understand to what extent the proposed methods are thwarted by adversarial training. This could increase the significance of the paper if working with intermediate feature representation makes the attack more robust to this type of defense.
In summary, I find the experiments satisfying, although they can definitely be made more convincing by adding additional strong baselines and demonstrating how much performance can really be attained in the 1000 classes scenario.
- Generally, not using the output layer at all is an interesting approach that deserves in my opinion to be discussed and investigated further.
- The paper is clearly written, and notation is rigorous.


Graph neural networks (GNNs) are a powerful tool to learn representations on graphs by iteratively aggregating features from node neighbourhoods. Many variant models have been proposed, but there is limited understanding on both how to compare different architectures and how to construct GNNs systematically. Here, we propose a hierarchy of GNNs based on their aggregation regions. We derive theoretical results about the discriminative power and feature representation capabilities of each class. Then, we show how this framework can be utilized to systematically construct arbitrarily powerful GNNs. As an example, we construct a simple architecture that exceeds the expressiveness of the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theory on both synthetic and real-world benchmarks, and demonstrate our example's theoretical power translates to strong results on node classification, graph classification, and graph regression tasks.

The paper proposes minor modifications to Graph Convolutional Networks (GCNs) that as proven by the authors enable learning of local features in networks, namely the aggregation over powers of the adjacency matrix (effectively counting random walks within the neighborhood) and aggregating over connections within nodes in the neighborhood.
GCNs are of interest to
The paper is well written and clear.
The mathemeatical derivations are clearly structured.
The authors provide a large set of experiments on simulated and real data from different domains and relevant supervised tasks (node classifiation, graph classification and graph regression).
Source code is provided.
experimental results:
- For the simulation in Table 2, a 1-layer network seems like it would hield an advantage for the unstructured Erdös Renyi graphs. As I understand, the proposed GCN should by construction count the triangles (target variable) for each node embedding and then predict a linear function of those aggregated. My intuition says that this may even be an advantage over adding more layers. The text states the opposite. Here it would be interesting to see the effect of adding layers, or aggregating over longer random walks.
Here, it would be important to get the prediction of an untrained baseline in addition to the GCN baseline (e.g. the expected number of triangles/4cycles based on number of nodes in the Erdös Renyi model) to understand the scale the Mean Squared Error lives on and what the relative improvement means.


Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation. VIBI adopts an information theoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.

[Due to the rebuttal, my score was raised from a weak reject to a weak accept]
Summary
The paper addresses the problem of interpreting predictions/decisions of a black-box classifier/regressor by masking the parts of the input that were most relevant. The proposed approach consists of, first, manually designing “cognitive chunks” of input data, e.g. individual words for sentiment classification or fixed-size image-patches for image-classification. Then, a variational IB framework is used to infer which of these chunks are relevant for the classifier’s decision. Additionally, there is a (hard) constraint, making sure that only a fixed (small) number of chunks is used. The bottleneck variable, in this case, is a sparse-chunk representation of the data. The latter is obviously a more compressed representation of the data, but importantly it is a more compressed representation that contains the largest possible amount of relevant information about the decision (because of properties of the information-bottleneck objective). Both factors together, according to the paper, constitute a “good” (i.e. brief but comprehensive) explanation which allows for interpretability and attribution of the black-box system’s decision. The method is evaluated on three tasks (sentiment prediction, image classification, TCR to epitope binding prediction) and performance is reported to be on-par or better than state-of-the-art methods.
Contributions
-) Application of the IB-method for generating summaries of decision-relevant input-data, which are good candidates for interpretability. The theoretical properties of the IB objective are appealing for producing interpretable data-summaries.
-) Adaptation of the variational IB framework, using bits and pieces reported in the literature such that the bottleneck variable is a sparse, binary vector over “cognitive chunks”.
-) Experimental evaluation, where human judges rate the “interpretability“ of various state-of-the-art attribution methods.
Quality, Clarity, Novelty, Impact
The paper addresses a timely and important problem, particularly the IB framework could add some solid theoretical footing (the “theory of relevant information”) to the field of interpetability methods. The paper is well written (though it needs another pass for typos, etc.), related methods and literature are discussed and compared against, and the specific variational IB objective is introduced nicely. Large parts of the method (deep variational IB, VI with categorical variables) have been published before, but these parts are combined in a novel and original way. My main issues with the current paper are (I) interpretability and comprehensiveness are not necessarily the same as maximum compression of maximally relevant information, (II) the method (in theory) depends strongly on the quality of the approximator, this is currently not mentioned and not explored, (III) the experimental section is currently not very strong, in particular the MNIST experiment. See more details for the main issues below. Overall, I personally think that the main idea of the paper is interesting, mature and fleshed out enough for a publication, however, the experimental section is somewhat lacking and (II) is missing from the current manuscript. While the method has theoretical advantages, empirically it seems to perform more or less equal to L2X (but the chunks produced seem qualitatively different which is interesting). I am therefore slightly leaning towards suggesting a major revision of the paper, but I am happy to be convinced otherwise by the other reviewers and the authors during discussion/rebuttal. (I would rate the paper as "borderline", but it seems that this year's review system only allows for "weak reject" or "weak accept", so I'll go for "weak reject" for now).
Improvements / major issues
(I) Good compression of highly relevant information is not (always) the same as good interpretability/comprehensiveness. In the limit, the bottleneck variable captures a minimal sufficient statistic, i.e. a maximally compressed version of all relevant information - for finite beta, the bottleneck approximates such a minimal sufficient statistic. From a theoretical point of view this is very appealing, since it is guaranteed to cover a maximal amount of information (given a certain level of compression). But the way this information is represented matters a lot for interpretability - any reversible mapping of the bottleneck variable does not change its information content but can have substantial effects on interpretability, e.g. consider encrypting or randomly perturbing elements of the explanation (i.e. the selected cognitive chunks). This is a major open problem, and some theoretical grounding in the IB framework helps by talking about this problem in very concrete terms. While I would not expect the paper to solve the problem in full generality, some discussion, and perhaps adding a “shortcomings” section would be nice.
(II) The relevance of information is measured via I(t;y), which ultimately boils down to the approximation q(y|t). The quality of this approximation is crucial, which can of course be seen by how it influences the tightness of the bound. While I appreciate that the paper investigates the quality of the approximation to some degree (by inspecting the approximator fidelity), I would highly appreciate a thorough discussion of this issue (because ultimately the method will produce cognitive chunks that are relevant for q(y|t), not p(y|x) - the interpretations can be trusted only if q matches fairly well). It would be very interesting to see how quickly interpretability degrades with lower-quality q(y|t) - the latter would of course require more experiments with human “interpreters” which I would not expect to be easily feasible within the rebuttal period.
Another interesting experiment to test the match between q(y|t) and p(y|x) would be to “minimally intervene” on the input-chunks suggested by the method and see whether that actually affects the predictions of the black-box models. E.g. do small random perturbations to the selected cognitive chunks in the MNIST digits change the prediction of the black-box classifier? Compare this against small random perturbations in arbitrary chunks of the input.
(III) Experimental section: I’m fairly happy with the IMDB experiment, and the TCR to Epitope binding is a nice non-standard application but I find the quality and significance of the results a bit hard to judge. My main concern though is the MNIST experiment: what I would have expected was the following: cognitive chunks are shown to participants and they need to guess the correct number (just like in the IMDB experiment). In the experiment reported in the paper, I’m afraid that there’s a certain bias for judges favoring explanations that lie on the digits rather than off digits. It remains unclear whether they simply prefer the chunks selected by VIBI over other methods, or whether they have actually gained more understanding of how the black-box makes decisions.
(IV) Table 2 can easily be misleading because entries with highest mean-accuracy are marked in bold, regardless of whether confidence intervals overlap with other entries or not. Please fix this by either only marking entries in bold where the error bars don’t overlap with an entry in the same row, or marking all entries in bold that lie within the error bars of the best-performing entry. Particularly for “Approximate Fidelity” VIBI often does not perform significantly better than L2X but performs roughly equally well. Of course it would also help to run more repetitions to potentially shrink confidence intervals.
(V) Please state the (parametric form) of the prior r(z*) used for the experiments. Also state the analytical expression for the KL-term in the final objective that this prior leads to.
Minor comments
a) Please add some discussion on how the method depends on hand-crafting cognitive chunks, and how hard/easy this might be for different domains.
b) Rather than fixing r(z), other papers have proposed to optimize the prior as well (typically in the context of VAEs / VIB) which is well justified from an IB perspective. It might be interesting to explore these possibilities for VIBI as well in the future.
[1] Fixing a broken ELBO. Alemi et al. 2017
[2] The beta-VAE’s Implicit Prior. Hoffman et al. 2017
c) Instead of fixing the number of cognitive chunks in advance, it could also be interesting to infer that number as well (as a future extension of the method). This could either be achieved via a sparsity-inducing prior r(z), or perhaps by borrowing some ideas from the Deterministic IB [3], and its variational version.
[3] The deterministic information bottleneck. Strouse and Schwab. 2016


Models that output a vector of responses given some inputs, in the form of a conditional mean vector, are at the core of machine learning. This includes neural networks such as the multilayer perceptron (MLP). However, models that output a symmetric positive definite (SPD) matrix of responses given inputs, in the form of a conditional covariance function, are far less studied, especially within the context of neural networks. Here, we introduce a new variant of the MLP, referred to as the matrix MLP, that is specialized at learning SPD matrices. Our construction not only respects the SPD constraint, but also makes explicit use of it. This translates into a model which effectively performs the task of SPD matrix learning even in scenarios where data are scarce. We present an application of the model in heteroscedastic multivariate regression, including convincing performance on six real-world datasets.

This paper explores the problem of deep heteroskedastic multivariate regression where the goal is to regress over symmetric positive definite matrices; that is, the deep learning model should take as input data points, and produce a conditional covariance matrix as the output. The key challenge in this setting is how to ensure the predicted matrix is positive definite (and thus follows the non-linear geometry of these matrices), how the neural network can be trained for this task, and what loss function can be used for effective training. The paper proposes a neural network with bilinear layers in this regard, and uses the von Neumann divergence as the loss function to regress the predicted covariance against a ground truth SPD matrix. The gradients of the von Neumann divergence are provided for learning via backpropagation. Experiments on several synthetic datasets and small scale datasets are provided, showcasing some benefits.
Pros:
1. The use of von Neumann divergence as a loss for this task is perhaps novel.
2. The use of \alpha-derivatives, while computationally demanding, is perhaps novel in this context as well.
Cons:
1. I do not think the problem setting or the proposed framework is entirely new or is the best choice of its ingredients. Specifically, the idea of using second-order neural networks have been attempted in several prior papers, including the ones the paper cite (such as Ionescu et al. ,2015). Several other papers in this regard are listed below.
[a] Second-order Temporal pooling for action recognition, Cherian and Gould, IJCV, 2018
[b] Deep manifold-to-manifold transforming network, Zhang et al, ICIP, 2018
[c] Statistically motivated second-order pooling, Yu and Salzmann, ECCV, 2018
In comparison to these methods, it is not clear how the proposed setup is novel, or in what way is method better. There are no comparisons to these methods, and thus it is difficult to judge the benefits even empirically.
2. There are also models that predict the mean and covariance matrices directly from the model, such as the works below. The paper should also include and perhaps compare to their datasets.
[d] Deep Inference for Covariance Estimation: Learning Gaussian Noise Models for State Estimation, Liu et al, ICRA, 2018
3. I do not think the use of von Neumann divergence as a loss is the best choice one could have, esp. for a deep neural network learning setting. This divergence includes the matrix logarithm, which is perhaps computationally expensive. This is a problem when using other popular loss/similarity functions on SPD matrices (such as the log-Euclidean metric). Perhaps a better option is to use the Jensen-Bregman log-det divergence, as suggested in [a] above; this divergence is symmetric and also has computationally efficient gradients. It is unclear why the paper decided to use von Neumann.
4. The experiments are not compelling, there are no comparisons to alternative models and the datasets used are small scale. Thus, it is unclear if the design choices in the paper have any strong bearing in the empirical performances.
Overall, the paper makes an attempt at designing neural networks for learning SPD matrices. While, there are some components in the model that are perhaps new, the paper lacks any justifications for their choices, and as such these choices seem inferior to alternatives that have been proposed earlier. Also, the experimental results are not convincing against prior works.


Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running to improve both. In each cost volume, we encode not only photo-metric consistency across multiple input images, but also geometric consistency to ensure that depths from multiple views agree with each other. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.

The paper tackles Structure from Motion, one of the canonical problems in computer vision, and proposes an approach that brings together geometry and physics on one hand and deep networks on the other hand. Camera unprojection and warping (of depth maps and features) are used to build a cost volume onto hypothetical planes perpendicular to the camera axis. Similarly, various camera poses are sampled around an initial guess. A deep network regresses form the cost volume to a camera pose and a depth map. The method can be applied iteratively, using the outputs of the current stage as the initial guess of the next one. Training is supervised, and the the results are evaluated on multiple datasets.
I am inclined to recommend accepting the paper for publication, because it addresses a canonical problem, outperforms the state of the art on multiple datasets and brings together geometry / physics and deep learning, which is IMO very a promising and underexplored direction.
I found the method section a bit difficult to read though, and even after several readings I cannot get my head around it. Specifically, here are some issues that I hope the Authors could clarify.
1. In Sec. 3 the Authors write "We then sample the solution space for depth and pose respectively around their initialization". However in Sec 3.2 they write "we uniformly sample a set of L virtual planes {dl} Ll=1 in the inverse-depth space". In what way are the planes "around their initialization"? If the initial depth map spans over multiple orders of magnitude, will the planes be uniformly sampled between the minimum and maximum disparity of the initial map? If yes, it seems that the initial depth map is not really needed, just its minimum and maximum value is needed, but then how come the method can be applied iteratively with respect to depth?
2. The Authors mention that depth maps are warped onto the virtual planes using differentiable bilinear interpolation. Is there a mechanism to protect from interpolating across discontinuities? If no, were bleeding edge artifacts observed?
3. In the introduction, the Authors point that prior methods have trouble dealing with textureless, reflective or transparent approaches, but it's not clear form the paper where it addresses these cases, and if yes, what is the mechanism for that.
Lastly, if the authors are not planning to release the code, the implementation details section is a bit too high-level and does not contain enough details to reimplement the Author's technique. For example, "our network learns a cost volume of size L × W × H using several 3D convolutional layers with kernel size 3 × 3 × 3"  - more details about this network are needed, as well as the others in the paper.


More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU-or memoryconstrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric-the TSC Score-to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.

This paper presents an algorithm to generate images for training a student network through distillation. The paper claims the use of the same training data is not necessarily beneficial when it comes to improving the accuracy of the student being trained.
The paper sits on the empirically driven side with sufficient experiments in the context of random forest and CNN.
As a second contribution, the paper proposes a scoring process to evaluate the quality of datasets generated by GAN methods. There are little experiments on this side.
Pros:
- I like the paper and the idea behind being able to improve or even train a student network when the original data is not present.
- Metrics tailored to the problem are relevant.
Negs:
- While there are plenty of experiments, there is a lack of detailed descriptions.
- The scoring for GANS seems to be barely tested. In the scoring GAN, The fact that TSCS drops is enough to be a valid metric (or better than existing?). In the TSC vs inception, it is hard for me to see the unrealistic artifacts and, according to the text, that is not what TSCS is measuring, right? What is the influence of using different student-teacher configuration? (on the time to produce the scores the paper claims NiN and LeNet, is there any difference if using other architectures (ResNet family for instance)? At least, in that case, the time changes. It would be nice to see the stability of this metric to demonstrate that the need for training an inexpensive model is correct.
For the TSC vs Inception, the GANs are subjectively assessed, isn't it (as to select well-trained to Inferior). Would it be possible to see exactly the same images between the two of them? Seems like the difference in quality for IS is significantly larger than for the proposed metric (even in the last gan there is a slight increase in the metric).
It seems to me that IS is just a quality metric based on how a single image looks like. Would it be possible to disentangle the training and the scoring in the proposed metric? What if my hyperparameters for doing the single epoch are totally wrong?
In the general idea of the GAN, while I like it, there is little about how the GAN is actually trained. I guess this GAN is trained using some sort of real data and therefore, the comparison is not totally fair. How many images were used to train this GAN? What would happen if those images are used directly in the distillation framework?
How many images are generated in section 4? Is the influence of pfake related to the dataset? If I have to train using another dataset, how do i set that parameter?


The understanding of sensor data has been greatly improved by advanced deep learning methods with big data. However, available sensor data in the real world are still limited, which is called the opportunistic sensor problem. This paper proposes a new variant of neural machine translation seq2seq to deal with continuous signal waves by introducing the window-based (inverse-) representation to adaptively represent partial shapes of waves and the iterative back-translation model for high-dimensional data. Experimental results are shown for two real-life data: earthquake and activity translation. The performance improvements of onedimensional data was about 46 % in test loss and that of high-dimensional data was about 1625 % in perplexity with regard to the original seq2seq.

This work explains how to use a seq2seq encoder-decoder neural network on the case of multivariate time series. The authors name this particular application of seq2seq the wave2wave network. Given a multivariate time series covering a time interval, it is split into subintervals of equal length, such that each block is a matrix. This matrix becomes an input into a recurrent encoder. On the decoder side, the similar matrix is produced at the output. The proposed neural network is tested on two data sets: an earthquake and activity translation.
Strenghts:
+ application on activity translation and evaluation by human raters is interesting
+ the proposed work is a nice application of an encoder-decoder architecture in case of multivariate time series
Weaknesses:
- there is no methodological novelty. The proposed network uses a standard architecture and the authors use straightforward partitioning of the signal in equal-sized windows. Due to this, inventing a new name wave2wave is not fully justified.
- the paper has many grammar mistakes and typos (for example, first sentence in the introduction seems to miss a few words). The writing is vague and imprecise and occasionally interferes with understanding. It would be important for the authors to carefully proofread the paper for language and clarity of presentation.
- the experimental results are not explained in a sufficient detail. I did not understand what is the objective of the earthquake application and how does the data set looks like (e.g., what do you mean by “365 x 3 training data”). The details of experimental design, architecture and hyperparameter choice are not provided in the earthquake application. It is not clear what the authors mean by “simple encoder-decoder” and “simple seq2seq”, I guess those are some baselines. For the activity application, it would be useful to see some illustration of the neural network output when video is predicted from motion capture. It would help to clarify the accuracy measures – perplexity and human understability. How many humans were involved in rating and what were they asked to do? How did the humans rate the non-video outputs?
Overall, given the quality of writing and lack of methodological novelty this paper is not ready for publication.


Encoder-decoder convolutional neural networks (CNN) have been extensively used for various inverse problems. However, their prediction error for unseen test data is difficult to estimate a priori, since the neural networks are trained using only selected data and their architectures are largely considered blackboxes. This poses a fundamental challenge in improving the performance of neural networks. Recently, it was shown that Stein's unbiased risk estimator (SURE) can be used as an unbiased estimator of the prediction error for denoising problems. However, the computation of the divergence term in SURE is difficult to implement in a neural network framework, and the condition to avoid trivial identity mapping is not well defined. In this paper, inspired by the finding that an encoderdecoder CNN can be expressed as a piecewise linear representation, we provide a close form expression of the unbiased estimator for the prediction error. The close form representation leads to a novel bootstrap and aggregation scheme to prevent a neural network from converging to an identity mapping so that it can enhance the performance. Experimental results show that the proposed algorithm provides consistent improvement in various inverse problems.

1. Summary
The authors address the problem of efficiently employing the SURE estimator as a network training regularizer. They show that for CNN autoencoders this can be efficiently computed. Their other contribution is a bagging/boosting technique which is proved to avoid trivial solutions. The proposed architecture, motivated by the theoretical statements, is shown to outperform classic and 2019 state of the art image reconstruction algorithms in MRI and EDX.
2. Decision and arguments
Unfortunately this paper is outside my expertise so I can’t evaluate the novelty of the theoretical accomplishments. However taking that as a given, they well-motive the proposed architecture and achieve impressive experimental results. The experiments are well described.
3. Questions
a) Why do Table 1 and Figure 3 provide different PSNR and SSIM values?
b) Is there any way to measure accuracy to ground-truth with the EDX data? Or are the results just qualitative?
c) With respect to Figure 2, and in general for autoencoders, the input and output have the same dimension. So how do you reconcile this with undersampled MRI and EDX data? I understand you train on fully sampled data—then how do you input undersampled data? Are the unknown samples set to zero?


Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world. In this work, we propose to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks. We develop learning-based approaches to inducing causal knowledge in the form of directed acyclic graphs, which can be used to contextualize a learned goal-conditional policy to perform tasks in novel environments with latent causal structures. We leverage attention mechanisms in our causal induction model and goal-conditional policy, enabling us to incrementally generate the causal graph from the agent's visual observations and to selectively use the induced graph for determining actions. Our experiments show that our method effectively generalizes towards completing new tasks in novel environments with previously unseen causal structures.

[Summary]
This paper proposes an interactive agent that tries to infer the underlying causal structure by interacting with the environment; the authors called it "causal induction."  The inferred graph will later help the agent complete goal-directed tasks referred to as a "causal inference" stage. Notably, the agent directly learns from visual inputs.
Both the induction and inference phases heavily rely upon the attention mechanism, which ensures that the agent only focuses on the relevant components of the causal graph. During the induction phase, the agent incrementally updates the predicted causal graph through each interaction using an attention-based edge decoder. During the inference phase, the attention bottleneck also showed to improve the agent's generalization ability.

They have shown that the proposed model outperforms several baselines in a synthetic environment that uses switches to control lights. They have also demonstrated the model's generalization ability by operating on unseen causal graphs and new task goals.
[Major Comments]
My primary concern about this work is the scope of its applicability.
For the causal induction phase, the proposed method makes a strong assumption that it can access the ground truth causal relationship during training. The authors can directly read this information from the synthetic environments used in this paper, yet, in more complex real-world situations, we might not know the underlying causal structure for supervised training the induction model.
For learning the goal-conditioned policies, the authors also assume that they have access to the ground truth causal graph. They use this information to generate the expert demonstrations, which, I presume, are deterministic and unimodal (correct me if I'm wrong). In the real world, a human may be able to infer the underlying causal structure from the observations by interacting with the environment and provide the demonstration data. However, the demonstration may be noisy or form a multi-modal distribution. While I agree that learning from demonstration is an effective way of guiding the learning of the policy, I'm not sure if the method can generalize to more realistic scenarios.
For inferring the causal graph, the authors also assume that they know the "cause" set and the "effect" set, which is already a DAG by construction. Instead of inferring the direction of the edge, they are solving an easier problem of deciding whether a directed edge between a "cause" node and an "effect" node exists or not. The assumption on the graph structure also limits the method's applicability, as, in the real world, the direction of the edge is not always known in advance. Smoking may cause lung cancer, but it is possible that lung cancer may make people smoke more.
I feel this paper makes strong assumptions on both the induction and inference stages, as well as the structure of the causal graph, which greatly limits the applicability of the approach.
[Detailed Comments]
I also have a few questions regarding the details of this paper.
In Section 3.1, the authors said that "N is the number of actions in the environments," which is a bit confusing. Before this point, the authors did not discuss the relationship between the size of the graph and the size of the action set. Only until Section 3.2 did I realize that N is the number of both "cause" set and "effect" set. It would be better to discuss the size of the graph and the action set at an earlier position.
How is the expert planner implemented? Is the expert's policy deterministic and unimodal? I feel this is an important detail to include.
It is related to the previous question. What will happen if we learn the goal-conditional policy from scratch? How much does imitation learning help with policy learning? Again, the assumption that we know the ground truth causal graph may not be feasible in the real world.
In Section 3.3, the authors said that "the expert's action is added to the memory of the policy." However, the authors also noted that "the policy has no memory" in Section 3.2, which seems to contradict each other. Does the "memory" mean replay buffer in Section 3.3?
Does the number of switches fixed across all environments and always the same as the number of lights? Also, are all the lights mounted in the same location? I'm wondering if the model is invariant to the order of the cause nodes and effects nodes in the graph, and can it generalize to larger environments, more lights, and different room configurations.
In Figure 4, TCIN seems to have the best performance in the "Masterswitch" environment when there are 500 seen causal structures. What might be the reason?


In this paper we prove new universal approximation theorems for deep learning on point clouds that do not assume fixed cardinality. We do this by first generalizing the classical universal approximation theorem to general compact Hausdorff spaces and then applying this to the permutation-invariant architectures presented in PointNet (Qi et al) and Deep Sets (Zaheer et al). Moreover, though both architectures operate on the same domain, we show that the constant functions are the only functions they can mutually uniformly approximate. In particular, DeepSets architectures cannot uniformly approximate the diameter function but can uniformly approximate the center-of-mass function but it is the other way around for PointNet. Additionally, even when the point clouds are limited to at most k points, PointNet cannot uniformly approximate center-of-mass and we obtain explicit error bounds and a method to produce geometrically derived adversarial examples.

This work examines the fundamental properties of two popular architectures -- PointNet and DeepSets -- for processing point clouds (and other unordered sets). The authors provide a new universal approximation theorem on real-valued functions that doesn't require the assumption of a fixed cardinality of the input set. They further provide examples of functions that can't be mutually approximated by PointNets and DeepSets.
- It is important in order to know the representational power and fundamental limitations of basic algorithmic building blocks. However, for the two base architectures that are examined in this work, UATs where already provided in the original manuscripts. The presentation in this paper does remove the assumption of a fixed cardinality, but since this seems to be a mild assumption, it is not clear what is gained by this (beyond mathematical elegance). The paper doesn't give any hints here.
- The paper shows two specific functions, where one can be approximated by PointNet but not by DeepSets and vice-versa. The authors again note that this might vanish when a fixed cardinality of the input point cloud is assumed. Again, it seems that this is a mild restriction and I'd like the authors to elaborate on the importance of removing this restriction.
- The paper is clearly targeted at a specialist audience and invokes dense and advanced mathematical concepts. It might find a better audience at a venue that is more specialized in this type of work (e.g. applied mathematics or more theory-focused machine learning venues).
Summary: UATs are important and interesting, however, they do exist for the architectures that are targeted in this paper. It is not clear what is gained by the main difference, i.e. removing the cardinality. I'd be grateful if the authors could comment on this.
Disclaimer: While I believe to have a reasonable mathematical background, I'm not an expert in this field and my assessment is primarily based on the bottom line of these proofs.
=== Post rebuttal update ===
I'd like to thank the authors for their efforts and additional insights. The additional illustration improves the accessibility of the paper. However, the rebuttal does not alleviate my concerns about additional impact beyond the UATs in the original paper. I thus maintain my recommendation of weak reject


Point clouds are a flexible and ubiquitous way to represent 3D objects with arbitrary resolution and precision. Previous work has shown that adapting encoder networks to match the semantics of their input point clouds can significantly improve their effectiveness over naive feedforward alternatives. However, the vast majority of work on point-cloud decoders are still based on fully-connected networks that map shape representations to a fixed number of output points. In this work, we investigate decoder architectures that more closely match the semantics of variable sized point clouds. Specifically, we study sample-based point-cloud decoders that map a shape representation to a point feature distribution, allowing an arbitrary number of sampled features to be transformed into individual output points. We develop three sample-based decoder architectures and compare their performance to each other and show their improved effectiveness over feedforward architectures. In addition, we investigate the learned distributions to gain insight into the output transformation. Our work is available as an extensible software platform to reproduce these results and serve as a baseline for future work.

Summary
This paper studies the problem of 3D point cloud autoencoding in a deep learning setup, and in particular, the choice of the architecture of a 3D point cloud decoder. The main claim of the paper is the adaptation of the PointNet architecture (a set of fully-connected layers applied per point) from encoding to decoding purposes. To achieve that, the authors propose three similar mechanisms to inject noise in the bottleneck shape representation, which are used to sample per point feature vectors suitable for a PointNet-like decoder. These include appending a low dimensional gaussian noise sample to the bottleneck shape representation, adding individual gaussian noise sample to each bottleneck shape representation dimension or using trainable variances per bottleneck shape representation dimension. Resulting networks are optimized by minimizing Chamfer distance between the ground truth and output point clouds. The main motivation of their choice of these architectures is their ability to output an arbitrary size point clouds, compared to naive multilayered fully-connected network applied to a bottleneck shape representation and outputting a "flattened" fixed-size point cloud.
To verify their claims, the authors considered autoencoding task for ModelNet40 dataset. They compared a naive multilayered fully-connected decoder to PointNet-like decoders with all three proposed noise-injection mechanisms. This comparison was also performed for 5 variations of the proposed models differing is network size. To compare quantitatively, the authors present all results graphically, showing the dependency of resulting average Chamfer distance from the network size. This graph shows that using trainable per dimension variance for noise injection works consistently better, than other options; also, it shows inconsistent results for the baseline multilayered fully-connected decoder. In qualitative comparisons, the authors compare output point clouds visually and study the influence of sampling from various noise dimensions on the reconstructed points.
Review
Although the authors of this paper touch a very interesting topic in generative modeling of point clouds, I do not believe this paper should be published in its current state.
Cons:
1. The main issue in this paper is the absence of comparisons to any external approaches. Even though comparison to their variation of the baseline model is provided, it is impossible to verify where their results stand compared to the related work. There is a whole stream of works in the same task using ShapeNet dataset, which the authors completely omit. Even if the authors do not want to perform experiments on ShapeNet, they, at least, could have provided comparison to AAE and PC-GAN results from PC-GAN paper [1] for generative modeling metrics on ModelNet40. As the authors stated, there are at least two works, PC-GAN and PointFlow, which allow sampling of the arbitrary size point clouds and comparison to these approaches is crucial (especially since the proposed approach is technically significantly simpler compared to hierarchical GANs from PC-GAN and continuous normalizing flows combined with VAEs from PointFlow papers).
2. The paper misses numerous citations to related work in generative 3D shape modeling:
- voxel grid based, (since the authors mention them): [2, 3, 4, 5, 6, 7]
- point cloud based: [8, 9, 10]
- graph based: [11, 12]
All these approaches propose various types of decoders for 3D shapes represented accordingly and some of them perform evaluations for pure generative, autoencoding task, which, again, can be used for comparison.
3. I believe, there is a factual mistake and arguable diminishing of the current state of the art in the related work section:
The authors state, that both PC-GAN and PointFlow models "inhabit GAN settings". However, PointFlow is, in fact, an approximate likelihood model and is considered as an instance of Variational Autoencoder with a shape prior and a point cloud decoder implemented as continuous normalizing flows (a special case of invertible neural networks). The authors also state, that it is "unclear... if they are capable of outperforming the baseline MLP approach" when they talk about these papers, while, in fact, both PC-GAN and PointFlow explicitly show that by comparing to [9], which use the multilayered fully-connected baseline decoder. It is also not unclear if these approaches need specific loss functions, contrary to what authors say. They were designed so they need their particular loss functions, which does not disqualify them from the comparison. These papers are the main competitors of this work and should be addressed properly instead of being diminished.
Pros:
1. Besides the cons, the paper is written clearly and is easy to follow.
2. Rigorousness with respect to comparison of the proposed options (Table 1 and the fact that every model was retrained 15 times to obtain average result).
3. Qualitative experiments with control over noise dimensions are novel and interesting.
Overall, this could have been a paper about a nice and simple alternative to PC-GAN and PointFlow for arbitrary size point cloud modeling, but, unfortunately, it missed the opportunity to properly address the current state-of-the-art in its current version.
Small remarks and questions for authors:
1. Citation of Valsesia at al. in the related work should be reorganized to be consistent with other citations. Omitting full names there and in case of PC-GAN will also be more consistent.
2. All figures in the paper should be updated for better visibility of both graphics and text. Figures 1-4, 7 - text is barely visible, the scheme blocks and lines are too thin and overall lack readability. Figures 5-6, 8 - point clouds are almost invisible even with zooming, remove coordinate grid in the background, zoom in on the objects, so they take as much space in the images as possible, use bigger point sizes for better visibility.
3. Maybe I do not follow something, why have you ignored applications of your point cloud decoders, and particularly generative modeling task and its evaluation protocols, presented for example in [9]?
[1] C.-L. Li, M. Zaheer, Y. Zhang, B. Póczos and R. Salakhutdinov. Point Cloud GAN. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2019
[2] C. Choy, D. Xu, J.-Y. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
[3] R. Girdhar, D. Fouhey, M. Rodriguez, and A. Gupta. Learning a predictable and gen- erative vector representation for objects. In ECCV, 2016.
[4] J. Wu, C. Zhang, T. Xue, W. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In NeurIPS, 2016.
[5] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient
convolutional architectures for high-resolution 3D outputs. In ICCV, 2017.
[6] S. R. Richter and S. Roth. Matryoshka Networks: Predicting 3d geometry via nested shape layers. In CVPR, 2018.
[7] R. Klokov, J. Verbeek, and E. Boyer. Probabilistic Reconstruction Networks for 3D Shape Inference from a Single Image. In BMVC, 2019
[8] T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. AtlasNet: A papier-mâché approach to learning 3D surface generation. In CVPR, 2018.
[9] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning Representations and Generative Models for 3D Point Clouds. In ICML, 2018.
[10] P. Mandikal, K. Navaneet, M. Agarwal, and R. Babu. 3D-LMNet: Latent embedding matching for accurate and diverse 3D point cloud reconstruction from a single image. In BMVC, 2018.
[11] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y. Jiang. Pixel2Mesh: Generating 3D
mesh models from single RGB images. In ECCV, 2018.
[12] C. Wen, Y. Zhang, Z. Li, and Y. Fu. Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation. In ICCV, 2019.


We target the problem of finding a local minimum in non-convex finite-sum minimization. Towards this goal, we first prove that the trust region method with inexact gradient and Hessian estimation can achieve a convergence rate of order O(1/k 2/3 ) as long as those differential estimations are sufficiently accurate. Combining such result with a novel Hessian estimator, we propose a sample-efficient stochastic trust region (STR) algorithm which finds an ( , √ )-approximate local minimum withinÕ( √ n/ 1.5 ) stochastic Hessian oracle queries. This improves the state-of-the-art result by a factor of O(n 1/6 ). Finally, we also develop Hessian-free STR algorithms which achieve the lowest runtime complexity. Experiments verify theoretical conclusions and the efficiency of the proposed algorithms.

This paper applies the spider algorithm (Fang et al., 2018) for reducing variance in first-order stochastic optimization to a second order optimization algorithm, i.e., trust region algorithm.
Due to the new gradient and Hessian estimators in Fang et al., 2018, the proposed stochastic trust region algorithms in this paper achieve $O(\min\{1/\epsilon^2,\sqrt{n}/\epsilon^{1.5}\})$ stochastic second-order oracle (SSO) complexity. This result improves the SSO of stochastic variance-reduced cubic (SVRC) by a factor of $n^{1/6}$. This paper has a moderate contribution because of the new algorithms and improved complexity results. However, the idea of variance reduction is not novel and the result is not surprising since the improvement comes purely from spider (Fang et al., 2018) and thus this work is somewhat incremental. The paper is in general fairly written without many typos or unclear statements. But some places are verbose and necessarily complicated. Moreover, the comparison between this paper and existing work is not clear and straightforward. Lastly, the presentation of the current paper is unnecessarily verbose and complicated.
The comparison of this paper with a similar paper by Zhou & Gu (2019) is not convincing. In the remark after Corollary 4.1 and in Section 6.1, the authors mentioned that the SRVRC algorithm proposed by Zhou & Gu (2019) achieves similar complexity as this paper. But the authors did not present the complexity of SRVRC in Table 1. This is not appropriate since it is very similar and related to this paper. The authors should compare with existing work in a more clear and fair way.

In Remark 3.1 and the discussion after that, the authors argued that the exact step size control is crucial to the sample efficiency of stochastic trust region algorithms. However, the cubic regularization based algorithm in Zhou & Gu (2019) can also achieve the same second order oracle complexity. Therefore, I don’t think the arguments in the two paragraphs after Remark 3.1 are the key reason for the improvement. Instead, the spider estimator that greatly reduces the variance is the key point leading to the improved sample efficiency.
I find the presentation of this paper is very verbose and unnecessarily long (10 pages, 8 algorithm boxes and 8 theorems/lemmas in the main paper). Many places can be simplified or combined in order to increase the readability of the main theorems. Some intermediate results may also be moved to the appendix if necessary.
Algorithm 2 and 3 are almost identical since the gradient and Hessian estimators ($g^k, H^k$) are represented in the same form. I don’t see the point of repeating the algorithms twice. In the Estimator 3, the two options can be simply combined by setting $s_2’=\min\{1/\epsilon, n\}$ since according to Lemma 4.1 $s_2’=1/\epsilon$ in option II.
The paper talks about $\epsilon$-SOSP, $\tilde{O}(\epsilon)$-SOSP, $12\epsilon$-SOSP and so on in many places. It would be better to be consistent and use the same notation.
In the text after Corollary 4.1, there are some typos in the complexities where a $\min$ operator is missing. Moreover, the complexity of Zhou & Gu (2019) presented here seems not correct. I quickly checked their paper and found in their table that the SFO complexity of Zhou & Gu (2019) is $\tilde{O}(\min\{n/\epsilon^{3/2},n^{1/2}/\epsilon^2,1/\epsilon^3\})$, which is in fact smaller than the complexity of STR1 in this paper. Therefore, I think the comparison in this paragraph is not correct.


Generative Adversarial Networks have made data generation possible in various use cases, but in case of complex, high-dimensional distributions it can be difficult to train them, because of convergence problems and the appearance of mode collapse. Sliced Wasserstein GANs and especially the application of the Max-Sliced Wasserstein distance made it possible to approximate Wasserstein distance during training in an efficient and stable way and helped ease convergence problems of these architectures. This method transforms sample assignment and distance calculation into sorting the one-dimensional projection of the samples, which results a sufficient approximation of the high-dimensional Wasserstein distance. In this paper we will demonstrate that the approximation of the Wasserstein distance by sorting the samples is not always the optimal approach and the greedy assignment of the real and fake samples can result faster convergence and better approximation of the original distribution.

This paper proposes two alternative approaches to max-sliced Wasserstein GANs. They are based on the authors’ claim that there is a “flaw” in the Wasserstein-1 distance between probability distributions on the one-dimensional space. Briefly, the authors’ argument says that the “flaw” is that the optimal transport may not be unique, some of which are better for network learning than others. One proposal, described in Section 2.2, is to find a plausible transport plan in a greedy manner. The other proposal, described in Section2.3, is a hybrid of the greedy approach in Section 2.2 and the original sliced Wasserstein distance.
The working hypothesis in this paper, that the above “flaw” is indeed problematic in learning in max-sliced Wasserstein GANs, has not been confirmed in any sense in this paper. Algorithm 1 is meant to explain one of the proposal, the greedy approach, but I found that several undefined symbols are used there, so that it seems hard to understand it. Numerical experiments in Section 3 are not well described. Because of these, I would not be able to recommend acceptance of this paper.
Algorithm 1 seems to heavily rely on Algorithm 1 in Deshpande et al., 2019. This paper does not provide any explanation about why one should sample n data for each i running from 1 to n (line 3), what the “surrogate loss” is (line 4), what ¥omega is (line 4), why one should care about the surrogate loss between ith data and ith generated sample (line 4), as well as what D^i_k means (line 11).
I do not understand how the Pearson correlation coefficient between the generator (or fake) distribution and the real distribution in Section 3. As far as my understanding, fake samples and real samples are sampled independently, so that correlation coefficient should ideally vanish in any case. Also, the KL divergence is not symmetric, so that whether KL(P_F,P_R) or KL(P_R,P_F) was evaluated has to be explicitly stated. Furthermore, recalling that the Wasserstein distance has originally been introduced to the GAN literature in order to alleviate the problems associated with the KL-based divergence (Jensen-Shannon), I do not understand either why the authors chose to use the KL divergence in their performance comparison.
The “flaw” argued in this paper does not apply to the Wasserstein distance in general, but specifically to the Wasserstein-1 distance between one-dimensional distributions. This fact should be stated clearly.
Page 2, equation (2): The subscript ¥mathbb{P} should read p.
Page 2, two lines below equation (3): w should be italicized.
Page 4, line 4: if the(y->ir) probability
Page 4, line 24: has no effect (t)on the inference complexity
Page 5, Algorithm 1, line 17: g of ¥theta g should be a subscript of ¥theta.
Page 6, line 3: which signals th(e->at) the
Page 6, line 12: was executed for (500.000->500,000) iterations.
Page 6, line 15: Figure number is missing.
Page 7, lines 9-10: (10.000->10,000) random projection(s) and used


The demand for abstractive dialog summary is growing in real-world applications. For example, customer service center or hospitals would like to summarize customer service interaction and doctor-patient interaction. However, few researchers explored abstractive summarization on dialogs due to the lack of suitable datasets. We propose an abstractive dialog summarization dataset based on MultiWOZ (Budzianowski et al., 2018). If we directly apply previous state-of-the-art document summarization methods on dialogs, there are two significant drawbacks: the informative entities such as restaurant names are difficult to preserve, and the contents from different dialog domains are sometimes mismatched. To address these two drawbacks, we propose Scaffold Pointer Network (SPNet) to utilize the existing annotation on speaker role, semantic slot and dialog domain. SPNet incorporates these semantic scaffolds for dialog summarization. Since ROUGE cannot capture the two drawbacks mentioned, we also propose a new evaluation metric that considers critical informative entities in the text. On MultiWOZ, our proposed SPNet outperforms state-of-the-art abstractive summarization methods on all the automatic and human evaluation metrics.

Authors proposed an enhanced Pointer-Generator model called SPNet. The key difference between SPNet and PG are the separate handling or using of speaker role, semantic slot and domain labels. Authors also proposed a new metrics called Critical Information Completeness (CIC) to address ROUGE's weakness in assessing if key information is missing in the output.
SPNet considers speak role by using separate encoders for each speaker in the dialog. The hidden state vectors of all speakers are concatenated for next layer.
Semantic slot is modeled by delexicalizing the input, i.e. replacing values (18:00) with their semantic category (time). The actual value is later recovered from input text by copying over the corresponding raw tokens according to the attention layer. The domain labels are incorporated by combining categorization task loss into the final training loss.
Authors used the MultiWoz dataset to evaluate the model and compared it with state-of-the-art Pointer-Generator and Transformer models. ROUGE and proposed CIC metrics all show clear improvements in SPNet. The best performance was observed when all three improvements over SPNet are leveraged. Authors also provided example generated summary and discussed the difference between SPNet PG and baseline. An additional human evaluation was conducted which confirmed the quality gain.
The main concern of Reviewer is the inconsistency in the paper.
1) Authors claimed to "propose an abstractive dialog summarization dataset based on MultiWOZ (Budzianowski et al., 2018)" in the abstract and introduction, which sounds like part of their contribution is creating a new dataset, but in experiment section there's no discussion about how the dataset was created or used at all. The same claim reappeared as the first sentence in the conclusion section.
2) Authors emphasized two drawbacks in the beginning of the paper, but didn't discuss or show any evidence of those drawbacks from data later.
The above inconsistency suggests the paper may not be quite ready for publication.
Other issues found by Reviewer:
1) In equation (7), value() seems to be the word while on the right hand side it's a numerical value (max a_i^t). Did Authors mean argmax?
2) In Table 1, dialog domain seems to provide very marginal improvement, does it justify the complexity added?
3) In Section 4.3, why do we need to train a customized embedding? The process and parameter for the embedding training was not described.
4) In Section 4.3 "batch size to eight" better be consistent as "batch size to 8" (minor issue).


Many machine learning systems today are trained on large amounts of humanannotated data. Annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases. To improve data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models. Such a multi-stage approach results in redundant annotations and may often produce imperfect "ground truth" labels that limit the potential of training supervised machine learning models. We propose a new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difficulties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies. Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classifier accuracy estimation from unlabeled data. We conduct an extensive experimental evaluation of our method on 5 crowdsourcing datasets of varied difficulty and show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations, as well as significant reductions in the required annotation redundancy. We further conduct an ablation study to evaluate the effect of both end-to-end learning and instance features and show that both contribute to the performance gains achieved by the proposed method.

Update after author response:
I would like to thank the authors for their thoughtful response, and for addressing some of the concerns raised by the reviewers. One of my main complaints arose from a misunderstanding that none of the baselines model worker competencies and task difficulty. The authors clarified that MMCE does model those, making it a competitive baseline. Given that and the other additions to the draft, I am changing my assessment from 3 to 6.
---------------------------
In this paper, the authors extend the classical probabilistic model of Dawid-Skene (DS) for predicting the final label of crowdsourced tasks. The extensions include explicit modeling of image difficulty and worker competence as a function of image and worker features respectively, as well as a more expressive formulation of learned functions in terms of neural networks (NNs). The authors show that the proposed approach outperforms several baselines on 5 different datasets.
The problem studied in the paper is relevant since more and more data is needed to train ever more complex models, and often crowdsourcing is the way to generate such data. The paper is clearly written and conveys its central idea concisely. I liked the paper but I believe the contribution to be incremental in its current state. Here are the main issues in my opinion:
1. The presentation suggests that somehow the proposed approach is novel in its end-to-end framework. However, the central idea is very similar to Dawid-Skene (1979), and its subsequent augmentations like the ones in Carpenter (2008) that model both image difficulty and worker competence. These previously proposed models are also end-to-end approaches so that they can infer worker competence and image difficulties while also outputting a final label. Therefore, I believe the novel contributions are then slightly different formulation of these variables, and using NNs to learn the function parameters. I see that the proposed model can be somewhat more general since it uses semantic image features (as opposed to indicator features) but then it learns worker embeddings starting from random initializations which will not generalize to new workers, necessitating a re-run of the whole EM loop.
2. If Q is a stochastic matrix, its rows should sum to 1. I don’t understand the summation on Q in eq. 4. I am also a little confused by eq. 5 since the joint likelihood should just be the prior times the conditional likelihood of the observed annotations. At least trying to do it in my head doesn’t lead to what’s presented in eq. 5. It may just be an issue of notations which can be simplified by using conditional distributions instead of expectations of indicator random variables (same quantity).
3. As far as I understand, none of the baselines considered explicitly model the task difficulty and worker competence. Therefore, the proposed model enjoys this extra level of expressiveness making its superior performance relatively unsurprising. I skimmed Zhou et al. (2015) and simple DS was already competitive on some of the datasets so I would think inclusion of its successors that model image difficulty and worker competence could perform quite well.
3. Minor issue but eq. 6 is not majority vote as stated just above. It’s the maximum likelihood estimate P(y_i = k | D) assuming each worker is an unbiased estimator of the true label.
4. Minor formatting issues: “learn a models” (sec 3.2), “lantent variables” (sec 4) “and by merging” (sec 5).
In summary, even though I like that the paper is clearly written and tackles an important problem, I am not convinced that the contribution is substantial or the experiments very insightful. I would recommend addressing these issues and resubmitting the paper.
Carpenter, B. (2008). Multilevel bayesian models of categorical data annotation. Unpublished manuscript, 17(122), 45-50.


In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent selfsupervised learning approaches are ill-suited to the remote sensing domain. In addition, most remote sensing applications currently use only a small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose a new self-supervised training objective, Contrastive Sensor Fusion, which exploits coterminous data from multiple sources to learn useful representations of every possible combination of those sources. This method uses information common across multiple sensors and bands by training a single model to produce a representation that remains similar when any subset of its input channels is used. Using a dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors. These representations outperform fully supervised ImageNet weights on a remote sensing classification task and improve as more sensors are fused. Our code is available at https://storage.cloud.google.com/public-published-datasets/csf_code.zip.

The paper proposes an unsupervised method to embed (remotely sensed) image patches such that they are clustered by high-level "image content" rather than by low-level pixel statistics. The argument is that in remote sensing one often has access only to a subset of all possible images/channels, so it would be useful to get the same embedding for each subset depicting the same scene. The basic idea is to generate, during training, two versions of the same patch by randomly dropping out channels, and to penalise the difference between the two corresponding embeddings, using the InfoNCE loss. The latter ensures that the patch remains "recognisable" among a set of random non-matching patches, so as to avoid learning a trivial embedding that is the same for any input.
The high-level goal, to train an encoder that always gives a similar encoding (and consequently similar downstream results) for a given patch, independent of the sensor channels used, is certainly appealing. It is also true that one would like to do this with unsupervised learning or transfer learning, since labelled data is scarce in remote sensing. Unfortunately, in my understanding the paper does not make much progress towards that goal.
Novelty is limited, the work is a variation of the CMC method. Yes, it splits channels randomly, not by views; and uses a Siamese architecture, not separate encoders; which it can afford to do since there are no viewpoint differences. And it computes the loss at more than one layer - which is a fairly trivial and popular thing to do. Moreover it is not well justified in this case, since all that matters in a feed-forward architecture is that the final embeddings are similar. Constraining lower layers might empirically help - in particular speed up learning - but conceptually there is no reason to reduce the lower layers' capacity to somehow turn different inputs into the same output. If they are really very different (think RADAR images) this could in principle even hurt.
Both in the discussion and in the experiments, I miss crucial and obvious baselines. First, it does in my view not make sense to use pre-trained ImageNet features as the only baseline. Yes, they are known to work surprisingly well also for remote sensing data - but, unsurprisingly, not nearly as well as pre-training on similar remote sensing images. A more natural baseline would be to pre-train on the same type of imagery, with a proxy task for which it is easy to get labels automatically  - actually the paper implicitly suggests that labels from OpenStreetMap might be a good candidate. Beyond reducing the domain gap, this also has the advantage that one is not limited to 3 channels, and that one can use an architecture suitable for the task (e.g., standard architecture with a lot of pooling are known to not perform well for semantic segmentation of remote sensing data). Labels for supervised learning might in many remote sensing tasks be scarce, but to get any old labels and pre-train a reasonable proxy task is still easy, as map data is abundant.
Perhaps even more important is another natural baseline. In spirit, the proposed method is closer to a generative embedding, since the supervision signal is not a discriminative task, but rather encourages embeddings to stay unique for each patch, and hence in principle fully decodable. Arguably, the most obvious way to achieve the same thing is a Siamese pair of auto-encoders for the two training patches, with an additional loss on the difference between the latent representations. Without that baseline it is impossible to tell whether the InfoNCE loss does a good job - in some sense it feels like making the task harder if one demands not only that a patch of water can be decoded from the embedding, but that it should be distinguishable from any other patch of water that resembles it even in raw pixel space.
The data used in the experiments is somewhat unconvincing, as it is really the opposite of diverse, multi-modal remote sensing channels. The same four R-G-B-NIR channels from sensors with not so different resolutions. The claim that the method homes in on high-level similarity and overcomes low-level radiometric differences would be much more credible if one uses short-wave infrared, thermal, perhaps even Radar images. Moreover, the gains are disappointingly small: even in the best case of 12 channels, the improvement over ImageNet pre-training is roughly from 4.5/10 correct neighbours to 5/10 correct neighbours. The even lower gains with fewer channels underline the need to have a pre-trained baseline without a massive domain gap. The classification accuracy of kNN does improve with the proposed embedding (at least for some combinations - for Pleiades RGB it is the same), but that experiment is unnatural. If one has the labels to do kNN, one might as well use them to train a proper classifier, i.e., add a couple of fully connected layers and fine-tune in a supervised fashion.
A number of technical details in the experiments remain unclear. Section 4.2 suggests the channels in Fig. 4 were added in a fixed order - why? There are many potentially viable combinations of, say, 4 channels, not just one. Furthermore, Fig. 5 suggests to me that the learned similarities are in fact not "high-level", but (up to rotation invariance) fairly low-level texture properties like "homogeneous green patch", "bright straight line on dark background", etc. I am not sure what suggests to the authors that there is much sensor invariance. Again, channels with really different wavelength would be more convincing here.
In terms of presentation, the paper repeatedly makes strong, but unsupported claims about the special properties of remote sensing images.  It makes no sense to state that standard pictures often have "one subject" - ImageNet does, but a look at standard datasets like Cityscapes or DepthInTheWild shows that it is not true in general. In the same vein, I don't see why the content changes more unpredictably at the same location across time - arguably the rate of change is much lower, because only large-scale changes impact the image, and because there aren't many dynamic occluders. And the "distributional hypothesis" certainly also holds for remote sensing images, just at different scale - otherwise people would not use all sorts of smoothness priors, super-pixel segmentations, etc. when working with them. And I do not understand why the adjacent-patch approach of CPC needs nearby patches to be similar - in my view it only requires that nearby patches are not conditionally independent, so basically any type of patterns.
A small detail: some references, while not wrong, seem rather arbitrary. E.g., (Wu 2018) is a little-known random example. Why not use earlier, more standard references - for semantic segmentation of remote sensing data with deep learning one could for example think of (Maggiori, Audebert, Marmanis, Kampffmeyer, Sherrah, ...).
Overall, while the paper brings up a valid question, it does not give a convincing answer. The justification of the method and its novelty is to some degree contrived, the dataset is ill-suited to really prove the point, and the natural baselines are missing.


By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering to use multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills. By introducing an extra latent variable for transitional skills and a compensation term for intrinsic reward, our LTS method discovers both primitive and transitional skills by maximizing the mutual information with compensation. By considering various robotic tasks, our results demonstrate the effectiveness of LTS on learning both diverse primitive skills and transitional skills, and show its superiority in smooth transition of skills over the state-of-the-art baseline DIAYN. In addition, we further show the significance of transitional skills for learning downstream tasks.

What is the specific question/problem tackled by the paper?
This paper addresses the problem of learning and using useful skills through unsupervised RL. While past work (VIC, DIAYN) found skills by maximizing the mutual information between skills and states, this paper (LTS) aims to find transitional skills to move between the primitive skills.
The proposed method is, if the primitive skills found by DIAYN are labeled w_i, to find skills z_[i,j,k] which corresponds to the kth state while transitioning from w_i to w_j. These transition skills are learned jointly with the primitive skills using the mutual information objective.
The results in the paper are on mountain car, cart pole, and pendulum. The plots show that distinct primitive skills can be learned in these environments, and that the transition skill transitions between them. The results do not show the actual performance of using these skills for any specific task, which was the motivation presented in the introduction. The
Is the approach well motivated?
The approach is not well motivated. The paper claims that transitional skills are necessary, but does not show any evidence of an agents behavior improving by using transitional skills.
I argue to reject the paper in its current state; Sections 3 and 4 need to be rewritten to be more easily understandable, and the need for transitional skills should be more empirically motivated.
Clarity Issues:
- The method is presented in a very confusing way. I am not confident that I understand how the transitional skills are defined.
- Figure 2: What is the “statistical characteristic” and “feature value”? The vague terms are not defined. Which value? What statistic?
- Figure 5: I see that LTS transitions more smoothly between skills, but why is DIAYN’s fast transition bad? Clearly it still succeeds at reaching the other state.
--------------------------------
Update
Due to the additional experiments that show the benefit of LTS, I have increased my score to weak reject.
Section 6.5: Thank you for adding this additional experiment. This section lacks details about how the experiment was run. How was the meta-policy trained? In “3) LTS-master: Transfer policy with optimal weights learned by meta-policy”, why is there transfer? Transfer from what to what?
“Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills.” What does this sentence mean?
Reading over the paper again, the paper needs several editing passes to fix the grammar, as a number of sections (like above) are not understandable. While the paper has improved and is on a good track, I do not think it is yet ready for publication.


While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predictive representations offline and using these representations for reward shaping, we gain access to reward signals that understand the structure and dynamics of the environment. In particular, our method achieves better learning by providing reward signals that 1) understand environment dynamics 2) emphasize on features most useful for learning 3) resist noise in learned representations through reward accumulation. We demonstrate the usefulness of this approach in different domains ranging from robotic manipulation to navigation, and we show that reward signals produced through predictive coding are as effective for learning as hand-crafted rewards.

*Synopsis*:
  This paper proposes using the features learned through Contrastive Predictive Coding as a means for reward shaping. Specifically, they propose to cluster the embedding using the clusters to provide feedback to the agent by applying a positive reward when the agent enters the goal cluster. In more complex domains they add another negative distance term of the embedding of the current state and goal state. Finally, they provide empirical evidence of their algorithm working in toy domains (such as four rooms and U-maze) as well as a set of control environments including AntMaze and Pendulum.
  Main Contributions:
  - Using the embedding learned through Contrastive Predictive Coding for reward shaping.
  - A reward shaping scheme that seems generally applicable to any embedding.
  *Review*:
  I think the paper provides a compelling motivation, and is well written. I think using the embedding learned through CPC could provide meaningful semantics for representation learning and for reward shaping (as done in the current paper), and encourage the authors to continue down this line of inquiry. Unfortunately, I have several concerns over the method as currently implemented and the empirical comparisons (specifically with the chosen competitors) which I detail below. Given these concerns I am unwilling to recommend accepting this paper, unless several of these concerns are addressed.
  1. This algorithm, by nature, is purely offline as the CPC and clustering all are currently done offline. Furthermore, the clustering portion of this approach requires states to be randomly sampled from the environment to create a nice set of clusters which are representative of the environment's full state space. These two requirements significantly limit this approach, especially when looking at domains where simulation is not possible, or the underlying state distribution is unknown. By and all, I don't think this means we should completely discount this method entirely and the authors do mention this as a detriment to their algorithm in section 6.3. I'm wondering if this paper should look at implementing an online version before publication, but think this is less prescient to the other concerns.
  2. I am concerned about the current policy learning scheme (i.e. tiered policy (1) go to the correct cluster (2) go to the goal) which seems to only be used by your approach. This invalidates the comparisons made with the other reward shaping schemes as this goes beyond reward shaping (you are learning two separate policies).
  3. The current competitors are unsatisfactory as they don't include other reward shaping techniques from the literature. Also the related works section seems to completely disregard this part of the literature. I would recommend comparing your approach to (methods you even cite!):
     - "Reward Shaping via Meta Learning" by Zou et. al https://arxiv.org/pdf/1901.09330.pdf
     - "Potential based reward shaping" by Gao et. al. https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/viewPaper/10930
     (I'm sure there are others beyond these)
  4. I'm also concerned with the current significance of the results, particularly AntMaze, Half Cheetah, and Reacher. I'm especially concerned because I don't feel your competitors are not relevant/representative of the current state of reward shaping.
  5. It would be worthwhile to try this method on several RL algorithms (i.e. Q-learning, TRPO, SAC, DDPG, etc...). This will help readers understand if this approach is a general method, or only applicable to PPO.
  6. I quite like the idea of predictive coding (albeit the original scheme presented by Rao and Ballard 1999) as an unsupervised representation learning scheme, but am unsure this is critical for your method and the current approach is not really predictive coding in a sense (or at least the ideas I'm familiar with from cognitive computational neuroscience). I am concerned with the predictive coding idea being highlighted here as a key ingredient, but none of the papers containing the originating idea of predictive coding are mentioned. Rao and Ballard is one, but there is a rich literature following from this work into the free energy formulation (Friston) and active learning. These ideas should appear in your introduction, as you heavily rely on them. Also there are other predictive coding schemes for unsupervised representation learning (such as PredNets from David Cox https://coxlab.github.io/prednet/), which I believe should be mentioned. In light of these other methods, I'm not sure your discussion on only using predictive features for reward shaping is accurate, and instead these claims should be softened for only features learned through CPC.
  Missing experimental settings:
  - Number of runs tested
  - What are the error bars in your plots?
  I would also like to recommend "Deep Reinforcement Learning that Matters" by Henderson for a reference on how to conduct meaningful Deep RL experiments using policy gradient methods (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/16669). I think this paper could benefit from the recommendations made there.
  More questions/clarifications:

  Q1: What constitutes success in the grid world domains for table 1?
  Q2: If you were to train a single policy using your reward shaping scheme (i.e. not the tiered approach currently employed) how would the new policy perform? Are there problems with the current scheme where you have to have the tiered policy?
  Q3: Why CPC and not say an autoencoder or VAE? A comparison over many types of unsupervised embedding learning algorithms could be interesting and make your method more general than currently presented. It could also strengthen your argument for using CPC.
  Q4: How long were the trajectories for training CPC?
  *Other minor comments not taken into account in the review*
  - I think your labels are backwards in table 1.


We first pose the Unsupervised Progressive Learning (UPL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes increases with time. If some limited labeled data is also available, those representations can be associated with specific classes, thus enabling classification tasks. To solve the UPL problem, we propose an architecture that involves an online clustering module, called Self-Taught Associative Memory (STAM). Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical representations rather than specific examples. The goal of this paper is to introduce the UPL problem, describe the STAM architecture, and evaluate the latter in the UPL context.

The paper presents a new problem formulation in the broader area of lifelong learning.
Specifically, the Unsupervised Progressive Learning (UPL) problem, requires a learner to consume a stream of data, where each data point is associated with a class, but only very few labels are provided. Similar to continuous learning the learner can only access current and not access previous data in the stream. The paper clearly describes how this setup is different to commonly other learning setups studied, including continual learning.
To solve this problem the paper proposes a deep-learning-free approach based on clustering and novelty detection.
Strength:
-	I think the problem formulation is very realistic and interesting and I am not aware of it being explored previously
-	The newly proposed architecture STAM is also interesting.
-	The paper evaluates STAM on UPL on three datasets, MNIST, EMNIST, SVHN
-	The paper ablates several aspects of the model evaluates the effect of hyper parameters.
Weaknesses:
1.	The paper misses to include baselines, both, w.r.t. the proposed method, as well as for the learning problem.
1.1.	The paper argues that deep learning cannot work, I am wondering, why don’t include standard baselines, e.g. auto-encoder based with a prototype stored whenever a labeled example is seen. (For single pass continual learning, see e.g. [B], although that work is supervised)
1.2.	Hebbian learning inspired algorithms might similarly form a valuable baseline
2.	The UPL problem makes sense to me, however, the experimental setup could be improved. Specifically, the paper uses the entire dataset to find hyperparameters. This is highly concerning as this basically means the entire dataset has been seen multiple times before the final pass over the dataset (the one with the best hyperparameters) is performed. This is basically a contradiction to the setup of UPL, which assumes all data is seen only once.
3.	Related Work:
3.1.	[A] seems to study a similar setup and proposes an approach which should also be applicable in this work.

Conclusion:
The paper’s contribution is a new learning setup (UPL) and an approach (STAM). While it remains unclear if STAM can generalize to realistic images, the idea of UPL is very interesting and speaks for accepting the paper, although there are several weaknesses the authors should address.
It would also be great if the authors plan to release the exact experimental setup (i.e. which images are sampled in which order and which ones are labeled) so future work can compare to this work.
Reference:
[A] Continuous Online Sequence Learning with an Unsupervised Neural Network Model
Yuwei Cui, Subutai Ahmad and Jeff Hawkins; Neural Computation, 2016
[B] Gradient episodic memory for continual learning, D Lopez-Paz, MA Ranzato ; NeurIPS, 2017
==== Post rebuttal comment:
Thanks for the clarifications and additional experiments;
I do think the the paper should be accepted and increased my score.
Regarding 2) I was thinking more of using a small part of the dataset to do hyper parameter selection, see e.g.
Efficient Lifelong Learning with A-GEM
A Chaudhry, MA Ranzato, M Rohrbach, M Elhoseiny
International Conference on Learning Representations (ICLR)


ABSTRACT Deep (neural)  networks have been applied productively in a wide range of supervised and unsupervised learning tasks. Unlike classical machine learning algorithms, deep networks typically operate in the overparameterized regime, where the number of parameters is larger than the number of training data points. Consequently, understanding the generalization properties and the role of (explicit or implicit) regularization in these networks is of great importance. In this work, we explore how the oft-used heuristic of data augmentation imposes an implicit regularization penalty of a novel measure of the rugosity or "roughness" based on the tangent Hessian of the function fit to the training data.

This paper shows that a penalty term called rugosity captures the implicit regularization effect of deep neural networks with ReLU (and piecewise affine in general) activation. Roughly, rugosity measures how far the function parametrized as a deep network deviates from a locally linear function.
The paper starts by showing that the amount of training loss increased from adding data augmentation is upper bounded in terms of (roughly) a Monte Carlo approximate to a Hessian based measure of rugosity. It then formally derives this measure of rugosity for networks with continuous piecewise affine activations. Finally, experimental evaluation for classification tasks on MNIST, SVHN and CIFAR shows that data augmentation indeed reduces the rogusity by a significant amount particularly when using the ResNet structure. A somehow surprising message is, however, that if one imposes explicit regularization with rugosity in lieu of data augmentation, then the better generalization usually seen from data augmentation no longer presents, though one does get a network with smaller rugosity.
Comments:
It is quite interesting to see that the rugosity measure proposed in the paper captures at least some aspects of the implicit regularization effect of data augmentation both in terms of theory (i.e. Theorem 1) and practical observations. My feeling is that rugosity is mostly a measure of the smoothness of the function parametrized by the neural network. From that perspective, how is the rugosity as a smoothness measurement for neural networks with piecewise affine activations different from the Lipschitz constant for general neural networks? My guess is that data augmentation also decreases the Lipschitz constant of a neural network near the training data points, but regardless of whether this is true or not, it is not clear if and how rugosity is better than Lipschitz constant for characterizing the implicit regularization of data augmentation.
In addition, there have been many recent studies on showing that gradient penalty / Lipschitz regularization are useful for achieving better generalization and adversarial robustness, see e.g. [a,b,c]. The results in this paper on showing that regularizing rugosity does not improve accuracy seem to contradict with the conclusion of these prior studies. It is unclear to me whether this is caused by insufficient experimentation or if there is any fundamental difference between rugosity and Lipschitz regularization that I am missing.
[a] Finlay et al., Lipschitz regularized deep neural networks generalize and are adversarially robust
[b] Gouk et al., Regularisation of Neural Networks by Enforcing Lipschitz Continuity
[c] Thanh-Tung et al., Improving generalization and stability of GANs