Recent semi-supervised learning (SSL) methods often have a teacher to train a student in order to propagate labels from labeled data to unlabeled data. We argue that a weakness of these methods is that the teacher does not learn from the student's mistakes during the course of student's learning. To address this weakness, we introduce Coaching, a framework where a teacher generates pseudo labels for unlabeled data, from which a student will learn and the student's performance on labeled data will be used as reward to train the teacher using policy gradient. Our experiments show that Coaching significantly improves over state-of-the-art SSL baselines. For instance, on CIFAR-10, with only 4,000 labeled examples, a WideResNet-28-2 trained by Coaching achieves 96.11% accuracy, which is better than 94.9% achieved by the same architecture trained with 45,000 labeled. On ImageNet with 10% labeled examples, Coaching trains a ResNet-50 to 72.94% top-1 accuracy, comfortably outperforming the existing state-of-the-art by more than 4%. Coaching also scales successfully to the high data regime with full ImageNet. Specifically, with additional 9 million unlabeled images from OpenImages, Coaching trains a ResNet-50 to 82.34% top-1 accuracy, setting a new state-of-the-art for the architecture on ImageNet without using extra labeled data.

This paper provides a simple but novel coaching method for teacher-student based semi-supervised learning framework. The coaching method consists of a two-stage update: first update Student network according to the pseudo label produced by Teacher network on the unlabeled dataset; second update Teacher network according to the Student network's performance on the labeled dataset. The authors propose a novel policy gradient update for the Teacher network. The authors evaluate the coaching method on several different semi-supervised learning dataset CIFAR-10, SVHN and ImageNet. The comparison with baselines are thorough. I appreciate that the authors explain the design of the experiments and tuning in details.
I vote for acceptance but I still have some questions. I would be willing to increase my score if the authors address my questions in the rebuttal.
1. The motivation of coaching is not accurate. In the second sentence of Introduction, the authors mention "Although coaches do not play as well as the players". However, we are training neural networks. There is no such thing as "coaches do not play as well as the players". They are the same parametric neural networks. (The authors also use the same architectures for both teacher and student neural networks.) I would remove this analogy sentence.
2. The authors mention that their coaching method beats the fully supervised learning on the CIFAR-10. I am not convinced by this result. The author explained that this is due to the less overfitting in the student network. However, besides coaching, there are many other regularization method we can use to avoid overfitting. Using far less labels in training strictly reduces the amount of information we have. There is a simple test the authors can do. We can use the full labelled data set and use sampled results from Teacher network to train a Student network on it. We can perform coaching on this regime. This coaching on full dataset should do better than coaching on partially-labeled dataset (4000 labels).
3. The state of art for CIFAR-10 is 99% now [1]. It would be nice to see the performance of author's approach applies to the state-of-art network/structure.
References:
[1] Huang, Yanping, et al. "Gpipe: Efficient training of giant neural networks using pipeline parallelism." arXiv preprint arXiv:1811.06965 (2018).
-----
The authors' response does not answer the questions about motivation and why the method works, which is also questioned by the two other reviewers. For example, why the authors need to have two networks is unclear; I guess that true labels should be more helpful to guide a student network to learn than a teacher network. Even though the authors seem to have state-of-art semi-supervised learning results, some extra explanations are needed.


Deep neural nets (DNNs) compression is crucial for adaptation to mobile devices. Though many successful algorithms exist to compress naturally trained DNNs, developing efficient and stable compression algorithms for robustly trained DNNs remains widely open. In this paper, we focus on a co-design of efficient DNN compression algorithms and sparse neural architectures for robust and accurate deep learning. Such a co-design enables us to advance the goal of accommodating both sparsity and robustness. With this objective in mind, we leverage the relaxed augmented Lagrangian based algorithms to prune the weights of adversarially trained DNNs, at both structured and unstructured levels. Using a Feynman-Kac formalism principled robust and sparse DNNs, we can at least double the channel sparsity of the adversarially trained ResNet20 for CIFAR10 classification, meanwhile, improve the natural accuracy by 8.69% and the robust accuracy under the benchmark 20 iterations of IFGSM attack by 5.42%.

Summary: This paper presents an observation that Feynman-Kac formalism principled ResNet ensemble [1] yields sparser network weights compared to those of standard Resnet.  Based on this observation, the authors combine the ensemble model with Lagrangian based sparsification methods to obtain sparse and robust models.
Pros:
- Obtaining sparse and robust networks is an important and challenging problem that could be of interest to a large audience.
- The paper provides an interesting observation that EnResNet [1] yields sparser network weights compared to standard ResNet, and further leverage it to obtain sparse and robust models.
Cons:
- The paper has limited novelty. It has already been shown in the original EnResNet paper [1] that EnResNet is more robust to adversarial attacks. Thus the only additional contribution of this paper is the observation that EnResnet is more sparse than standard ResNet, and combination of EnResnet with sparsification methods.
- The paper is not well justified on why one should use sparsifying techniques such as RVSM, RGSM with the Feynman-Kac formula principled EnResnet. The authors state that this enables sparsity to meet robustness; however, in all experimental results, the robustness actually decreases with the increase in sparsity which is opposite to the claims made in the paper.
- The authors only report robustness on white-box gradient-based attacks. Thus it is not clear whether the method will generalize to black-box/gradient-free attack approaches such as NAttack [2].
Minor comments:
- Why didn’t you report the accuracy on the clean examples? This is important in showing that the method generalizes well to clean examples while maintaining robustness.
- Why use only 20 iterations to evaluate the attack? Will the model maintain its robustness with the increase in the iteration and epsilon?
- Figures are not referenced anywhere in the text.
- It would be better to put the results on CIFAR100 in the main paper rather than in the appendix.
- Page 2, “lower cases”  -> “lower case”.
- Page 2, “Related Works” -> “Related work”, it’s better to have it as a separate section.
Overall, while the paper provides an interesting observation, it has limited contributions due to lack of novelty. Further, inadequate experimental validation makes it difficult to see if the claims made in the paper are actually true. Thus I believe that this paper requires substantial improvements in order to be accepted to top-tier publication venues such as ICLR.
References:
[1] Bao et al. 2018, https://arxiv.org/abs/1811.10745
[2] Yadong et al. 2019, https://arxiv.org/abs/1905.00441


Neural networks have significantly benefitted real-world tasks. The universality of a neural network enables the approximation of any type of continuous functions. However, a neural network is regarded as a non-interpretable black box model, and this is fatal to reverse engineering as the main goal of reverse engineering is to reveal the structure or design of a target function instead of approximating it. Therefore, we propose a new type of a function constructing network, called the white box network. This network arranges function blocks to construct a target function to reveal its design. The network uses discretized layers, thus rendering the model interpretable without disordering the function blocks. Additionally, we introduce an end-to-end PathNet structure through this discretization by considering the function blocks as neural networks

This paper presents White Box Network (WBN), which allows for composing function blocks from a given set of functions to construct a target function. The main idea is to introduce a selection layer that only selects one element of the previous layer as an input to a function block. This allows for both introducing function priors as well as interpreting the learned function. The paper also presents a setting where each function block is a neural network that can be learned end-to-end using a PathNet style setting and shows positive transfer across MNIST and CIFAR classification tasks.
This presents an interesting technique to enforce learning composition of function blocks while learning the target function. This is important for both interpretability of the learned function as well as for introducing prior information about useful functions in a given domain. The extension to learnable functions (in the form of neural networks) for learning pathways for different tasks is also promising.
The biggest weakness of the paper is that it does not compare both theoretically as well as empirically with several closely related techniques such as RoutingNetworks[1], Modular Networks[2], Neural RAM[3], Compositional Recursive Learner[4], etc. (please find the references below). Routing Networks allow for selecting among a set of function blocks given some inputs. Module Networks similarly introduce modular layer that determines the appropriate modules given the inputs from the previous layers. Neural RAM learns to compose differentiable functions to learn a target function (similar to learning the LLD programs).
It would be good to describe the differences between the proposed approach in WBN and these approaches, as they all seem to propose a similar solution of learning target functions by learning to compose function blocks and reusing the learnt computations for transfer learning. It would also be important to empirically evaluate the related approaches to better understand the pros/cons of WBN compared to these approaches.
What is the biggest size LLD programs that can be learned by WBNs? It would be interesting to evaluate the scalability of the approach to understand the limits of learning complex target functions.
I was also curious if instead of providing the four pre-defined function block (Identity, not, and, or), what would the behavior be if they were all neural networks and also learnt in an end-to-end fashion somewhat similar to [4].
For the MNIST and CIFAR classification tasks, the function blocks are neural networks themselves. After training them using the PathNet like training, is the learned network more interpretable? It might be interesting to see if the selection layer and the function blocks learned some semantic information that might be easier to distill.
For a better comparison with Neural RAM, it might also be interesting to empirically evaluate the performance of WBNs on algorithm induction tasks such as the one used in [3].
[1] Clemens Rosenbaum, Tim Klinger, Matthew Riemer. Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning. ICLR 2018
[2] Louis Kirsch, Julius Kunze, David Barber. Modular Networks: Learning to Decompose Neural Computation. NeurIPS 2018
[3] Karol Kurach, Marcin Andrychowicz, Ilya Sutskever. Neural Random-Access Machines. ICLR 2016
[4] Michael Chang, Abhishek Gupta, Sergey Levine, Thomas Griffiths. Automatically Composing Representation Transformations as a Means for Generalization. ICLR 2019
Minor:
page1: ?. -> ?
page 2: questions and answering --> question answering
page 3: y^(l) represents (m(l)+n(l))-dimensional vectors --> should this be m(l) + 2*n(l)?


We present a novel multi-source uncertainty prediction approach that enables deep learning (DL) models to be actively trained with much less labeled data. By leveraging the second-order uncertainty representation provided by subjective logic (SL), we conduct evidence-based theoretical analysis and formally decompose the predicted entropy over multiple classes into two distinct sources of uncertainty: vacuity and dissonance, caused by lack of evidence and conflict of strong evidence, respectively. The evidence based entropy decomposition provides deeper insights on the nature of uncertainty, which can help effectively explore a large and high-dimensional unlabeled data space. We develop a novel loss function that augments DL based evidence prediction with uncertainty anchor sample identification through kernel density estimation (KDE). The accurately estimated multiple sources of uncertainty are systematically integrated and dynamically balanced using a data sampling function for label-efficient active deep learning (ADL). Experiments conducted over both synthetic and real data and comparison with competitive AL methods demonstrate the effectiveness of the proposed ADL model.

The authors consider active deep learning. They propose decomposing predictive entropy into a) vacuity (lack of evidence) and b) dissonance (contradictory evidence). They frame this in terms of "subjective logic". In practice this is achieved by having the NN output the parameters of a Dirichlet, which allows an additional degree of freedom describing variance/vacuity. Dissonance is defined in terms of the support of contradictory classes. To get improved estimates of vacuity they augment the loss with a term regularizing the Dirichlet parameters to be small (low precision) at unlabelled points with higher KDE(unlablled points) than KDE(labelled points). They propose initially weighting vacuity and later dissonance as AL proceeds. Encouraging results are presented on simulated 2D data, MNIST and CIFAR10.
I think the basic idea of separating vacuity and dissonance is interesting, and the demonstration of the failings of existing ADL approaches is valuable. It wasn't clear to me how this "subjective logic" theory gets you to the specific definitions of vacuity/dissonance, or whether these were just proposed by the authors. Equation 6 seems to come out of nowhere (whereas the rest of the derivations using the Dirichlet are very intuitive).
The idea of encouraging the network to be uncertain far from data is also reasonable. While some Bayesian models such as Gaussian process regression with a RBF kernel give you this for free, it is certainly true that DL methods do not have this characteristic in general. Regularizing the r to be small seems like a reasonable way to do this, but I'm not convinced by the kernel density estimate part. DNNs can operate on very high-dimensional, structured inputs. Even the simplest of these, images, requires some degree of spatial invariance (achieved using convolutions) to obtain meaningful predictions. I find it very hard to believe a KDE can do anything meaningful in such spaces, even if you could find a sensible bandwidth (which isn't discussed at all). It is possible of course that random selection of unlabelled points to regularize in this way would work just as well. Unfortunately no ablation study is performed, so we don't know what the individual contribution of the three proposals (moving from vacuity to dissonance, augmented loss and Dirichlet likelihood) is.
How sensitive is the method to the vacuity/dissonance weighting?
The improvement over competing methods for MNIST and CIFAR10 appears to mostly manifest after the initial 20 or so acquistions.
Do these results extend to batch AL? For many applications that's more important.
Overall I thought this paper had some promising ideas but they need to be more thoroughly tested empirically to give some sense of how robust and generalizable the approach is.


We study the robustness of image classifiers to temporal perturbations derived from videos. As part of this study, we construct ImageNet-Vid-Robust and YTBB-Robust, containing a total 57,897 images grouped into 3,139 sets of perceptually similar images. Our datasets were derived from ImageNet-Vid and Youtube-BB respectively and thoroughly re-annotated by human experts for image similarity. We evaluate a diverse array of classifiers pre-trained on ImageNet and show a median classification accuracy drop of 16 and 10 percent on our two datasets. Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points. Our analysis demonstrates that perturbations occurring naturally in videos pose a substantial and realistic challenge to deploying convolutional neural networks in environments that require both reliable and low-latency predictions.

This paper presents new datasets based on ImageNet and Youtube-BB to assert networks performance consistency across time. Compared to previous work, it uses human labeler to further validate the dataset and discard frames that are deemed too different from the reference one. It provides results on image classification and detection using popular network architectures. Based on these results, the paper claims an accuracy drop of 10 to 16%.
The main contribution of this paper is to introduce a new, human annotated dataset for robustness assessment of image classifiers. In itself, it is valuable work, but it is not clear if the contribution is important enough for ICLR. However, I would still be ok with accepting the paper (better datasets are always useful) if it was not for the way the results are reported. I do not specifically have issues with "more stringent robustness metric" but it should not be used to claim incredible results (like an accuracy drop of 10 to 16% instead of 3% for previous work (Real et al. 2017)).
There is one thing for sure: using "accuracy drop" in this context is just misleading. The underlying concept to which "accuracy" refers is _not_ the "maximum error made by the network over the whole set of images". By this definition of accuracy, if the number of images around the reference frame were 100, missing a _single one_ each time (that is 99% of actual accuracy) would result, according to this peculiar redefinition, to a _0%_ accuracy. This is actually highlighted in Appendix G.1: the "accuracy" trend can only go down, since every supplemental frame brings one more chance to fail and obtain a 0% accuracy for this set of perturbed images.
Same thing goes for the detection, where the only frame that matters among all is the one _minimizing_ the AP. Same thing in Table 4, where the "accuracy" of the Original column means one thing (the amount of correctly identified images over the total number of images) while the "accuracy" of the Perturbed column right next to it means something completely different. Same thing in Table 2, which even provides a "delta" between two unrelated metrics.
I cannot see how this can be justified. Sure, there could be some usage for such strict metric, but again, this is _not_ accuracy and cannot be compared to any previous results. Having a more stringent metric is one thing, but in this case it just seems like a justification to get high drop numbers.
Keeping that in mind, these are the actual conclusions we can make from the paper:
1) Human reviewers removed or changed about 20% of the frames
2) This resulted in a relative accuracy improvement of about 4% for the reference frame (Table 4, column Original). The improvement for the perturbed frames are not actually provided.
3) The comparison (and improvements) to previous work due to the dataset cleaning remains unclear.
4) Comparison between different networks and training procedures
Overall, the paper presents impressive numbers but does not actually back them up. I am open to eventually consider acceptance given the value of the datasets, but the paper would then require a significant overhaul to remove all confusing aspects.


Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples. Inspired by the observation that the intrinsic dimension of image data is much smaller than its pixel space dimension and the vulnerability of neural networks grows with the input dimension, we propose to embed high-dimensional input images into a low-dimensional space to perform classification. However, arbitrarily projecting the input images to a lowdimensional space without regularization will not improve the robustness of deep neural networks. We propose a new framework, Embedding Regularized Classifier (ER-Classifier), which improves the adversarial robustness of the classifier through embedding regularization. Experimental results on several benchmark datasets show that, our proposed framework achieves state-of-the-art performance against strong adversarial attack methods.

Summary
========
This paper proposes to make neural networks robust to adversarial examples via dimensionality reduction.
The paper builds on and compares to prior approaches (e.g., MagNet and Defense-GAN) that are known to be broken, which casts serious doubts on the validity of the performed experimental evaluation.
The theoretical part of the paper is also very hard to follow.
For these reasons, I recommend rejection.
Comments
=========
The way I understand the proposed framework, it is trying to compress the data into a form that is as close as possible to a standard Gaussian distribution, while maintaining classification accuracy.
The intuition, formulated next to Figure 1, seems to be that compressing the data will force the network to only retain the most important features for classification, which will thus lead to higher robustness.
I have two main concerns with this approach:
- First, recent work by Ilyas et al ("Adversarial examples are not bugs, they are features") seems to suggest that models tend to learn features that are not robust yet generalize well. It is unclear why compressing the data would remove such features.
- Second, it is unclear why constraining the compressed data to be close to a Gaussian would have a positive effect on robustness. The authors claim that this is to remove some "pathological" mappings from the input space to the embedding space. Can you explain what you mean here? What are these pathological mappings and why should they be the source of non-robustness?
The theoretical analysis in Sections 2.3and 3. is very hard to follow. The authors first mention that "optimal transport theory [...] provides a much weaker topology than many other [distances]". What do you mean by this? What is the advantage of optimal transport theory that you are trying to exploit?
When Kantorovich’s distance is first introduced, the reader has no idea what any of the symbols represent. What are Y, U, \mathcal{U}, P_Y, P_C, etc?
The authors then mention a relation to the Wasserstein distance and to Wasserstein GANs. How does this relate to this paper?
I could not understand how Algorithm 1 related to the discussion in Section 3.
Algorithm 1 simply uses a standard cross-entropy loss to train the classifier C. Is this what you mean by minimizing the optimal transport cost between P_Y and P_C? This seems like a very convoluted way of justifying the use of the most standard loss function in ML.
Algorithm 1, step 6 mentions sampling from Q(Z|x_i). But isn't the encoder Q deterministic? If not, where does the randomness come from?
In the experimental section, the authors compare to Defense-GAN, which was shown to be broken in "The Robust Manifold Defense: Adversarial Training using Generative Models" by Jalal et al.
The authors mention that they evaluate robustness using an untargeted white-box PGD attack, but they do not specify which objective their attack is optimizing. As this papers proposed a different classification pipeline, adaptive the attacks to this pipeline is crucial.
The experimental results (Figure 2 & 3) raise a number of questions:
- Why does the ER-classifier variant without adversarial training perform so much better than the one with? The author's explanation about the difficulty of optimizing over the embedding space should be supported.
- Do the attacks reach 0% accuracy for large enough epsilon? The very high accuracy even for large epsilon suggests that the evaluated attacks are not evaluating the right objective to fool the classifier. E.g., on MNIST, eps=0.4 can nearly destroy the whole image so the accuracy should be expected to be lower.


In this paper, we develop a theory about the relationship between Ginvariant/equivariant functions and deep neural networks for finite group G. Especially, for a given G-invariant/equivariant function, we construct its universal approximator by deep neural network whose layers equip G-actions and each affine transformations are G-equivariant/invariant. Due to representation theory, we can show that this approximator has far fewer free parameters than usual models.

This paper proposes a universal approximation theorem for functions invariant and equivariant to finite group actions. It constructs the approximations using fully-connected deep neural networks with ReLU activations. It proves a bound on the number of parameters in the build equivariant model. The proof structure uses a decompostion of G-equivariant functions into Stab(1)-invariant functions, which are represented by a particular network.
The proof structure is reasonably well written, however more context and examples could help a reader unfamiliar with the literature. This could be achieved by proposing applications of the construction of the approximating network to a few concrete applications where equivariant representations are needed.
As-is the paper is difficult for a newcomer to the field, compared e.g. with (Maron et al. 2019b) and (Keriven & Peyré 2019) which have clearer expositions. On the other hand, the equivariant representation problem is given in more generality, stemming from finite group actions represented by actions of permutation groups.
My current assessment is “borderline” and might change in light of author responses and reviews from reviewers with more experience in this field to judge the significance of the results.
I think it is necessary to do a more in-depth comparison with respect to the existing work of (Keriven & Peyré 2019).
In this regard, I think the following could be done:
- Giving an application and the corresponding construction where the increased generality w.r.t. the graph networks of (Keriven & Peyré 2019)
- In the case of a graph network already addressed by (Keriven & Peyré 2019) , how would this construction apply and how would the constructed network compare?
What do the authors call "usual models" in the discussion of Theorem 2.3? I assume this means models that do not exploit the equivariance of the function, but this could be more explicit.
Typos:
p.5, in the discussion of Theorem 2.1: "> Then, by ... we may assume"
p.8, Prop 4.1 Proof: n devides M
p.11: Prop A.1 Proof: can be realized as a permutation action on R^n: R should be in \mathbb
p.17: “ane” instead of “and”
***
Updated review
I have the impression all 3 reviewers have had some trouble going through the main messages of the paper. Although - at least in my case - this might be partly due to limited in-depth expertise, this seems to indicates some deeper shortcomings in the writing, as well as the insights and intuitions offered by the paper, in order to be accessible to a larger audience.
I still believe the approach may have merits, however I do not recommend acceptance of the paper at its current state. In my opinion, the following points should be considered for a future resubmission:
* A friendlier introduction to the matter with more intuitions and examples where the method has interest and distinguishes itself from existing papers, e.g. (Maron et al. 2019b), (Keriven & Peyré 2019) (Review #2, review #3)
* At least 1 example (and possibly more) where the proposed method leads to a practical architecture (Review #1)
* A clearer explanation of the particular role played by Sn w.r.t. general finite groups in the construction (Review #1)


Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly. In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l 2 -defenses. Recent work (Cohen et al., 2019) shows that randomized smoothing can be used to provide certified l 2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER). The attack-free characteristic makes MACER faster to train and easier to optimize. In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN. For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius. Our code is available at https://github.com/MacerAuthors/macer.

This paper improves the robustness of smoothed classifiers by maximizing the certified radius, which is more efficient than adversarially train the smoothed classifier and achieves higher average robust radius and better certified robustness when the radius is not much larger than the training sigma. It proposes a novel objective which is derived by decomposing the 0/1 certified loss into the sum of 0/1 classification error and 0/1 robustness error. Three conditions are identified to make the optimization doable. Two surrogate losses (CE and hinge loss on the certified radius) for the two 0/1 errors are proposed as upper bounds of the 0/1 loss. Certified radius is derived as a function of the logits of Soft-RS to make the hinge loss differentiable. Numerical stability of the proposed objective is also analyzed by showing its gradient is bounded.
In general, the paper is well-written and the proposed objective is novel to my knowledge. I tend to accept the paper. Still, I am not sure about how much MACER improves upon the baselines, and would like to ask some questions.
1. Cross entropy is used as a surrogate for the 0/1 classification error. This is true for all cases (including all experiments in this paper) except for binary classification, where the cross entropy is less than 1 when the score on the correct class is around 0.5. It is not important but would be better if you could mention this point.
2. Have you ever tried using a tighter upper bound for the 0/1 classification error, e.g., using cross entropy loss only for the wrongly classified samples? How does it affect the results?
3. Despite showing better results, MACER seems to be using much more epochs than the two baselines (but the total hrs is smaller than (Salman et al. 2019)). Also, MACER is using a much larger k than (Cohen et al., 2019). From Figure 3 (a) we can see a larger k improves the result a lot, and from Figure 3 (b) it seems that setting lambda to a non-zero value only improves the accuracy when the radius is large. For fair comparisons, could the authors give the ACR with different values of lambda while keeping other hyper parameters unchanged? Is Salman's method still not as good when using the same number of epochs?


We propose Symplectic Recurrent Neural Networks (SRNNs) as learning algorithms that capture the dynamics of physical systems from observed trajectories. An SRNN models the Hamiltonian function of the system by a neural network and furthermore leverages symplectic integration, multiplestep training and initial state optimization to address the challenging numerical issues associated with Hamiltonian systems. We show SRNNs succeed reliably on complex and noisy Hamiltonian systems. We also show how to augment the SRNN integration scheme in order to handle stiff dynamical systems such as bouncing billiards.

This paper proposes to represent a Hamiltonian model of a physical system by a neural network. The parameters are then adjusted, so that the observations are considered maximally likely under a probabilistic model. The novelty is to consider a symplectic Leapfrog integration scheme for the Hamiltonian system, which is known to conserve important quantities such as volume in the state space. The proposed approach is shown to outperform the recent work "Hamiltonian Neural Networks" by a large margin on mass-spring chain dynamics and three body systems. The approach can even handle stiff dynamical systems such as bouncing billiards.
Overall, the work is solid contribution and a reasonable improvement over the recent work on HNNs is demonstrated. Therefore I recommend acceptance of the paper. However, I have some fundamental doubts on the motivation on this line of works. This might be, because I'm not too familiar with the subject, and I'm willing to increase my score if the doubts are cleared.
In the shown examples, to best of my knowledge, the "exact" Hamiltonians describing the physics of the system are well-known. Therefore, I'm unsure what is the advantage of trying to learn physical laws, that are already well understood. The paper argues that the learned Hamiltonian will correct for errors in the discretization, but one could instead use a better integration scheme or a finer time-discretization, based on classical theory which has been developed over the last 50 years which comes with strong convergence guarantees and error bounds. I would have liked to see a stronger motivation, why it is interesting to learn an Hamiltonian of a system, where the exact Hamiltonian is already known. It would also be enlightening to see some plots, which illustrate how "far" the learned Hamiltonian is from the analytical one.
Of course, one might argue that the ultimate goal is to have a learning based approach discover physical laws so far unknown to humans,  just from observations. But it is unclear why the inductive bias that the observations are generated by a Hamiltonian might be reasonable. It could very well be, that the law cannot be described by a Hamiltonian system.
From a high-level point of view, one might even argue that it is not too surprising that one can fit a parametrized Hamiltonian to observations generated by a Hamiltonian system better than a general purpose function approximator without such an inductive bias or better than a system based on a naive/unsuitable non-conservative integrator.
As a remark, often the exact Hamiltonian is known to be (strictly) convex. I'm wondering whether convex function approximators such as convex neural networks could provide an even stronger inductive bias. But it might be that a general purpose RNN can account better for the discretization errors.


We study the problem of defending deep neural network approaches for image classification from physically realizable attacks. First, we demonstrate that the two most scalable and effective methods for learning robust models, adversarial training with PGD attacks and randomized smoothing, exhibit limited effectiveness against three of the highest profile physical attacks. Next, we propose a new abstract adversarial model, rectangular occlusion attacks, in which an adversary places a small adversarially crafted rectangle in an image, and develop two approaches for efficiently computing the resulting adversarial examples. Finally, we demonstrate that adversarial training using our new attack yields image classification models that exhibit high robustness against the physically realizable attacks we study, offering the first effective generic defense against such attacks.

[Note: I gave a 3 for thoroughness even though I only read the paper once, because I believe that I carefully considered the paper while reading it.]
This paper argues that threat models such as L-inf are limited when considering physically realizable attacks, and provides evidence for this by showing that L-inf adversarial training is insufficient to fully confer robustness against physically realizable attacks in the literature such as the adversarial glasses attack. The paper then proposes an alternate threat model based on contiguous rectangular regions, and shows that adversarial training against this model does far better.
Overall this is a strong paper with thorough experiments, and was for the most part carefully written (although see some quibbles below). I particular liked the paper's carefully distinction between physically *realizable* and physically *realized* attacks, and the admission that not all realizable attacks would fall under the threat model (while still justifying why the model is interesting).
I am on the border of weak accept and strong accept. The main two points keeping me from strong accept are discussed below (and seem perhaps addressable by the authors):
1. The assertion that rectangular occlusions might be a fruitful model for realizable attacks is only lightly tested, since the attacks considered in this paper all fall into the rectangular occlusions threat model. Since one of the key points in the paper is that training in the L-inf threat model could leave vulnerabilities to non-Linf attacks, we might expect the same with rectangular occlusions. A more convincing demonstration would be to test robustness to physically realizable attacks that fall outside the rectangular model (perhaps even skewed or rotated rectangles, although a less synthetic example would be even better). A more minor point is that it would be nice to test the robustness of models trained on rectangular occlusions against L-inf adversarial attacks. This is relevant to knowing whether training on occlusions is giving up on adversarial robustness or actually helps against both (I actually think it's plausible that you would do well on L-inf, but it seems worth testing either way).
2. The high-level claimed take-away that adversarial training does not help against physically realizable attacks seems false in light of Figure 2, which shows that L-inf adversarial training substantially improves robustness relative to the baseline. A more accurate take-away would be that on some datasets adversarial training helps but still leaves a gap, while on others it does not help at all and perhaps hurts. I would prefer more careful wording as a reader who only goes through the introduction might not see Figure 2.
On #1, I should stress that the paper would still be interesting regardless of the outcome of the two experiments in #1; I just think that for thoroughness it would be nice to include them. The existing experiments are already thorough so this would be going above and beyond, but that is why it might help raise my score from weak to strong accept.
EDITED TO ADD: For #2, I would also be happy with an argument from the authors as to why the current language appropriately describes the experiments. I do not wish to dictate to the authors what their take-aways are, but more to open up for discussion a point that seemed slightly sloppy to me.
Minor comments:
Please avoid subjective intensifiers: "We then use an extensive experimental evaluation to demonstrate that our proposed approach is far more robust against physical attacks on deep neural networks than adversarial training and randomized smoothing methods that leverage lp-based attack models." Both "extensive" and "far" are unnecessary.
Make sure to use \citep vs \citet correctly.
First sentence of 2.1 is too verbose. Overall the prose in that paragraph is turgid, due to too many action phrases being turned into nouns. E.g. "The focus is on", "The typical goal is" both indicate *action* and could be profitably turned into verbs. See Williams and Bizup's book on Style.
"since our ultimate goal is to defend against physical attacks, untargeted attacks that aim to maximize error are the most useful" This seems weak; I don't understand why an attack being physical should go in line with it being untargeted; oftentimes an attacker will have a specific targeted goal.
"another advantage of the ROA attack is that it is, in principle,easier to compute than,  say,l∞-bounded attacks" This seems incongruous with the subsequent text, which admits that ROA if implemented naively would be slower than L-inf attacks.


Face completion is a challenging conditional image synthesis task. This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. We present structure-aware and frequency-oriented attentive GANs. The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective method of integrating the detected landmarks in generative learning. It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training. The proposed frequency-oriented attentive module (FOAM) encourages GANs to attend much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures. The learned FOAMs show a strong pattern of switching their attention from low-frequency to high-frequency signals. In experiments, the proposed method is tested on the CelebA-HQ benchmark. Experiment results show that our approach outperforms state-of-the-art face completion methods. The proposed method is also fast with mean inference time of 0.54 seconds for images at 1024 × 1024 resolution (using a Titan Xp GPU).

This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. It combines the masks, landmarks, corrupted images as inputs to generate completed images in high-resolution. The proposed frequency-oriented attentive module (FOAM) encourages GANs to highlight much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures.
Integrating the Progressive growing GAN (PGGAN) for high-resolution face completion is an interesting step-up work after the success of PGGAN on high-resolution image generation.
Basically the FOAM is proposed to merge the images from different resolution levels instead of a weighted summation. The novelty, however, is not strong as the simple summation provides reasonable performance as convinced in the progressive growing GAN paper. And there lacks quantitative analysis on ablation study as showed in Figure 4. Therefore, the contributions of each component are not convincing.
Using Progressive Growing GAN for high-resolution face completion is also studied in [1]. Figure 17, Figure 18, indicates the author used the same structure as used by [1]. However,  [1] is not cited and there is no comparison between these two models in the paper.
[1] Zeyuan Liu, et al  "High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks "


Despite the remarkable development of recent deep learning techniques, neural networks are still vulnerable to adversarial attacks, i.e., methods that fool the neural networks with perturbations that are too small for human eyes to perceive. Many adversarial training methods were introduced as to solve this problem, using adversarial examples as a training data. However, these adversarial attack methods used in these techniques are fixed, making the model stronger only to attacks used in training, which is widely known as an overfitting problem. In this paper, we suggest a novel adversarial training approach. In addition to the classifier, our method adds another neural network that generates the most effective adversarial perturbation by finding the weakness of the classifier. This perturbation generator network is trained to produce perturbations that maximize the loss function of the classifier, and these adversarial examples train the classifier with a true label. In short, the two networks compete with each other, performing a minimax game. In this scenario, attack patterns created by the generator network are adaptively altered to the classifier, mitigating the overfitting problem mentioned above. We theoretically proved that our minimax optimization problem is equivalent to minimizing the adversarial loss after all. Beyond this, we proposed an evaluation method that could accurately compare a wide-range of adversarial training algorithms. Experiments with various datasets show that our method outperforms conventional adversarial training algorithms.

The paper proposes to construct adversarial attacks by training a neural network to produce a distortion, rather than by constructing the distortion directly via PGD. It then uses this during adversarial training to produce networks that it purports to be more robust.
Pros:
-Faster speed of adversarial training could substantially democratize ability to work on adversarial ML
-Neural net attack could overcome gradient masking in some situations
Cons:
-Some of the main arguments seem incorrect, although the paper may still be strong if these incorrect arguments are removed and the experiments are further substantiated.
Details:
I think there are some interesting ideas here, which if further substantiated could be the base of a strong paper. However, I found the overall argument in the current paper to be misleading or at least mistaken. A core claim is that the neural-net generated attack is better than PGD because PGD only generates a "single attack" while the neural network generates many different attakcs. But PGD is an adaptive attack method that depends on the current model parameters, so it changes over the course of training just like the neural net attacker. Furthermore, both models approximate the same maximization over the L2 ball (in the neural net's case, rather than maximizing over the L2 ball we penalize the L2 norm, but a similar modification can easily be made to PGD). So it is not clear why we should expect the neural net to perform better, except perhaps that it is a different type of way of approximating the same objective that PGD approximates, and perhaps this approximation has some favorable properties? But this is never justified.
Here is one attempt to justify why the neural net attack should be better: perhaps while PGD often works well, it gets stuck on certain examples, e.g. due to gradient masking issues. While the neural net might initially get stuck on those examples (it also has to work by taking gradients), since it can share knowledge across examples it initially learns a good direction for the "stuck" examples by generalizing from the "unstuck" examples. This overall allows it to generate a more consistently good attack even in the presence of (partial) gradient masking. If we believe this story, then actually the neural net attacker should already work well just as an attack (i.e. just train it to generate good attacks against a fixed model for some set of examples). I would find this quite interesting if true but unfortunately it wasn't explored in the paper. One way to do it would be to take models that are believed to be robust based on attacking them with PGD, and then show that the neural net attacker is substantially more accurate.
I do find the point about faster speed interesting. This is given as a minor comment but it may be the biggest benefit of the current method. Adversarial training on large-scale datasets like ImageNet is computationally infeasible, and your method could potentially address this and allow people who don't have $100k+ compute budgets to actually work on adversarial ML.
I also have some comments on the evaluation:
-Reporting average accuracy across attacks in the evaluation is an inappropriate summary statistics. Min accuracy would be better, which in this case is zero for all methods. This suggests that you either did not adversarially train appropriately, or used too large of a norm (what norm did you use anyways? Or is this allowing the norm to be unbounded until you fool the model? In that case all the numbers should be zero for any reasonable attack.)
-For evaluation I would rather see the full curve of accuracy vs. allowed attack norm, rather than just a single summary metric. It's too hard to tell what is going on from a single number. It is also hard to compare against adversarial training methods that penalize the L2 norm versus constrain the L2 norm, as the former might do better than the latter simply due to more closely conforming to your evaluation of average distortion. Having the full curve helps better assess this. Average distortion is also only meaningful if the method is finding the minimum-norm attack point that changes the label, which most of the methods you consider do not do. You do include the full curve in Figure 4 (please make the font bigger though). Under the full curve it seems the improvement over other methods is minimal, and could possibly be due to hyperparameter tuning.
Finally, more minor but the Gaussian derivation does not make sense. It is unlikely that your norms actually follow a Gaussian distribution. A more appropriate claim would be that you are constraining or penalizing the expected norm of the perturbations. I would remove the math part as it does not add anything to the paper (and is also wrong as per preceding comment) and focus more on how you actually construct the generator network (this is only briefly discussed in the appendix and not at all in the main text, even though it is a key point to getting the method to work). You could also use the space for more detailed experiments, following best practices as in https://arxiv.org/abs/1902.06705 to ensure that your evaluation is sensible.


Generative Adversarial Networks (GANs) have been impactful on many problems and applications but suffer from unstable training. Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence. We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs. The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process. We establish the generalization error bound of iWGANs. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs. The empirical experiments show that our model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample. We illustrate the ability of iWGANs by obtaining a competitive and stable performance with state-of-the-art for benchmark datasets.

In this paper, an inference WGAN model (iWGAN) is proposed that fuses autoencoders and WGANs. By working on both the prime and dual problems, the proposed iWGAN takes an objective that is in general the upper bound of WGAN. The generalization error bound of iWGAN is analysed and its probabilistic interpretation under maximum likelihood estimation is provided.  The proposed iWGAN model is validated on both synthetic and real (MNIST and CelebA) datasets.
Pros:
This paper not only proposed the iWGAN model, but also provided some theoretical analysis about the iWGAN, such as the generalization error bound and the probabilistic interpretation.
Through the experiment, it seems that the proposed iWGAN works.  Especially, on the synthetic dataset, iWGAN seems to  have less mode collapse than WGAN.
Cons:
The major concerns of this paper lie in its experiment.
(1)	There is no quantitative comparison between iWGAN and WGAN, especially on the two real image datasets.  There are only some visual examples of the results in either the main body of the text or the appendix.
(2)	The proposed iWAN is only compared with WGAN.  There is no experimental comparison with other autoencoder GANs  in the literature,  although the paper stated that iWGAN has many advantages over other autoencoder GANs (in the abstract).
In addition to experiment, other concerns include:
(1)	The proposed method needs to be better motivated, although it is interpreted under the framework of maximum likelihood estimation. For example, what is the advantage to optimize the proposed upper bound of WGAN over its original objective? In the experiment, it seems iWGAN has less mode collapse than WGAN based on the synthetic data. But there is no explanation or discussion about why iWGAN could enjoy this favourable property.
(2)	The proposed framework is tightly based on WGAN. It is not clear whether it could be extended to other GANs, which limits the significance of the proposed work.
Due to the above concerns, rating is recommended as "3 Weak Reject."
Suggestions:  The authors are encouraged to provide extensive comparison with other autoencoder GANs and WGAN, especially in quantitative way.


Recent progress in image recognition has stimulated the deployment of vision systems (e.g. image search engines) at an unprecedented scale. As a result, visual data are now often consumed not only by humans but also by machines. Meanwhile, existing image processing methods only optimize for better human perception, whereas the resulting images may not be accurately recognized by machines. This can be undesirable, e.g., the images can be improperly handled by search engines or recommendation systems. In this work, we propose simple approaches to improve machine interpretability of processed images: optimizing the recognition loss directly on the image processing neural network or through an intermediate transforming model, a process which we show can also be done in an unsupervised manner. Interestingly, the processing model's ability to enhance the recognition performance can transfer when evaluated on different recognition models, even if they are of different architectures, trained on different object categories or even different recognition tasks. This makes the solutions applicable even when we do not have the knowledge about future downstream recognition models, e.g., if we are to upload the processed images to the Internet. We conduct comprehensive experiments on three image processing tasks with two downstream recognition tasks, and confirm our method brings substantial accuracy improvement on both the same recognition model and when transferring to a different one, with minimal or no loss in the image processing quality.

The goal in this work is to improve machine interpretability of images.
The authors main claims are:
-	Their proposed approach improves image recognition accuracy even without knowing subsequent recognition tasks and recognition models used to perform them (transferable model to different recognition models/tasks).
-	For this they propose what they call “Recognition-Aware” processing that combines image processing loss and recognition loss.
-	The approach is evaluated on three image processing tasks with two downstream recognition tasks:
o	Image super-resolution, de-noising, and JPEG-de-blocking processing tasks, with
o	Image classification and object detection recognition tasks.
The paper is well written and organized, experiments carried are extensive but the reuse of known neural networks, many simplifications (shortcuts), a not clear enough methodology (see below), limited processing & recognition tasks used to support it, do not justify in our opinion the main (over-arching) work’s claim:
-	In 3.2 optimizing recognition loss/Last paragraph:  “Interestingly, we find that image processing models trained with the loss of one recognition model R1, can also boost the performance when evaluated using recognition model R2, even if model R2 has a different architecture, recognizes a different set of categories or even is trained for a different task.”.
The paper would greatly benefit (to understand the context of the work or the explanations provided) from clarification of the many under-defined, not clearly introduced concepts it carries:
-	Meaning of “Network” is not clearly defined:
o	Abstract: “image processing network”.
o	Introduction: “the network maps an image to a semantic label”
o	Later in the paper only networks introduced are deep neural networks. That should be clear from beginning of the paper.
-	“Retraining/Adaptation”  in 1st paragraph page 2.
-	In 1. Introduction/Paragraph 1: You use “.. techniques .. have been proposed for making the output images look natural to human”:
o	Noise is part of nature. A de-noised (smoothed) image is not “more natural”.
o	Enhanced (processed) images are not necessarily “more” natural, rather they take advantage of the human visual perception characteristics to enhance recognition for example.
-	In 1. Introduction/Paragraph 3:
o	“.. of great importance that the processed images be recognizable”  Should explain the concept of image recognition! Because it could be related to contained objects, overall description (for captioning for example) etc.
-	“Image processing” in the context of the paper is intended only as “image enhancement for recognition”. Pattern detection, segmentation, object extraction etc. are not included in this restrictive definition. Should specify for example: image enhancement and restoration.
-	Figure 1: As an illustration, it’s completely counterproductive for your discourse as many simple image recognition algorithms would recognize the bird even in the noisy image.
-	In 3. Unsupervised optimization of recognition loss: The “unsupervised RA” process is not  clear enough to us especially the statement:
o	“.. only “unsupervised” for training model P, but the target pre-trained model R can still be trained in full supervision.”.
-	“We may not know what network architectures (e.g. ResNet or VGG) will be used for inference, what object categories the downstream model recognizes (e.g. animals or scenes), or even what task will be performed on the processed image (e.g. classification or detection)”.
o	Is your goal a universal “recognition model” applicable to anything?
-	I also have some trouble with the terminology:
o	In 1. Introduction/Paragraph 4: “It is also important that the enhanced machine semantics is not specific to any concrete recognition model”: “enhanced machine semantics”!
o	In 1. Introduction/Paragraph 4: “..transferable among different recognition architectures..”. Does “architectures” refer to deep neural networks (DNN)? If yes, is recognition performed only by DNN? What about the preceding bullet (“is not specific to any concrete recognition model”)?
-	In 1. Introduction/Paragraph 3:
o	 “.. we argue that image processing systems should maintain/enhance machine semantics”. Do not see what’s to argue here?
o	“Recognition-Aware Image Processing” is it simply put Image Processing techniques for recognition enhancement (“Recognition” still needs to be defined)?
-	In 2 Related work :
o	 “ .. we assume we do not have the control on the recognition model, as it might be on the cloud or decided in the future, thus we advocate adapting the image processing model only. This also ensures the recognition model is not harmed on natural images.” Care to explain?
o	: “to achieve better recover the face identity from low-resolution images”, Typo?
-	In 1. Introduction/Paragraph 1:
o	“ .. might not look “natural” to machines”: Care to explain this concept?
	Would advise to just keep the second part of the sentence.
-	In 1. Introduction/Paragraph 2: “One could specifically train a recognition model only on these output images produced by the de-noising model to achieve better performance on such images, but the performance on natural images can be harmed.”  Care to explain?.
o	More complicated images (noisier, multiple obstructions etc.) are recognized nowadays and true to actual applications.
-	3.4 using an intermediate transformer/Last paragraph:
o	“ .. that there are two instances for each image (the output of model P and T), one is “for human” and the other is “for machines”.”:
	The “Transformer” characteristics are not clearly defined for the intended output (For machines?).
	Why is output of model T not represented in Figure 2 (Right)?


Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with a maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum likelihood, PGAs generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAEs, PGAs can generate sharper samples than vanilla VAEs. Compared to other autoencoder-based generative models using simple priors, PGAs achieve state-of-the-art FID scores on CIFAR-10 and CelebA.

The paper considers generative models and proposes to change the VAE approach in the following way. While VAEs assume a generative model (prior on the latent space + stochastic generator) and aim at learning its parameters so as to maximize the likelihood of the (training) data distribution, the authors propose to derive the learning objective from a different view. In my understanding, they consider the two composed mappings generator(encoder) and encoder(generator) and require the first one to have the data distribution as a fixpoint and the second one to have the latent prior as a fixpoint. Starting from this idea they derive an objective function for the case that the mappings are deterministic and then further enrich the objective by either likelihood based terms or VAE based terms. The new approach is analysed experimentally on MNIST, CIFAR and CelebA datasets. The authors report quantitative improvements in terms of Frechet inception distance (FID) as well as sharper samples (when compared to standard VAEs).
I find the main idea of the paper highly interesting and compelling. Nevertheless, I would not recommend to publish the paper in its present state . The technical part is in my view very hard to comprehend. This is partially due to the disadvantageous notation chosen by the authors. Furthermore, the derivation of the individual terms in the objective is hard to understand and the arguments given in the text are not convincing:
- The two terms in the objective related to the fixpoint of the encoder(generator) mapping seem to enforce a fixpoint that is a mixture of the latent Gaussian prior and the encoder image of the data distribution. It remains unclear to me, why this is a good choice.
- The derivation of the additional likelihood based terms and VAE based terms is in my view hard to understand.
It should be possible (and I believe, is possible) to derive a simpler objective ab initio, starting from the main idea of the authors.
Side note: A close visual inspection of the presented generated samples seems to confirm the known doubts about the appropriateness of the FID measure for evaluation of generative models.


We propose a novel framework, ICNN, which combines the input-conditioned filter generation module and a decoder based network to incorporate contextual information present in images into Convolutional Neural Networks (CNNs). In contrast to traditional CNNs, we do not employ the same set of learned convolution filters for all input image instances. And our proposed decoder network serves the purpose of reducing the transformation present in the input image by learning to construct a representative image of the input image class. Our proposed joint supervision of input-aware framework when combined with techniques inspired by Multi-instance learning and max-pooling, results in a transformation-invariant neural network. We investigated the performance of our proposed framework on three MNIST variations, which covers both rotation and scaling variance, and achieved 0.98% error on MNIST-rot-12k, 1.12% error on Half-rotated MNIST and 0.68% error on Scaling MNIST, which is significantly better than the state-ofthe-art results. Our proposed model also showcased consistent improvement on the CIFAR dataset. We make use of visualization to further prove the effectiveness of our input-aware convolution filters. Our proposed convolution filter generation framework can also serve as a plugin for any CNN based architecture and enhance its modeling capacity.

*Paper summary*
The authors build an input transformation invariant CNN using the TI-Pooling architecture of (Laptev et al., 2016). They make some modifications to this, namely 1) they replace the convolutional filters, with input-dependent convolutional filters, and 2) they add a decoder network from the final representation, which reconstructs an input transformation rectified image, to encourage the final representation to be fully transformation invariant.
*Paper decision*
I have decided to assign this paper a reject because of two main reasons.
*Supporting arguments*
One reason is that the base architecture is not novel. This in itself is not a key issue, but I would expect the authors to have done some in depth analysis or experimentation otherwise to compensate for this. I regret, the authors may just not have known that the ideas were already explored in the literature. The second reason is that the work is not well placed in context with prior works. This is both evident in the lack of referenced works (see below for a list) and the lack of sufficient baselines, against which they compare. For instance, if the authors had considered “Learning Steerable Filters for Rotation Equivariant CNNs” by Weiler et al. (2018) they would have known that their MNIST-rot-12k results are not state of the art as they state. In Weiler et al., the authors report 0.714 test set error on MNIST-rot-12k compared to ICNN’s 0.98.
This all said, I think the paper is well-written and very clear. The structure is straightforward and the experiments seem repeatable from the descriptions made. The stated aims of the paper are also clear: to learn input transformation invariant CNNs using input-conditioned filters.
Unfortunately a lot of supporting material and prior work has been missed. I list a lot of them here.
Works on input-conditioned filters and invariance. These are the most important
-Dynamic Steerable Frame Networks, Jacobsen et al., 2017
-Dynamic Steerable Blocks in Deep Residual Networks., Jacobsen et al., 2017
Works on input-conditioned filters:
-HyperNetworks, Ha et al., 2016
-Dynamic Filter Networks, de Brabandere et al., 2016
Works on invariance:
-Invariance and neural nets, Barnard and Casasent, 1991
-Group Equivariant Convolutional Networks Cohen and Welling (2015)
-Harmonic Networks: Deep Translation and Rotation Equivariance: Worrall et al. (2017)
-Steerable CNNs, Cohen and Welling (2017)
-Spherical CNNs, Cohen et al. (2018)
-CubeNet: Equivariance to 3D Rotation and Translation, Worrall and Brostow (2018)
-Learning steerable filters for rotation equivariant CNNs, Weiler et al. (2018)
-Gauge Equivariant Convolutional Networks and the Icosahedral CNN, Cohen et al. (2019)
*Questions/notes for the authors*
- Please address the missing references
- Are the input-conditioned filters conditional on position in the activations, or are they shared across all spatial locations of the image? This is not clear from the text.
- The image reconstruction reminds me of Transforming Auto-encoders (Hinton et al., 2016) and Interpretable Transformations with Encoder-Decoder Networks (Worrall et al., 2017). How is your setup different?


Mixture models, a basic building block in countless statistical models, involve latent random variables over discrete spaces, and existing posterior inference methods can be inaccurate and/or very slow. In this work we introduce a novel deep learning architecture for efficient amortized Bayesian inference over mixture models. While previous approaches to amortized clustering assumed a fixed or maximum number of mixture components and only amortized over the continuous parameters of each mixture component, our method amortizes over the local discrete labels of all the data points, and performs inference over an unbounded number of mixture components. The latter property makes our method natural for the challenging case of nonparametric Bayesian models, where the number of mixture components grows with the dataset. Our approach exploits the exchangeability of the generative models and is based on mapping distributed, permutation-invariant representations of discrete arrangements into varying-size multinomial conditional probabilities. The resulting algorithm parallelizes easily, yields iid samples from the approximate posteriors along with a normalized probability estimate of each sample (a quantity generally unavailable using Markov Chain Monte Carlo) and can easily be applied to both conjugate and non-conjugate models, as training only requires samples from the generative model. We also present an extension of the method to models of random communities (such as infinite relational or stochastic block models). As a scientific application, we present a novel approach to neural spike sorting for high-density multielectrode arrays.

Summary:
This paper introduces a novel deep learning architecture for efficient amortized Bayesian inference over mixture models. Unlike previous approaches to amortized clustering, the proposed method allows us to treat local discrete labels of data points and infer the unbounded number of mixture components, making it more flexible as in the case of Bayesian nonparametrics. It is shown that the resulting algorithm can be parallelized and applied to both conjugate and non-conjugate models. The authors also suggest an extension to models of random communities and a novel approach to neural spike sorting for high-density multielectrode arrays based on the proposed method.
Strengths:
The paper is generally well written and the relationship to previous works is well described. Empirical results seem quite convincing, for example, the clustering results presented in Fig. 2 and Fig. 3 clearly show not only the inferred number of clusters, but also the posterior probability which indicates that reasonable samples are assigned higher probability.
Weaknesses:
- Overall, the idea looks very original and promising, but I find some technical details are not easy to understand under the current form, especially for non-experts in this domain. I would recommend the authors to elaborate a bit more on the proposed architecture and the variable-input soft-max function in Sect. 2.1.
- On page 8, the authors mention that the NCP is much more efficient compared with MCMC, for example, in the Gaussian 2D example. However, regarding the DPMM clustering model, it is known that MCMC methods are generally slower compared with variational inference, which is computationally faster. I think it would be interesting to add a discussion or comparison with variational inference in terms of computational efficiency.
- If I understand correctly, the NCP is essentially based on a sequential sampling procedure. The authors claim that the proposed method is easily parallelized using a GPU, but there does not seem to be sufficient details on the GPU-parallelization of sequential sampling.
Minor comments:
The size of some figures appears too small, for example Fig. 6 and Fig. 10, which may hinder readability.


This paper studies aligning knowledge graphs from different sources or languages. Most existing methods train supervised methods for the alignment, which usually require a large number of aligned knowledge triplets. However, such a large number of aligned knowledge triplets may not be available or are expensive to obtain in many domains. Therefore, in this paper we propose to study aligning knowledge graphs in fully-unsupervised or weakly-supervised fashion, i.e., without or with only a few aligned triplets. We propose an unsupervised framework to align the entity and relation embddings of different knowledge graphs with an adversarial learning framework. Moreover, a regularization term which maximizes the mutual information between the embeddings of different knowledge graphs is used to mitigate the problem of mode collapse when learning the alignment functions. Such a framework can be further seamlessly integrated with existing supervised methods by utilizing a limited number of aligned triples as guidance. Experimental results on multiple datasets prove the effectiveness of our proposed approach in both the unsupervised and the weakly-supervised settings.

The authors propose an unsupervised knowledge graph alignment framework, KAGAN, using adversarial training. Specifically, they utilize a triple discriminator to discriminate the aligned source triples and target triples, and a reward function to minimize the divergence between source triples and target triples. Also, they propose to leverage the lower bound of the mean KL divergence (mutual information) to resolve the mode collapse problem. The proposed method can be incorporated with a supervised method to be a weakly-supervised approach. Even though there are a family of unsupervised approaches for domain alignment, this paper is the first to solve the knowledge graph alignment problem in an unsupervised/weakly supervised way.
Strength:
1.	The paper addresses a critical knowledge graph alignment problem using GAN based on triplets, not usually entity alignment in literature, but also considers the relation alignment in knowledge graph.
2.	The paper tries to solve the problem in an unsupervised way, and shows the on-par performance with weakly supervised methods in FB15k dataset.
3.	The paper considers mode collapse problem and tries to solve the problem via mutual information rather than mean KL divergence, also gives the theoretical proof.
4.	Detailed experimental analysis and ablation studies show the effectiveness of the proposed method on small datasets.
5.	The paper is well-written and easy to follow.
I have two concerns as follows:
1.	The authors conduct experiments on multiple small KG datasets such as FB15K and WK15K. But the reviewer finds that the baseline papers authors mentioned also have experiments on larger datasets like WK120K (Chen et al., 2017a), WK60k, DBP-WD or DBP YG (Sun et al., 2018a). It is essential to conduct the experiment on larger datasets to verify the effectiveness of the proposed method.
2.	Is it necessary to construct a reward function to update the alignment function using REINFORCE algorithm? For instance, current distribution matching methods can define the discrepancy between the two distributions (such as target distribution and aligned source distribution). It can directly optimize the loss in an end-to-end differentiable way instead of a reinforcement learning way. It can avoid the sampling and provide a more stable optimization process.


Motivated by practical applications, we consider unsupervised domain adaptation for classification problems, in the presence of missing data in the target domain. More precisely, we focus on the case where there is a domain shift between source and target domains, while some components in the target data are systematically absent. We propose a way to impute non-stochastic missing data for a classification task by leveraging supervision from a complete source domain through domain adaptation. We introduce a single model performing joint domain adaptation, imputation and classification which is shown to perform well under various representative divergence families (H-divergence, Optimal Transport). We perform experiments on two families of datasets: a classical digit classification benchmark commonly used in domain adaptation papers and real world digital advertising datasets, on which we evaluate our model's classification performance in an unsupervised setting. We analyze its behavior showing the benefit of explicitly imputing non-stochastic missing data jointly with domain adaptation.

*Summary.* The paper presents and addresses the problem of performing domain adaptation when the target domain is systematically (i.e., not the result of a stochastic process) missing subsets of the data. The issue is motivated by applications where one modality of data becomes unavailable in the target domain (e.g., when deciding which ads to serve to new users, the predictor may have access to behavior across other websites but not on a specific merchant's website). The proposed method learns to map source and target data to a latent space where the representations for the source and target are aligned, the missing components of the target can be inferred, and classification can be performed successfully. These are achieved by adversarial/optimal transport loss on source and target features, a mean-squared error and adversarial loss on latent generation/imputation, and a cross entropy loss on source label prediction, respectively. Experiments are performed on digits and click-through rate (CTR) prediction and include a thorough set of baselines/oracles for comparison.
*Review.* While the problem statement is novel, I am unconvinced that the advertising experiment includes both a domain adaptation and imputation problem. I describe this in detail below. For this reason, I am giving the paper a weak reject.
*Questions that impacted rating.*
1. Ads experiment: From my understanding, the source domain is the traffic of users who have interacted with (clicked through to?) a specific partner and the target domain is the traffic of the users who have not interacted with that specific partner. The data that needs to be imputed is the click through rate for target users with that specific partner. In this case, it is not obvious to me why there is a domain shift between these two groups of users. This would imply that the traffic of source users and target users is different for other partners. I don't see why this would need to be true. Could the authors provide an explanation as to why this is the case (e.g., by showing that CTRs differ with other (partner, publisher) pairs between source and target). From my understanding, Table 5 only shows CTR averaged across all users in each domain, but does not show that the CTRs differ between source and target users for contexts/(partner, publisher) pairs (i.e., the results in table 5 could be due to the fact that the prior distribution over context is different for source and target users).
*Additional notes. Immaterial to rating.*
1. I personally felt that the motivation for UDA vs imputation in the first paragraph was a bit muddled. I think sticking to one example would make the motivation more clear to the reader. E.g., explain the prediction problem for medical imaging (which I assume is disease diagnosis, but it is not stated explicitly), describe how some medical imaging may be missing for certain patients (imputation), then explain that there may be noise across different medical imaging systems (UDA), then list the other applications where this arises with citations (e.g., These phenomena have also been documented in advertising applications [1], ...).
2. I was surprised by the difference between Adaptation-Partial and the other two train/test conditions in Figure 2 when p=30%. Out of curiosity, do the authors have an explanation for this discrepancy? I would have predicted that, if most of the information necessary for prediction was available in the remaining 70% of the image that the performance of these cases would be very similar.  I think it would be helpful to see the accuracy on the source domain and the labeled target domain to better understand that result.


Recently, the convolutional neural networks (CNNs) have shown great success on semantic segmentation task. However, for practical applications such as autonomous driving, the popular supervised learning method faces two challenges: the demand of low computational complexity and the need of huge training dataset accompanied by ground truth. Our focus in this paper is semi-supervised learning for semantic segmentation. We wish to use both labeled and unlabeled data in the training process. A highly efficient semantic segmentation network is our platform, which achieves high segmentation accuracy with a low model size and high inference speed. We propose a semi-supervised learning scheme to improve segmentation accuracy by including extra images without labels. While most existing semi-supervised learning methods are designed based on the adversarial learning techniques, we present a new and different approach, which trains an auxiliary CNN network that validates labels (ground-truth) on the unlabeled images. In the supervised training phase, both the segmentation network and the auxiliary network are trained using the labeled images. Then, in the unsupervised training phase, the unlabeled images are segmented and a subset of image pixels are picked up by the auxiliary network; and then they are used as ground truth to train the segmentation network. Thus, at the end, all dataset images can be used for retraining the segmentation network to improve the segmentation results. We use Cityscapes and CamVid datasets to verify the effectiveness of our semi-supervised scheme. Our experimental results show that it can improve the mean IoU for about 1.2% to 2.9% on the challenging Cityscapes dataset.

- This paper proposes a semi-supervised learning strategy for semantic segmentation of road scenes. Specifically, authors propose to include an auxiliary network that will predict the confidence (at pixel-level) of the predictions on unlabeled images. These confidence values will be used to  generate a new auxiliary ground-truth to retrain the network using the unlabeled images.
- Even though the idea is somehow interesting and results seem to improve with respect to the baselines, this paper is very similar to standard semi-supervised learning approaches for natural images that employ image proposals (i.e., EM-based methods). In those works, unlabeled images are segmented with the network trained on labeled images, generating some proposals. These proposals are later employed as a fake ground truth to re-train the network employing both labeled and unlabeled images. The only difference in this work is to employ the virtual confidence map to mask-out some pixels (those with lowest confidence values).
- Related work section is extremely weak. Authors merely mention few papers (some other relevant papers are missing), and throw a sentence for each one, without making connections between works. This makes difficult to place their work among the literature (e.g., which limitations of previous approaches the current method intend to address?). Authors should significantly improve this section.
- I am not sure about the fact that employing only the probability maps as input to generate a confidence map is reliable, if no other information is employed. These predictions (e.g., confidence map) will be based only on the probabilities obtained by the first network. While this is already a good indicator of the confidence of the network to make those predictions, I believe that input images should also be included. The intuition behind this is that there may exist some regions with similar probabilities (from first network), which are incorrectly classified (leading to 0-masked pixels on the auxiliary ground-truth) in some cases, while correctly classified in other situations (leading to 1-masked pixels on the auxiliary ground-truth).
- Eq.1) is basically the standard cross-entropy loss, with the difference of the weighting terms.
- a_{h,w} is the softmax of the auxiliary network, isn’t it?
- Further, authors threshold the values of the confidence map to generate the new auxiliary ground truth. Why not to use the raw values so that each pixel is weighted differently according to its importance?
- Authors make some claims which were never demonstrated. For example, they mention that the proposed approach performs better on small targets than previous approaches. Nevertheless, only mean results (over all the classes) are shown. To this end authors should report per-class performances, instead of the mean.
- Furthermore, authors make several over claims, misleading information. For example, they mentioned that they proposed a highly efficient segmentation method. Nevertheless, from Table 2 it can be observed that the proposed method ranks in the middle in terms of both speed and parameters, compared to other state-of-the-art models. Similarly, authors mention that their model is equipped with a carefully designed auxiliary loss function during training, while they basically employ a standard cross-entropy weighted by some values to account for imbalance between classes and between positive and negative pixels within the same class.
- The paper contains many grammatical errors.


We introduce a general approach, called test-time training, for improving the performance of predictive models when training and test data come from different distributions. Test-time training turns a single unlabeled test instance into a selfsupervised learning problem, on which we update the model parameters before making a prediction on this instance. We show that this simple idea leads to surprising improvements on diverse image classification benchmarks aimed at evaluating robustness to distribution shifts. Theoretical investigations on a convex model reveal helpful intuitions for when we can expect our approach to help.

The motivation is to increase accuracy of CNNs with unseen (unknown) distribution shifts. To this end, instead of the recent approach of training-time self-supervision, they adopted it for test-time [limited novelty]. More precisely, for a classification task, they considered two headed neural network with one head for main classification task and another for an auxiliary task (e.g. predicting rotation degree of rotated images). The feature extractor (up to k layers) is shared between these tasks thus updated using the two tasks, while two heads are updated according to each task. In test-time, the samples (drawn from shifted data distributions) are being used for updating the shared feature extractor through the auxiliary task. They have investigated their test-time approach in a series of experiments in online and offline settings on synthetic shift in data distribution of image classification tasks. The distributional shifts are synthesized adding some non-adversarial perturbation to clean images. The authors proof that their test-time approach can lead to lower error rate on given test-time samples when the underlying learning model is a linear regression model.
The authors incorrectly interchange OOD with domain shifts in their manuscript. The common usage of OOD set is to capture novel samples that are not properly following the training set distribution, e.g. from different concepts than the set of classes given in the training set. With an OOD set, a robust model dealing with it should be able to detect whether an instance is in-distribution or OOD, making a special decision for the latter case (e.g., rejecting the instance).
In domain shift, we look at the scenario where test objects are the same as used for training the model (i.e., correspond to one of the classes the model is processing), but might be perturbed or coming having a different distribution (e.g. SVHN for MNIST or vice versa). The approaches for domain shift concern to improve robustness of CNNs to such shifts in data distribution. Accordingly, the title of the paper inaccurately reflects of the claim of the paper and is misleading, this paper is not on learning with out-of-distribution instances.
The other important point is about catastrophic forgetting phenomena in online setting of their approach, which was not addressed thoroughly in the paper. How not to forget what the model has previously learnt a test-time training? I see this somewhat has been empirically shown in Fig 2 with accuracy on the original data, but what is the mechanism not to forget what have being learned so far?
Besides generalization enhancement, the advantage of test-time self-supervision over training-time joint self-supervised is not clear for the readers, particularly considering the problem of the pitfall of catastrophic forgetting phenomena in test-time training. This pitfall does not exist for training-time joint self-supervised approach. What are the advantages of this approach?
The claims (about synthetic shift in distribution shift) are well supported in a series of experiments, where the distribution shifts are synthesized using adding some non-adversarial perturbation to clean images. However, the other common experiments on real shifts in distribution (e.g. SVHN for MNIST and vice versa, and those performed in Volpi et al  2018) are missing, which can help the paper to being better supported and justified for its practicality.
[Volpi et al  2018]: Generalizing to Unseen Domains via Adversarial Data Augmentation, NIPS 2018
I found the paper rather difficult to follow and not very coherent in its organization. The main idea is fundamentally simple, but it is still difficult to get it from the text. It needed me 2-3 readings before really getting the point of the paper.
** Update ** I read other reviews and comments. The answer of the authors to my comments are somehow satisfactory, especially the point of changing from "out-of-distribution" to "domain shift", which avoid some confusion. I upgraded my rating to a "weak accept".


Energy-Based Models (EBMs) assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score matching (Vincent, 2011) has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising (Saremi and Hyvarinen, 2019). However, demonstrations of such models in high quality sample synthesis of high dimensional data were lacking. Recently, Song and Ermon (2019) have shown that a generative model trained by denoising score matching accomplishes excellent sample synthesis, when trained with data samples corrupted with multiple levels of noise. Here we provide analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose a novel EBM trained with multi-scale denoising score matching. Our model exhibits data generation performance comparable to state-of-the-art techniques such as (Song and Ermon, 2019) and GANs, and sets a new baseline for EBMs. The proposed model also provides density information and performs well in an image inpainting task.

This paper presents a method of learning of energy based models using denoising score matching. This technique has been used before but only with limited success. The authors hypothesize that this is due to the fact that the matching was only performed over a single noise scale. The main idea of this work is to employ a range of scales to learn a single energy function. This trick helps to alleviate the problem of noisy samples concentrating in a low-volume region of the ambient space.
It seems that the paper draws significant inspiration from the work by Song & Ermon, 19. The difference between the two appears to be minor:
1) The density is represented as a Boltzman distribution and therefore the score function is reduced to the gradient of the energy function (this has been done before)
2) Instead of conditioning the energy on the noise level the authors propose to use explicit scaling by the inverse temperature

Pros:
+ The paper is mostly well-written.
+ I think Section 2 does a good job at illustrating challenges in training energy-based models using denoising score matching with a single noise scale.
+ As compared to (Song & Ermon, 19) using the Boltzman distribution ensures that the learned score is an actual conservative vector field. Arguably, learning an image to scalar network is easier than learning an image to image one.
+ Samples from the model are of competitive visual quality.
Cons:
- Scaling energy by the inverse temperature seems to be one of the most important aspects of the paper but is only justified by “intuition from physics”. I’m not entirely sure that this is a valid assumption. In contrast, (Song & Ermon, 19) don’t put any hard constraints on the values of the score for different noise levels besides that they are produced by a single conditional network. I would appreciate if the authors discussed that difference in more detail.
- The authors don’t provide any analysis as to whether the annealed Langevin MC procedure leads to the samples from the right distribution.
- The quantitative results don’t seem to be better (actually, they are worse) than those from (Song & Ermon, 19).
Notes/questions:
* Abstract: “unmormalized” -> “unnormalized”
* Section 2.1, (1): \tilde{x} -> \tilde{\mathbf{x}}
* Section 2.2, paragraph 2: What does superscript C mean in the noisy manifold? Never defined.
* Section 2.2, paragraph 4: “some example” -> “some examples” (?)
* Section 3, paragraph 1: “CIFAT-10” -> “CIFAR-10”
* Section 4, paragraph 2: “for each T as a separate model”. I don’t think this is a correct statement. (Song & Ermon, 19) use a single conditional model for all the noise levels.
* Section 4, paragraph 2: “does not rely on explicit receive noise magnitude” -> “does not rely on receiving noise magnitude explicitly” (?) I also don’t quite understand this entire sentence. Does the model really infer the noise magnitude from a given image? It seems like in Equation (7) there is an assumption that the temperature T is equal to 1. I don’t feel like there is a lot of difference between the proposed model and (Song & Ermon, 19) when it comes to supplying noise information. I’d appreciate if the authors could clarify that bit for me.
My main concern about this paper is that it doesn’t seem like a big step from its starting point (Song & Ermon, 19). The modifications are shown to work empirically but don’t result in a significantly better model. Moreover, I feel like the paper could do a better job at justifying those changes. I’m giving a borderline score but willing to increase it if the authors address my questions.