A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019)  on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al., 2016) which is particularly suitable for this type of problem: the General Incompressibleflow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

This paper extends recent work by Khemakhem et al on nonlinear ICA to allow for unknown number of generative factors. This is tackling an important problem in the field of generative modeling, where one would like to extract the generative factors of a dataset that independently control its features (i.e. disentanglement).
The paper is very clearly written, does a great job at motivating the problem and presenting the recent results from Khemakhem et al, before extending them with some simple theorems and demonstrating their application on toy data + EMNIST.
I think that this research direction is extremely promising, and obtaining a theoretical understanding of when/why disentangling could work would be particularly valuable to the field, and I would lean towards acceptance so that this work gets more attention.
It is slightly surprising however that their empirical results seem to indicate that these theoretical conditions do not seem to be necessary, which should be investigated further (and might help illustrate the dichotomy in some claims and results obtained in the disentangling literature recently).
1.	It was unclear to me how one should/would decide what to use for $u$ or what to leave out to be factorized by the method.
	This may be out of scope for this current work, but one way to answer that would be to leverage datasets with more fully labelled factorised data (e.g. dSprites [1]) and present how one should leverage these with $u$, in order to identify the original generative factors.
2.	Related, using EMNIST was interesting, but given the lack of “accepted” generative factors to be recovered, it is hard to tell if the 22 variables found are “correct” or more similar to using a generative model which would entangled the data more (and hence would falsely introduce extra latent variables).
3.	The toy dataset, with its random RealNVP network to produce the data, was less “mixed” than I expected, looking at Figure 2B. In its projected view, the clusters are still rather easy to identify by eye, which surprised me somehow? Could you comment a bit more on how difficult is the task, or present a VAE baseline that would fail to explain that data?
4.	I did not see how the threshold for selecting 22 latent variables on EMNIST was set? Was the 23rd latent variable significantly less informative? The spectrum on Figure 3A was not precise enough to assess this fairly and the single mention of “measured by the standard deviations of test data transformed to the latent space” was too vague to reverse-engineer the decision.
5.	It was interesting to read about the observations of when this method should fail. It would be interesting if a dataset with explicit “gaps” would be constructed to analyze this case.
6.	Figure 4 might benefit from mentioning “what” each variable controls for directly in their individual caption / on top of them, instead of having to read this through in the full figure caption.
7.	The current model in the end is modeling the latent state using a mixture of gaussians (although these now have a theoretical connection to the true generative factors). How much does this differ to existing generative models using VAEs with a mixture of gaussian prior [2, 3]?
References:
[1] dSprites dataset: https://github.com/deepmind/dsprites-dataset
[2] https://arxiv.org/abs/1611.02648
[3] https://arxiv.org/abs/1902.03717


Lifelong learning aims at avoiding the catastrophic forgetting problem of traditional supervised learning models. Episodic memory based lifelong learning methods such as A-GEM (Chaudhry et al., 2018b) are shown to achieve the stateof-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks. While the model is trained on a new task, a reference gradient is computed on the episodic memory to guide the direction of the current update. While A-GEM has strong continual learning ability, it is not clear that if it can retain the performance in the presence of adversarial attacks. In this paper, we examine the robustness of A-GEM against adversarial attacks to the examples in the episodic memory. We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects. We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion (GREV) which is shown to be more effective. Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.

This paper does a good job of raising awareness of adversarial attacks in lifelong learning research with deep neural networks. This is the first time I have considered this problem, but not sure whether any prior work exists in the specific subfield.
At the conceptual level, many issues can arise when a lifelong learner is attacked, since systematic negative bias could be introduced in the training process and may be very difficult to remove, given the tendency to 'remember everything' which dominates current approaches.
The paper isolates one lifelong learning approach (A-GEM) which is characteristic of one (of many) different approaches to lifelong learning, and investigates its robustness to standard adversarial attacks and a novel attack developed within this paper, which is stronger, but specific to episodic memory approaches.
I cannot recommend acceptance at this point for the following reasons:
1) I am not sure what I can generalize away from this paper to the immediate subfield and beyond. The paper claims that the investigated method is SOTA, but it's not clear this is the case, even in restricted class of similar episodic memory based models, see [1] for an independent evaluation of many such approaches. Is there any reasons why conclusions about this particular method are indeed representative of its class?
2) While the paper does not explicitly make this claim, the title suggests that 'gradient reversion' attacks apply to lifelong learning models in general. Why is this class of approaches particularly informative such that conclusions may hold in general? Are other methods in this class more susceptible to these attacks and can the proposed attack be applied to the whole class, or even other types of approaches? This should be clarified!
References
[1] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, Tinne Tuytelaars,  Continual learning: A comparative study on how to defy forgetting in classification tasks, https://arxiv.org/abs/1909.08383


We propose a new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN). It is known that GAN can produce very realistic samples while VAE does not suffer from mode collapsing problem. Our model optimizes λ-Jeffreys divergence between the model distribution and the true data distribution. We show that it takes the best properties of VAE and GAN objectives. It consists of two parts. One of these parts can be optimized by using the standard adversarial training, and the second one is the very objective of the VAE model. However, the straightforward way of substituting the VAE loss does not work well if we use an explicit likelihood such as Gaussian or Laplace which have limited flexibility in high dimensions and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model with an implicit likelihood by an adversarially trained discriminator. In an extensive set of experiments on CIFAR-10 and TinyImagent datasets, we show that our model achieves the state-of-the-art trade-off between generation and reconstruction quality and demonstrate how we can balance between mode-seeking and mass-covering behaviour of our model by adjusting the weight λ in our objective.

The paper proposes to replace the KL-divergence in VAE training with the lambda-jeffreys divergence of which the symmetric KL-divergence is a special case. The paper proposes a pure implicit likelihood approach that uses three discriminator models to estimate the KL-divergences. Experiments are conducted on CIFAR-10 and TinyImageNet and several scores are reported to show that the proposed method performs as good if not better than current approaches.
---------
I think the paper tries to achieve too much in too little space and foregoes scientific exactness for the sake of claiming SOTA. Since there is a difference between claiming SOTA on a task and validating a new method, the small amount of space makes it difficult to substantiate both claims at the same time. In the rest of the review i will try to substantiate the claim:
1. The paper claims on page 2: "These models do not have a sound theoretical justification about what distance [...] they optimize". While the paper tries to substantiate its claims by showing theoretically that it does the right thing using the optimal discriminator, it leaves the question open what happens with any other discriminator. The theory does not justify non-optimal solutions. It is argued on page 6 that non-optimality of the discriminator serves as some form of regularization, but  this requires some justification.
Moreover, the paper uses LPIPS to measure reconstruction quality - but this measure is a deep neural network. So if those measures are good enough to compare solutions with and the theoretical justification of the proposed method is shaky in practice - why not use LPIPS for training?
2. The paper proposes the discriminator in order to allow for an implicit likelihood. However, the r-function used in the experiments does not fulfill the property of a well defined likelihood, and Theorem 1 does not hold, since technically the KL-divergence is infinity. If we ignore this by adding a small amount of Gaussian noise around the sampled cyclical shifts - like the r' used in the experiments, we can easily write down the explicit likelihood function since:
r(y|x)=\sum_i w_i N(y|Shift_i(x), \sigma)
where Shift_i is the i-th shift in the set described in the paper and w_i its probability  p(y|q). So the explicit solution of theorem 1 can be written down and another ablation study would be training the method with the explicit formulation for this KL-term(i.e. only training two discriminator models). If the results are not equivalent, this implies that the discriminator does not reach the optimum. The implications of that should be discussed regarding 1.
3.  Existing ablation studies are a bit of a straw-man: the paper compares changing r(y|x) by standard Gaussian or Laplace. However, we know that a large variance does not make any sense and almost all papers use tiny variances (e.g. in beta-VAE the beta-values tend to be very small, which is equivalent to small variances here).
---------------------------
Smaller things
- Are the experimental results all with the same architecture for encoder/generator for all results you compared to? if not, the effect of that should also be tested.
- my personal biased view on the generated images is: it looks worse than alpha-GAN. Every reconstructed image has a grey tone and the generated images also offer a strong grey palette. The details don't look better as well.
- typo inroduce->introduce


Semi-supervised learning approaches based on generative models have now encountered 3 challenges: (1) The two-stage training strategy is not robust. (2) Good semi-supervised learning results and good generative performance can not be obtained at the same time. (3) Even at the expense of sacrificing generative performance, the semi-supervised classification results are still not satisfactory. To address these problems, we propose One-stage Semi-suPervised Optimal Transport VAE (OSPOT-VAE), a one-stage deep generative model that theoretically unifies the generation and classification loss in one ELBO framework and achieves a tighter ELBO by applying the optimal transport scheme to the distribution of latent variables. We show that with tighter ELBO, our OSPOT-VAE surpasses the best semi-supervised generative models by a large margin across many benchmark datasets. For example, we reduce the error rate from 14.41% to 6.11% on Cifar-10 with 4k labels and achieve state-of-the-art performance with 25.30% on Cifar-100 with 10k labels. We also demonstrate that good generative models and semi-supervised results can be achieved simultaneously by OSPOT-VAE.

Contributions:
The main contribution of this paper lies in the proposed OSPOT-VAE for semi-supervised classification. The proposed model is (i) an one-stage framework that unifies the generation and classification loss in one ELBO, (ii) achieves a tighter ELBO by applying optimal transport to the distribution of the latent variables, and (iii) achieves SOTA semi-supervised learning results without sacrificing generative performance on many benchmarks.
Strengths:
(1) Novelty: This paper contains novelty in terms of two aspects. First, the derivation of Eqn. (13) unifies both generation loss and classification loss naturally. The derivation seems different from a standard semi-supervised VAE. Second, the use of optimal transport (OT) estimation to tighten the ELBO results in improved classification performance. The use of OT seems novel in this context.
(2) Writing: The paper is generally well written. However, I also found section 3 is hard to follow, with details in the Weaknesses section.
(3) Experiments: The results look impressive. OSPOT-VAE achieves better performance than other generative approaches.
Weaknesses:
(1) Clarity: My biggest concern lies in the clarity of section 3, which I think is confusing in its current shape. After several rounds of reading, below is my understanding. In section 3.1, the authors first derive the objective in Eqn. (13). This objective is derived based on the use of mutual information (Eqn. (8)). By the end, no special cross-entropy loss term like in Eqn. (7) is needed.  Second, in order to tighten the bound, section 3.2 introduces optimal transport estimation. Combining these two losses together (see in Algorithm 2) results in the final algorithm. However, generally, I feel the presentation of section 3 is confusing. One may need to present the big picture (framework) first, and then dig into details.
Besides that, I have some questions as below.
a) There are many hyper-parameters in the final objective, for example, the mutual information I_z, I_c is also considered as hyper-parameters. How all these hyper-parameters are selected?
b) Is there any intuition on why Eqn. (13) is already enough without introducing the empirical cross-entropy loss like in Eqn. (7)? It seems to me adding this additional cross-entropy loss will not hurt performance.
c) In Eqn. (12), how is the last term D_{KL}(p(c|X)||q_{\phi}(c|X)) calculated, since we cannot get the true p(c|X)?
d) As shown in Table 4, the use of optimal transport seems to be the key. However, section 3.2 is hard to follow. For example, In Algorithm 1, Line 6 & 7 considers the true posterior of z and c is a Gaussian and a multinomial distribution, respectively. Why doing this? Supported by theory? I also feel the calculation of L_{Rz} and L_{Rc} in Line 8 & 9 is hard to follow.
e) How reliable is this optimal transport estimation is? And how it contributes to the final classification performance? Since this loss term is not a classification loss, then is there any intuition on why it serves as a regularization to help the final performance so much?
f) Results in Table 5 on Cifar10 seem not imply OSPOT-VAE tightens the ELBO. Adding results on MNIST will be better, since the ELBO on MNIST is widely benchmarked.
(2) Minor:
a) I am not sure why section 2.1 is needed. Also, the feature matching GAN paragraph in section 2.2 seems also unnecessary. If you really want to discuss the use of GAN for semi-supervised learning, then there is a lot of other work that needs discussion.
b) There is a missing right bracket in the second term of Eqn. (9).
Overall, I think the results look quite impressive. However, the presentation of the method part is unclear to me, and could be much improved.


The advance of node pooling operations in Graph Neural Networks (GNNs) has lagged behind the feverish design of new message-passing techniques, and pooling remains an important and challenging endeavor for the design of deep architectures. In this paper, we propose a pooling operation for GNNs that leverages a differentiable unsupervised loss based on the minCUT optimization objective. For each node, our method learns a soft cluster assignment vector that depends on the node features, the target inference task (e.g., a graph classification loss), and, thanks to the minCUT objective, also on the connectivity structure of the graph. Graph pooling is obtained by applying the matrix of assignment vectors to the adjacency matrix and the node features. The proposed method can also be used as a stand-alone module to cluster vertexes in annotated graphs and solve unsupervised problems. We validate the effectiveness of the proposed pooling method on downstream tasks, including supervised graph classification and a set of unsupervised tasks, which reveal the limitations of existing GNN pooling approaches.

This paper proposes a graph pooling method by utilizing the Mincut regularization loss. It is an interesting idea and performs well in a number of tasks. However, due to the limitation of novelty and poor organizations, this paper cannot meet the standard of ICLR. The detailed reasons why I give a weak reject are listed as follows:
1. Even though the proposed minCUT pool is interesting, the contribution is not enough to get published in the ICLR. If I understand correctly, the only difference is the unsupervised loss, compared with the previous work, Diffpool [1].

2. The paper needs to be reorganized to demonstrate its contribution. The proposed method section only has around 1.5 pages, making it difficult to understand the proposed method clearly. Therefore, more details and analyses about the proposed method should be included to support and clarify the idea.

3. The paper needs to be improved for its theoretical derivations and proof. For example, it is not clear why Equation (6) is correct, which is the main contribution of this paper. The authors provide intuitive thoughts but there are not theoretical derivations and proof. The term $L_c$ comes from Equation (2) but why is it correct to only compute the trace?

4. Some experiments cannot support the claim very well.  For example, the graph clustering experiments are not convincing. The goal of graph pooling is to learn high-level graph embeddings but not perform graph clustering. It is not proper to evaluate the graph pooling method using graph clustering tasks. Or, the author should clarify the motivation to do this experiment. If the model is trained for graph classification or node classification, then why should the node clusters lead to high NMI or CS scores?
[1]. Ying et al., Hierarchical Graph Representation Learning with Differentiable Pooling, NIPS 2018
==========Update===========
I have read authors response and other reviews. While the authors address some of my concerns, I still believe the contribution/novelty is limited. I am sticking to my score.


We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the RaoBlackwellization of three different estimators. Combining our estimator with RE-INFORCE, we obtain a policy gradient estimator and we reduce its variance using a built-in control variate which is obtained without additional model evaluations. The resulting estimator is closely related to other gradient estimators. Experiments with a toy problem, a categorical Variational Auto-Encoder and a structured prediction problem show that our estimator is the only estimator that is consistently among the best estimators in both high and low entropy settings.

Edited after rebuttal:
I'm satisfied with the clarifications provided by the authors about where they see this work being applicable. I think they have correctly identified a setting that is of interest to a broad audience and where their approach is more attractive than the alternatives. I now recommend accepting this paper.
The authors develop a generic method for estimating expectations under discrete distributions based on sampling without replacement and apply it to the task of estimating gradients for training models with discrete latent variables. The proposed estimator is unbiased and the authors prove it has lower variance than the naive Monte Carlo estimator.
I weakly recommend rejecting the paper. While the derivations and theorems presented in the paper are correct, the experiments are sensible and support the claims made, my impression is that the range of applications where the proposed method is a better alternative to the existing algorithms is extremely narrow. I would be willing to raise my rating to Weak Accept if the authors convince me that their method is useful more broadly than I think it is.
The method derivation is generally well-written and easy to follow. However, the account of the sum-and-sample estimator is somewhat misleading. As is, it suggests that it is natural to sum over k-1 elements and only draw one sample, while in fact choosing how many elements to sum over (I'll call this number S in my review) and how many samples to draw from the remaining set is the crucial step needed to make this algorithm work well. This is discussed by Fearnhead and Clifford [1] in the context of resampling in particle filters and I believe is also addressed by Liu et al. (2019). I understand that the authors are guiding the presentation towards a version of sum-and-sample they can compare with in theory and in experiments, but in the process they're doing a disservice to the reader. I think the authors should state the importance of choosing S well early on and then either compare with methods that attempt to set S optimally throughout experiments, or clearly explain that they restrict their discussion to a particular setting where S is difficult to optimize and explain why.
There are also additional baselines that should be considered. This includes methods such as stratified or systematic sampling, which are often discussed within the context of resampling [2]. Another strong but somewhat obscure baseline is given by Duffield et al. [3]. While it is possible that none of those methods are applicable in the experimental setting chosen in the paper, the authors should be very clear about how broadly they claim superiority of their method.
On the topic of experimental evaluation, the authors seem to have focused on the setting where sampling is done using stochastic beam search, presumably because in such a setting some of the strong baselines mentioned above, including sum-and-sample with the optimal choice of S, are not applicable. If that's true, I would like to see a detailed discussion about what it is about the problem setting that makes alternative baselines inapplicable. My current impression is that the proposed method is only worth considering when used with a stochastic beam search and I don't think such settings are particularly common. This is my main complaint about the paper.
On the subject of applicability, the experiments only test the proposed method in very low k settings, where the exponential algorithm for computing the leave-one-out ratio can be applied. While the appendix mentions that there is a faster alternative based on numerical integration, the claim that it can be efficiently integrated is not substantiated anywhere.
The plots could use some improvement. It took me a lot of squinting at Figures 2 and 3 to figure out what's going on, since the lines overlap too much - I couldn't even find the Unordered line in Figure 2 at first. Also I'm confused about why lines for sum-and-sample in Figure 1 extend beyond 8 evaluations, and VIMCO does not. Also figure captions could be a little more self-contained.
Finally, I would recommend stating Theorem 1 in more concrete terms, such as "estimator A has lower variance than estimator B".
[1] P. Fearnhead and P. Clifford. On-line inference for hidden Markov models via particle filters. Journal of the Royal Statistical Society, 65:887–899, 2003.
[2] R. Douc, O. Cappé, and E. Moulines. Comparison of resampling schemes for particle filtering. In ISPA, 2005.
[3] N. Duffield, C. Lund, and M. Thorup. Priority sampling for estimation of arbitrary subset sums. Journal of the ACM, 54, 2007.


Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r = +1 for matching the demonstrated action in a demonstrated state, and a constant reward of r = 0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive results compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This paper is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards.

Summary
-------------
The authors propose SQUIL, an off-policy imitation learning (IL) algorithm which attempts to overcome the classic drift problems of behavioral cloning (BC). The idea is to reduce IL to a standard RL problem with a reward that incentivizes the agent to take expert actions in states observed in the demonstrations. The algorithm is tested on both image-based tasks (Atari) and continuous control tasks (Mujoco) and shown effective against GAIL and a simple supervised BC approach.
Comments
--------
Overcoming the limitations of BC algorithms is a relevant problem and this work presents a simple and interesting solution. The idea is to use any off-policy RL algorithm (assuming access to the true environment) with a reward that favors imitation. The paper is well-organized and easy to read. The experiments are also quite convincing and demonstrate that the proposed methodology is, in fact, more robust to distribution shift (i.e., less prone to overfitting the train set) than standard supervised BC and GAIL. Some comments/questions follow.
1. The considered settings (e.g., MDPs, IL/IRL problems) are never formalized in the paper (which directly starts by describing the method in Section 2). It would be good to add some preliminaries to allow even readers unfamiliar with this problem to easily go through the paper.
2. Though the approach solves one of the main problems of many BC/non-IRL algorithms, it still has some of the other limitations. For instance, the recovered policy is non-transferable to different dynamics, while a reward function is. Furthermore, it cannot recover optimal behavior if the expert is sub-optimal.
3. In many practical cases, the assumption that we can access the true environment is not reasonable. Suppose, for instance, that we have a dataset of driving demonstrations from a real car, and that is all we can obtain. Of course, we cannot start interacting with the real car to imitate the driver's behavior and we need to rely on simulators, which poses additional difficulties due to the inevitable approximations. Do the authors think that a batch RL algorithm could be adopted to solve this task (i.e., by solely using the expert data)? The ablation study in section 4 seems to answer this question negatively, at least for large distribution shifts.
4. Below equation (1), why "r does not depend on the state or action"? It should be R(s,a) = INDICATOR{(s,a) \in D_{demo}}, so it does depend on (s,a).
5. D_{demo} is missing from the inputs to Algorithm 1
6. I did not understand why, in Section 3.2 (above Equation 8), the authors mention that (7) squared is equivalent to the squared Bellman error of Equation 1. Where is r? It should be equivalent only from (s,a) \in D_samp, for which r=0, but for (s,a) \in D_demo we have r=1 and that term is missing in (7). Have I misunderstood something?
7. In Figure 1, why does GAIL perform better with shift than no-shift?
8. Why are some curves in Figure 3 (bottom) above the expert performance?
9. It would be interesting to show how the proposed method compares to IRL-based approaches for continuous states/actions such as [1] or [2]. These algorithms should be applicable to the considered domains (at least Mujoco ones).
[1] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (pp. 49-58).
[2] Boularias, A., Kober, J., & Peters, J. (2011, June). Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (pp. 182-189).


To reliably detect the out-of-distribution images based on already deployed convolutional neural networks, several recent studies on out-of-distribution detection have tried to define effective confidence scores without retraining the model. Although they have shown promising results, most of them need to find the optimal hyperparameter values by using a few out-of-distribution images, which eventually assumes a specific test distribution and makes it less practical for realworld applications. In this work, we propose a novel out-of-distribution detection method MALCOM, which neither uses any out-of-distribution samples nor retrain the model. Inspired by the observation that global average pooling cannot capture the spatial information of the feature maps from convolutional neural networks, our method aims to extract informative sequential patterns from the feature maps. To this end, we introduce a similarity metric which focuses on the shared patterns between two sequences based on normalized compression distance. In short, MALCOM uses both the global average and spatial pattern of the feature maps to accurately identify out-of-distribution samples.

** post rebuttal start **
After reading reviews and authors' response, I decided not to change my score.
I remark that this paper still requires a lot of revision; comparison in the main paper is somewhat unfair and all new results are in the appendix.
Also, the performance of their replication of the prior method is far lower than reported. In worst case, the performance gain from the compared method would be from their incorrect implementation on the prior works. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number. Ideally, if their method is evaluated in the same condition, it should outperform prior works in any case.
Detailed comments:
1-(a). Adversarial attack and OOD (which is hard to detect) are closely related: they are both in off-manifold. Their main difference would be, while adversarial attack is very close to the clean data in the data space, OOD is relatively far from the in-distribution in the data space.
Though it does not talk about OOD, you may refer to [R1] for analysis in perspective of manifold. The difficulty of OOD detection can be considered to be coming from overlapped manifolds in the latent space.
[R1] Stutz et al. Disentangling Adversarial Robustness and Generalization. In CVPR, 2019.
1-(b). Though the numbers are much lower than those reported in the original paper, I am happy to see the fair comparison. However, according to the original paper, simple FGSM is used for validation, so I am not sure such a huge difference can actually happen. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number.
2. I am happy to see that their revised method has better performance.
** post rebuttal end **
- Summary:
This paper proposes an out-of-distribution detection (OOD) method under constraints that 1) no OOD is available for validation and 2) model parameters should be unchanged. They specifically address a problem of the state-of-the-art method satisfying the constraints, and propose a new distance metric inspired by data compression. Experimental results on several benchmarks with different deep neural network architectures support their claim.
- Decision and supporting arguments:
Weak reject.
1. The problem setting is clear and their approach is interesting and makes sense. However, the method for comparison is not properly set. As the authors addressed, Mahalanobis detector proposed by Lee et al. (2018b) requires validation to determine weights for feature ensembling, but the validation can be done without OOD data by generating adversarial samples as proposed in the same paper. Although Table 2 in Lee et al. (2018b) shows that the performance is not better than the case when we have an explicit OOD data for validation, it reasonably works well. Therefore, rather than comparing with the vanilla version (only using last latent space) or the alternative "assemble" method (concatenating all average-pooled features), they had to compare their method with the model validated by adversarial samples, which essentially satisfies the constraints.
2. Also, I wonder the main body of the proposed method itself is really effective or some minor tweak they made is essential. According to Table 2 in the submission, their method is better than "Mahalanobis vanilla" only when all components are applied. Though the idea is interesting, I am skeptical about the effectiveness of the proposed method.
- Comments:
1. As addressed by the authors, feature concatenation ("assemble") is not effective for the Mahalanobis method but the proposed method. How about to do an ablation study about weighted averaging vs. concatenation on the proposed method as well? Again, weights can be validated by adversarial samples to satisfy the constraints.


Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented without implementing a new operator by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results showed that multi-sample dropout significantly accelerates training by reducing the number of iterations until convergence for image classification tasks using the ILSVRC 2012 (ImageNet), CIFAR-10, CIFAR-100, and SVHN datasets. Multi-sample dropout does not significantly increase computation cost per iteration for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated. Experiments also showed that networks trained using multi-sample dropout achieved lower error rates and losses for both the training set and validation set.

PAPER SUMMARY: The paper proposes a new and efficient implementation of dropout in which multiple dropout samples are obtained from a single input during training. The authors claim that this enhanced dropout technique (i) accelerates training and (ii) improves generalization by achieving lower error rates than standard dropout in training and validation sets. Experiments in image classification tasks for 4 different datasets (CIFAR-10, CIFAR-100, IMAGENET and SVHN) are presented and the effect of the number of dropout samples and dropout ratio are investigated.
REVIEW SUMMARY: Despite the clear computational advantages of the proposed implementation, this paper should be rejected because  (1) the authors claim multi-sample dropout to be a “new regularization technique”, but the conceptual differences between standard dropout and their proposed method are not clearly stated, (2) one of the main claims regarding “better generalization” is not well supported by their experiments, and (3) the experiments do not consider the stochasticity of the methods they are evaluating.
DETAILED COMMENTS: The multiple-sample dropout proposed by the authors leverages a parallel structure to provide significant acceleration in training, which is supported by experimental results. However, their claims regarding the introduction of a “new regularization technique” and “better generalization” are not convincingly proven in the paper. As mentioned in the paper, the proposed technique of multiple-sample dropout is analogous to the original implementation of dropout, but with larger batch size and duplicated samples. Therefore, the advantage of the proposed method is simply computational efficiency, but not a new “regularization technique”. In addition, while the paper claims the method provides better generalization, the results shown in Figure 6, in which the training losses for their multi-sample dropout and original dropout with duplicated samples are not significantly different, suggest that their validation error might not be significantly different either. The experiments shown in Figure 2, from which they conclude that their method generalizes better, are not conclusive since the validation error is evaluated at a stage in which the original dropout might have not converged, given the slower training. Therefore, the observed differences in validation error might come from the differences in training speed, and not from a better generalization ability of the model. Moreover, in the experiments the authors present single realizations of each training scenario, disregarding the stochastic nature of dropout-based techniques. In particular, the validation error curves observed in Figure 3 (where they analyze the effect of the number of dropout samples) seem very noisy, and the differences observed might simply come from the stochasticity of the training process. The authors should have executed multiple training runs with the same set of hyperparameters, and reported the statistics of their performance (same for results presented in Figure 2 and Table 1).  Overall, the advantages regarding training acceleration and computational efficiency are clear and well supported by the experiments. However, the novelty of the proposed technique and their claim regarding better generalization is not well supported theoretically or experimentally.
For the experiments, the authors should consider:
1)	Report validation errors after convergence of both methods
2)	Run the training scenarios multiple times and report performance statistics instead of single run performance
3)	Report validation curves for experiments in section 3.4 (Figure 6).


Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices. However, it is difficult to change the model size once the training is completed, which needs re-training to configure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models as little as possible. In experiments on multiple image-classification tasks, our method exhibits favorable performance compared with other methods.

This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.
Some of the ideas are interesting and would be worth developping further. However, the paper in the current state cannot be accepted for the following reasons: (1) the novelty is low, this very type of decomposition is already widely studied (2) the paper is not clear as to what the contributions are, and why they are justified, theoretically or empirically, (3) the review and comparison with the state-of-the-art is lacking and (4) the experimental setup is simplistic and not convicing. (5) overall the paper is imprecise.
Main comments
* Applying SVD to the matricized weights of deep neural networks is not new. Actual contributions need to be separated from existing works.
* The related word needs to be reviewed. Many references are missing. In particular, the proposed method could be considered as a special case of tensor based methods.
* The references that *are* listed in the related work are not properly reviewed: the authors aim to not compare with them claiming that they require re-training. Lebedev et al provide a method that works both for end-to-end training or post-hoc, by applying tensor decomposition to the trained weights. Fine-tuning is optional and done to recover performance.
* How was the complexity-based criterion obtained? Why use both M and P, since M includes P? How do the proposed criterion compare to simple measures, e.g. explained variance?
* The authors should compare with other compression techniques: layer-wise compression (e.g. Lebedev et al, Speeding-up convolutional neural networks using fine-tuned cp-decomposition, ICLR 2015) or full network compression (e.g. Kossaifi et al, T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019).
Missing experiments
* The proposed learning loss needs to be compared with the original one to demonstrate any potential performance improvement. Currently, it is not clear whether it is actually helping. In other words, there should also be a comparison with \lambda = 1 or 0.
* Does the proposed loss have an effect on the selected rank? On the actual rank of the weights? On the distribution of the eigenvalues?
* The BN correction needs to be experimentally motivated: since the network is trained with a loss that incorporates the low-rank network, is that needed? Does the propose loss affect performance? How does performance change with and without that BN correction?
* Experiments on CIFAR 10-100 is not sufficient to be convincing, the authors should ideally try a more realistic, large scale dataset, e.g. ImageNet.
* VGG-15 is not convincing to show-case compression, as more than 80% of the parameters are in the fully-connected layer
* The authors assume that the SVD decomposition of the weights does no change significantly at each step: is there any empirical evidence supporting this assumption? This most likely depends on the experimental setup (batch-size, learning rate, etc).


Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners. As networks grow in size and complexity, training time is reduced through low-precision data representations and computations. However, in doing so the final accuracy suffers due to the problem of vanishing gradients. Existing state-of-the-art methods combat this issue by means of a mixed-precision approach utilising two different precision levels, FP32 (32-bit floating-point precision) and FP16/FP8 (16-/8-bit floating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtain performance gains.This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision fixed-point representations. The novel training strategy, MuPPET, combines the use of multiple number representation regimes together with a precision-switching mechanism that decides at run time the transition point between precision regimes. Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efficiency compared to state-ofthe-art approaches. Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, MuPPET achieves the same accuracy as standard full-precision training with an average training-time speedup of 1.28× across the networks.

Summary:
This paper proposes a training strategy called Multi-Precision Policy Enforced Training(MUPPET). This strategy aims to reduce training time by low-precision data representation and computations during the training stage. According to the gradient diversity, the authors introduce a precision-switching mechanism which chooses the best epoch to increase the precision. The validation accuracy and training time across several networks and datasets are shown in the experiments. However, the results are not superior enough compared with the state-of-the-art.
My detailed comments are as follows.
Positive points:
1. This paper proposes a new reduced-precision training scheme to speed up training by progressively increasing the precision of computations from 8-bit fixed-point to 32-bit floating-point. This scheme moves to reduced-precision fixed-point computations while updating an FP32 model in order to push the boundaries of reduced-precision training.
2. The authors propose a metric to decide when to switch the precision inspired by gradient diversity introduced by [1]. In this paper, the gradient diversity is enhanced by considering gradients across epochs instead of mini-batches. The proposed metric can be seen as a proxy for the amount of new information gained in each training step. Therefore, the metric can decide the most appropriate epoch at run time to increase the precision.

3. The proposed low-precision CNN training scheme is orthogonal and complementary to existing low-precision training techniques.

Negative points:
1. The proposed approach does not match the description in this paper. The authors describe “This approach enables the design of a policy that can decide at run time the most appropriate quantization level for the training process”. In fact, this approach just decides which epoch to increase the quantization level while the levels of quantized precisions are fixed, rather than deciding the most appropriate quantization level.
2. The setting of quantized precision levels (8-, 12-, 14- and 16-bit precisions) is confusing. Please illustrate how to choose the number of quantized bit and the number of quantized precision levels.
3. The presentation of the precision switching policy is confusing and the notations are unclear. For example, in section 3.3, the ratio “p” needs more description because it is a key value in the policy, but lacks an explanation in this section. So please explain more about the motivation of ratio “p” in this section. 	In section 3.3, in step 5 of the proposed precision switching policy, the authors do not explain the meaning of “y”.
4. In figure 2, the precision switch is not triggered even though the value of p violates the threshold more than 2 times, which mismatches the description in section 3.3.
5. The proposed strategy has no obvious advantages. There are some scenes that the proposed strategy does not perform well. For example, the Top-1 validation accuracy on ImageNet of AlexNet and ResNet with MuPPET strategy is much lower than FP32 baseline. Compared with [2], the proposed method is more complex but not superior enough.
6. The authors do not show the training and validation curves. However, the training and validation curves are common used to show more details of the training process, such as in [2] and [3]. Please show and analyze the training and validation curves of the proposed scheme and the baseline.
Minor issues:
Some spelling and grammar mistakes.
Reference：
[1] Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning. In 21st International Conference on Artificial Intelligence and StatiZZstics (AISTATS), pp. 1998–2007, 2018.
[2] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018.
[3]  Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In 32nd International Conference on Machine Learning (ICML), pp. 1737–1746, 2015.


State-of-the-art Unsupervised Domain Adaptation (UDA) methods learn transferable features by minimizing the feature distribution discrepancy between the source and target domains. Different from these methods which do not model the feature distributions explicitly, in this paper, we explore explicit feature distribution modeling for UDA. In particular, we propose Distribution Matching Prototypical Network (DMPN) to model the deep features from each domain as Gaussian mixture distributions. With explicit feature distribution modeling, we can easily measure the discrepancy between the two domains. In DMPN, we propose two new domain discrepancy losses with probabilistic interpretations. The first one minimizes the distances between the corresponding Gaussian component means of the source and target data. The second one minimizes the pseudo negative log likelihood of generating the target features from source feature distribution. To learn both discriminative and domain invariant features, DMPN is trained by minimizing the classification loss on the labeled source data and the domain discrepancy losses together. Extensive experiments are conducted over two UDA tasks. Our approach yields a large margin in the Digits Image transfer task over state-of-the-art approaches. More remarkably, DMPN obtains a mean accuracy of 81.4% on VisDA 2017 dataset. The hyper-parameter sensitivity analysis shows that our approach is robust w.r.t hyper-parameter changes.

<Paper summary>
The authors proposed Distribution Matching Prototypical Network (DMPN) for unsupervised domain adaptation. DMPN extracts features from the input data and models them as Gaussian mixture distributions. By explicitly modeling the distributions that the features follow, the discrepancy between the distribution of source data and that of target data can be easily evaluated. DMPN is trained by jointly minimizing two kinds of loss, which are classification loss on the source data and domain discrepancy loss that is calculated via the explicit models. Experimental results on two popular benchmark datasets validate the advantage of DMPN over other state-of-the-art methods.
<Review summary>
The proposed method seems simple but empirically performs well. The paper is well written and easy to follow, so we can maybe easily implement it. However, I have several concerns mainly about the details and theories of the proposed method, which makes my score a bit lower than the border line. Given clarifications in an author response, I would be willing to increase the score.
<Details>
* Strength
 + The motivation of using ProtoNet for domain adaptation seems reasonable.
 + The proposed method performs well in the experiments.
 + The paper, especially the experiment section, is well written and easy to follow.
* Weakness and concerns
 - Several points on the proposed loss (GCMM and PDM) are not sufficiently discussed.
  -- Why do we need two kinds of loss? These losses seem to play almost same role. Since PDM loss corresponds to target-side log likelihood regularization term (Eq. (3)), I wonder if we really need GCMM loss.
  -- Since the authors explicitly model the feature distributions by Gaussian mixtures (GMs), it might be possible to calculate a standard divergence between source and target data distributions by using the parameters of GMs. Compared with such a straightforward approach, the proposed method seems to be ad-hoc and is not theoretically validated. What term of divergence (or distance) does it minimize?
  -- When a certain class does not appear in pseudo-labeled target data, how can we calculate GCMM loss? (specifically, \mu^{et}_c)
  -- Are Eq. (3) and Eq. (6) correct? These are defined as total loss, not average, over each domain. It means that the scale of the coefficients for these terms changes according to the number of training data, but the sensitivity analysis in Fig. 2 does not show such effect.
  -- Since the proposed losses heavily depend on the pseudo labels on the target data, it should be important to carefully set a proper threshold for the confidence. Is the proposed method sensitive against the change of this threshold? If so, how can we tune it?
  -- How can we know p(c) in advance?
 - The theory shown in 3.5 is not sufficiently validated.
  -- The authors state ````we minimize the first term through minimizing the domain discrepancy losses," but it is not sufficiently supported, because the relationship between the proposed losses and H-delta-H divergence is not clear.
 - I am concerned about whether the proposed method works well with harder datasets such as Office-Home dataset, because each class data are modeled by a simple Gaussian distribution in the proposed method.
* Minor concerns that do not have an impact on the score
 - Using both f^s_i and F(x^s_i; \theta) is confusing.
 - Typo in Eq. (7): PMD -> PDM


Neural sequence-to-sequence models are finding increasing use in editing of documents, for example in correcting a text document or repairing source code. In this paper, we argue that existing seq2seq models (with a facility to copy single tokens) are not a natural fit for such tasks, as they have to explicitly copy each unchanged token. We present an extension of seq2seq models capable of copying entire spans of the input to the output in one step, greatly reducing the number of decisions required during inference. This extension means that there are now many ways of generating the same output, which we handle by deriving a new objective for training and a variation of beam search for inference that explicitly handle this problem.
In our experiments on a range of editing tasks of natural language and source code, we show that our new model consistently outperforms simpler baselines.

This paper study the problem of editing sequences, such as natural language or code source, by copying large spans of the original sequence. A simple baseline solution to this problem is to learn a sequence to sequence neural network, which generates the edited sequence conditioned on the original one. This method can be improved by adding a copying mechanism, based on pointer networks, to copy tokens from the input. However, most of such existing approaches can only copy one input token at a time, which is a limitation when most of the input should be copied, which is the case for most editing tasks. In this paper, the authors propose a mechanism that can copy entire spans of the input instead of just individual tokens. In that case, a particular sequence can often be generated by many different actions (eg. copying individual tokens, pairs of tokens, or the whole span). It is thus important to marginalize over all the actions that generated a particular sequence. This can be done efficiently, using dynamic programming, if the probability of an action depends on the generated tokens only, but not on the sequence of actions used to generate them. In the case of neural network, this means that the decoder of the model takes the tokens as input, instead of the spans. To represent spans, the authors propose to use the concatenation of the hidden states corresponding to the beginning and end of the span. Then the probability of copying a span is obtained by taking the dot product between this representation and the current hidden state of the decoder, and applying the softmax. The authors evaluate the proposed approach on the following tasks: code repair, grammar error correction and edit representations (on wikipedia and c# code).
The paper is well written, and easy to follow, even if some sections could be a bit more detailed (for example, the section on
beam search decoding). The problem studied in the paper - copying spans from the input - is interesting, and has applications in NLP or code generation. I think that the the proposed solution is technically sound.
However, I have some concerns regarding the paper. First I believe that many relevant prior works are not discussed in the paper, making some technical contributions of the paper not novel. For example, previous methods were proposed to copy
spans from the input [1], to edit existing sequences [2], or to marginalize over different generation sequences
by conditioning only on the generated tokens (instead of the actions the generated the sequence) [3,4]. The body of work on iterative refinement for sequence generation is also probably relevant to this paper [5,6]. Additionally, I found the experimental section a bit weak, as most of the baseline used to compare seem a bit weak. The proposed method is mostly compared on datasets that are relatively new, or on tasks such as the grammar error correction where strong methods were excluded.
Overall, I found the paper well	written, and the proposed method to make sense. Unfortunately, I believe that the work is a bit incremental, most of the technical contributions having already been published. Since the experimental results are not very strong, I do not think the paper is good enough for publication to the ICLR conference.
== References ==
[1] Sequential Copying Networks, Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou, AAAI 2018.
[2] QuickEdit: Editing text & translations via simple delete actions, David Grangier, Michael Auli, 2017
[3] Latent Predictor Networks for Code Generation, Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, Phil Blunsom, ACL 2016
[4] Training Hybrid Language Models by Marginalizing over Segmentations, Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, Armand Joulin, ACL 2019
[5] Mask-Predict: Parallel Decoding of Conditional Masked Language Models, Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer, EMNLP 2019
[6] Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement, EMNLP 2018


Graph-structured data is prevalent in many domains. Despite the widely celebrated success of deep neural networks, their power in graph-structured data is yet to be fully explored. We propose a novel network architecture that incorporates advanced graph structural information. In particular, we leverage discrete graph curvature, which measures how the neighborhoods of a pair of nodes are structurally related. The curvature of an edge (x, y) defines the distance taken to travel from neighbors of x to neighbors of y, compared with the length of edge (x, y). It is a much more descriptive structural measure compared to previously ones that only focus on node specific attributes or limited topological information such as degree. Our curvature graph convolution network outperforms state-of-the-art on various synthetic and real-world graphs, especially the larger and denser ones.

In this paper, the authors proposed a novel graph neural network (GNN).
The proposed model combines discrete Ricci curvature of graph edge with GNN.
The combination method is simple but effective, which is easy to implement and is compatible with existing GNN models like GCN.
Experimental results show the potentials of the proposed model to node classification.
My main concerns include:
1. The complexity of the proposed model should be discussed. For a graph with V nodes and E edges, it seems that the authors need to solve E linear programming problems. The scale of each problem is decided by the number of neighbors per node. For large-scale graphs with dense connections, the scalability of the proposed method will be a problem. When the GCN models are with adaptive pooling layers, the curvature will change with respect to the layers of the model, and the scalability problem may become more serious.
2. For the graphs with isolated nodes, do we need to calculate the curvature associated with such nodes? If yes, how to calculate?
3. The authors introduce additional parameters to model the weight tau_{xy} from the curvature kappa_{xy}, i.e., Eq. (3.4). I wonder whether we can use kappa_{xy} directly as the weight tau_{xy} or not. It would be nice if the authors can add an experiment for this variant.
Overall, I think the idea of the proposed method is interesting. Introducing Ricci curvature to GCN is a potential method to take advantage of structural information beyond the adjacency matrix. However, the complexity of the proposed method should be analyzed in-depth.


Recent studies have made tremendous progress in style transfer for specific domains, e.g., artistic, semantic and photo-realistic. However, existing approaches have limited flexibility in extending to other domains, as different style representations are often specific to particular domains. This also limits the stylistic quality. To address these limitations, we propose Cascade Style Transfer, a simple yet effective framework that can improve the quality and flexibility of style transfer by combining multiple existing approaches directly. Our cascade framework contains two architectures, i.e., Serial Style Transfer (SST) and Parallel Style Transfer (PST). The SST takes the stylized output of one method as the input content of the others. This could help improve the stylistic quality. The PST uses a shared backbone and a loss module to optimize the loss functions of different methods in parallel. This could help improve the quality and flexibility, and guide us to find domain-independent approaches. Our experiments are conducted on three major style transfer domains: artistic, semantic and photo-realistic. In all these domains, our methods have shown superiority over the state-of-the-art methods.

Summary:
In this study, the authors propose a new method for performing artistic style transfer for  arbitrary image and styles. The new method employs a cascade/serial architecture for performing the style transfer. The authors test their method using human preference studies.
In summary, I found the architecture choice to be minimally explored. More importantly, a vast majority of the results to demonstrate the relative merits of this method were qualitative. The minimal quantitative results were unconvincing and left many unanswered questions about how well one could trust these results.
Major Comments:
1. No experiments to explore the architecture hyperparameters.
A natural question might be how the quality of the method varies systematically as the number of methods N grows. Presumably, if N=1, this would recover previous methods.
2. Authors are missing an important reference and point of comparison for arbitrary style transfer.
  Exploring the structure of a real-time, arbitrary neural artistic stylization network
  Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens
  https://arxiv.org/abs/1705.06830
  http://goo.gle/2oiDKaT
3. Minimal quantitative analysis.
A vast majority of the results (30 of 32 figures) are qualitative comparisons and the paper is sorely lacking an emphasis on quantitative comparisons. This is a large and notable problem in this paper and a quantitative comparison *should* constitute the primary thrust and central result of such a paper to convincingly demonstrate to a reader that the proposed method is indeed to superior to other techniques. I wish the authors dedicated more emphasis in this paper to a detailed quantitative comparison for these methods. As a starter, the analysis presented as the final two appendix figures (31 and 32) should be front and center in the result section of the paper.
4. User study for quantitative comparison is incomplete and unconvincing.
Table 1 and Appendix Figure 31 and 32 represent the primary result of this paper as these results and comprise the user studies to quantify how much better this method is to previous methods. These studies however are fairly unconvincing as lots of details are omitted and and I am concerned about the rigor of the human studies including but not limited to:
  4a. How long did each human study each image? What controls were added to the study to ensure that all images were equally studied by humans? For instance, were any golden tests employed to ensure user engagement throughout the study?
  4b. What was the repeatability of each measurement of preference? If a single human was presented the same image twice, how consistent were there ratings? For that matter, how consistent were the ratings across humans? I presume that some humans preferred some styles over others but how systematic was this?
  4c. What types of user testing scenarios were explored to ensure minimal bias in the results? Were multi-choice, paired choice or force choice employed? What about minimal or maximal time limit enforcement?
  4d. How can I have confidence that the authors did not cherry pick images and styles that favored their method? For that matter, I would expect that some methods work better on some styles or images. I would expect to see analysis accordingly to break down which styles/images work better on different slices of the data.
  4e. The statistical significance of Figure 31 and Figure 32 is not provided. What would an error bar look like with resampling?


Large datasets on natural language inference are a potentially valuable resource for inducing semantic representations of natural language sentences. But in many such models the embeddings computed by the sentence encoder goes through an MLP-based interaction layer before predicting its label, and thus some of the information about textual entailment is encoded in the interpretation of sentence embeddings given by this parameterised MLP. In this work we propose a simple interaction layer based on predefined entailment and contradiction scores applied directly to the sentence embeddings. This parameter-free interaction model achieves results on natural language inference competitive with MLP-based models, demonstrating that the trained sentence embeddings directly represent the information needed for textual entailment. The inductive bias of this model leads to better generalisation to other natural language inference datasets, and the resulting sentence embeddings are better for other semantic tasks.

***Update***
I'd like to thank the authors for responding to my questions and for the additional experiments. I think the new sentence embedding experiments make the paper quite a bit stronger - it would be interesting to scale them up to using SNLI + MNLI to see how much further they can go (right now they are still below Sentence-BERT which also was trained on SNLI in addition to MNLI). I think this paper is an interesting idea, and my only other main concern is the transfer results to other NLI datasets. I think it would be a good idea to confirm that the difference between the two approaches is due to biases perhaps though an error analysis. I am borderline on this paper, but I feel enough improvement has been made to raise my rating.
This paper proposes a new way to train sentence embedding models using NLI data such that very few parameters are used for classification. This is in contrast to prior work where an MLP is used. They define an interaction layer where operations are applied to the sentence embeddings to produce just 5 scores, which are then fed into a softmax layer for the final prediction.
The reason for their approach is they hypothesize that the the classification layers are encoding some of the information, presumably lessening the amount of information in the sentence embeddings and also preventing them from having a direct interpretation.
Their model does surprisingly well on the entailment datasets, only about 1-1.5 points or so worse than using a MLP, and so they indeed demonstrate that the sentence embeddings contain a lot of entailment information. However, I do have some concerns about their claims that this helps in transfer learning to other NLI tasks. Their main results show transfer performance comparing their approach and using an MLP, but it seems that overall, on all datasets, their approach transfers more poorly. Two datasets that they do worse on they try to discount for either having terrible, below random performance (which is true) and the other for having the same biases as MNLI. However, if that was the case, I don't see why their model would perform worse since both theirs and the baseline would benefit from having these biases, but their model performs about 11 points less.
So therefore, I don't find the transfer experiments convincing, though it is interesting how different the models do on some of these tasks - model performance is surprisingly task dependent
What I propose is for the authors to investigate if their sentence embeddings are in fact noticeably different than the ones trained in the more conventional matter. They could evaluate on sentence embedding and probing tasks (like SentEval) and see how the two models compare. It would be interesting to see wha encoded information differs between the models.
Im summary, I think this is an interesting experiment and it's nice to see that the MLP isn't doing a lot of heavy lifting (which also might be slightly counter to their hypothesis about the MLP containing a lot of entailment information). However, I find the transfer experiments unconvincing and the paper is short on analysis about when their model does better on transfer and when having an MLP helps, or how the learned sentence embeddings of the two models differ.


It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally unreasonable behavior nearby input examples. Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples. However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited. Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions. Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models. MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial. Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants.

Notes:
-Paper claims that adversarial examples stem from locally non-linear behavior.  However, wasn't this the exact opposite of the conclusion of "Explaining and Harnessing Adversarial Examples" (Goodfellow 2015)?  It is cited in this paper but I think the proper conclusion is the opposite of what is written here.
-Linearity however may still be an important component since it simplifies the problem of adversarial robustness.
-Many techniques try to introduce adversarial robustness through transformations during inference time.
-Paper claims that adversarial training induces locally linear behavior - but I'm rather skeptical of this claim.  Think about something like KNN with k=1 and euclidean distance.  This should be L2-robust on the training set, yet very non-linear.
-I understand the contents of Figure 1, but I'm not sure exactly what conclusion I should draw from it.
-The novel procedure presented is "Mixup Inference".  This involves classifying using interpolations of test inputs.
-Two mechanisms are proposed for why this could help.  One is that the magnitude of the perturbation will shrink after doing mixup (although the signal in the original image will also shrink, so I'm not sure if I like this argument).  The second argument is that the adversarial perturbation will have to appear with different random examples which will force the attack to be more "universal" to succeed.  This second argument I find much more compelling.
-The notation $y_s \sim p_s(y_s)$ is rather abusive since the random variable and the same have the same name but this is common in machine learning.
-The technique if I understand correctly (Algorithm 1) amounts to using an average of the prediction at the original point along with an average of the mixes going to all other points in the dataset.
-The paper is a bit slow to explain what distribution lambda will come from - but it effects the algorithm a lot (especially if the distribution is symmetric or asymmetric).
Comments:
-There is another paper (Shimada 2019) that also uses interpolation at test time and should be cited here, although I admit that paper is written in a confusing way so the connection may not be immediately obvious: https://arxiv.org/pdf/1906.08412v1.pdf
-I have a suggestion for the organization of the paper that I think would improve it.  I would suggest to first introduce the method in a clear fashion (after motivating it), along with the equations.  Then, *after that*, clearly and separately introduce the analysis of the "optimal linear model": "a well mixup-trained model F can be denoted as a linear function H on the convex combinations of clean examples".  I think that would make the paper much clearer.
-For example, how can we actually know what G_k() is unless we know the adversarial perturbation (which shouldn't generally be possible during inference)?  I found this discussion to be rather confusing (basically section 3.1) although admittedly it might be my own fault.
-Why does mixup-inference hurt the clean accuracy by 10% (table 2)?  This seems like quite a lot to me.  Still the degree of robustness does seem impressive.  And I also believe that the obtained robustness of this technique along with AT is state of the art.
-It might be nice to see examples of attacks on the resulting model, especially the one with the best robustness.  It's possible that Linf-bounded attacks against *this model* will be perceptible, which would support lowering the epsilon-attack-budget (which is actually a good thing for the research field as it's evidence that maybe this epsilon shouldn't be considered practically imperceptible).
Review:
This paper was interesting, because it has nice experimental results and seems like a good idea, but I feel like the paper needs to be improved.  The biggest issue is that the paper repeatedly claims that adversarial robustness can be improved by making networks more linear, yet I believe that this is the opposite of what prior work has found.  I also found the exposition of the idea to be confusing as it simultaneously introduces the idea and an analysis of the technique - I would much prefer if the technique were introduced first and the analysis under some optimal assumptions moved to a different section.


Title
Deep neural networks (DNNs) are poorly-calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. We first adapt prototypical learning for training of a classification model for DBLE. It yields a representation space where the distance from a test sample to its ground-truth class center can calibrate the model performance. At inference, however, these distances are not available due to the lack of ground-truth labels. To circumvent this by approximately inferring the distance for every test sample, we propose to train a confidence model jointly with the classification model by merely learning from mis-classified training samples, which we show to be highly beneficial for effective learning. On multiple datasets and DNN architectures, we demonstrate that DBLE outperforms alternative single-modal confidence calibration approaches. DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.

Summary:
The paper proposes a method to do confidence calibration for deep neural networks. It uses standard
episodic training for prototypical networks, and first shows empirically that the distances of the
embedded test point to its ground truth class center embedding (*not* the predicted class embedding)
are indicative of the confidence of the prediction. Further it proposes to exploit this by training
an auxiliary confidence prediction MLP carefully. To do so they demonstrate that the training needs
to be done of erroneously predicted training examples cf. all the traning examples. They show
results with MLP+MNIST, VGG11+CIFAR10, ResNet50+CIFAR100 and ResNet50+TinyImageNet.
Detailed comments:
The paper is interesting but largely empirical. It shows empirically that:
1. when prototypical networks (and episodic training) is used the distance of test example to true
class center reflects the confidence.
2. this does not hold when `vanilla' training is used
3. an auxiliary MLP can be used to learn to predict this while training only with erroneously
classified training examples cf. all training examples

The results are reported on three networks (MLP, VGG11 and ResNet50) on different benchmarks of
image classification. The confidence prediction improvements wrt baselines are non trivial, while
keeping the accuracy similar, and the computation cost lower than competing methods. Ablations
studies are also convincing.

I would have two broad critical comments on the paper:
1. Would this generalize to other image classification tasks and datasets. Generally distance
(embedding) based networks perform less than softmax based networks on bigger datasets, so an
immediate disadvantage if that happens, is that you would be trading off accuracy cf. vanilla
networks, for better confidence prediction using the required distance based network here.
2. The vanilla training is never formally detailed. I am assuming it was softmax + cross entropy loss
with gradient descent. Would some other loss be helpful? Specially the metric learning based losses like
contrastive or triplet losses come to mind, since they are also distance based. Does the method work
with prototypical networks only or it generalizes to other distance based methods as well?

Minor comments:
The notations are a bit confusing sometimes, and require going back and forth a bit. Eg. \mu is used
for representation of feature (Eq4) while it usually denotes a mean of some sort (so the reader's expectation
could be that it represents class center). Similarly, boldface p is used for class centers, which is
again a bit confusing as being a probability of some sort. In general the notations are different
from the original prototypical networks paper (which I needed to revise); keeping them similar would
help the reader. The contribution of the present paper is more than that anyway.
The erroneously classified training examples are called errors (eg. just before eq8). By errors one
could think that it is a difference between some sort of prediction and the ground truth. Explicitly
calling them erroneously classified training examples would help the reader as well.
The notation \odot is not explained (before eq.9).


Neural networks have succeeded in many reasoning tasks. Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, while less structured networks fail. Theoretically, there is limited understanding of why and when a network structure generalizes better than other equally expressive ones. We develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its structure aligns with the algorithmic structure of the relevant reasoning procedure. We formally define algorithmic alignment and derive a sample complexity bound that decreases with better alignment. This framework explains the empirical success of popular reasoning models and suggests their limitations. We unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP). We show that GNNs can learn DP and thus solve these tasks. On several reasoning tasks, our theory aligns with empirical results.

This work seeks theoretical and empirical proof of the reasoning capacity of neural networks. The authors build on a body of research that demonstrates the usefulness of different neural network architectures for different reasoning problems. For example, Deep Sets have been proposed to answer questions about sets (e.g., a summary statistic), and GNNs about graph related problems, such as shortest path.
I anticipate that readers would be very satisfied with the intuition behind the main result: neural networks that “align” with known algorithmic solutions are better able to learn the solutions. Many architectures have been proposed over the years, often with a high-level justification for the architecture’s form. For example, Relation Networks noted the difficulty with learning n^2 relations using an MLP, which is an observation reflected in this work’s explanation of the difficulty with learning a for loop.
Provided here is a justification for these high-level design decisions. The authors provide some theory and experimental results to demonstrate their proposed notion of alignment, and show that NNs that align with known algorithmic solution do well, while those that do not align do not do well. In particular, I appreciate both the positive and negative evidence, since demonstrating lack of alignment (and poor performance) is a necessary condition to show alongside alignment (and good performance).
I’d like to caution the authors regarding their main conclusion, which is stated a few times in the paper:
“This perspective suggests that whether a neural network can learn a reasoning task depends on whether there exists an algorithmic solution that the network aligns with”.
I think this logic is not precisely correct, and I would modify this to:
“If the structure of a neural network aligns with a known algorithmic solution, then it can more easily learn a reasoning task than a neural network does not align”.
This is a subtle but important difference. In particular, the original logic does not capture situations where an algorithmic solution is not known, but a neural network can otherwise still learn a solution (consider object classification). I think even the corrected logic as I’ve spelled it out above might not be quite right either, since it does not consider situations where the algorithmic solution exists, but it obtuse. Would a neural network easily learn such a task?
Overall I think the paper is clearly written, and the experiments are adequate. Unfortunately I am not well-versed in the theoretical literature on this topic, so my assessment of the proofs is limited, and I will need to defer to the other reviewers on these matters. My surface level assessment of them is that the logic seems generally sound, but I cannot make any strong statements placing them in the context of previous work, nor can I properly evaluate the nuances. Nonetheless, as a whole, I think this is a strong contribution and a nicely put together piece of work.


Although convolutional neural networks (CNNs) are inspired by the mechanisms behind human visual systems, they diverge on many measures such as ambiguity or hardness. In this paper, we make a surprising discovery: there exists a (nearly) universal score function for CNNs whose correlation is statistically significant than the widely used model confidence with human visual hardness. We term this function as angular visual hardness (AVH) which is given by the normalized angular distance between a feature embedding and the classifier weights of the corresponding target category in a CNN. We conduct an in-depth scientific study. We observe that CNN models with the highest accuracy also have the best AVH scores. This agrees with an earlier finding that state-of-art models tend to improve on classification of harder training examples. We find that AVH displays interesting dynamics during training: it quickly reaches a plateau even though the training loss keeps improving. This suggests the need for designing better loss functions that can target harder examples more effectively. Finally, we empirically show significant improvement in performance by using AVH as a measure of hardness in self-training methods for domain adaptation.

The paper proposes, when given a CNN, an image and its label, a measure called angular visual hardness (AVH). The paper shows that AVH correlates with human selection frequency (HSF) [RRSS19].
Pros:
1. The authors experimented with a representative set of trained models in my opinion. (More on this in Con2)
2. In Section 6, the authors acknowledge a substantive counter-example/argument. (More on this in Con2)
Cons (I put the ones that weighed the most on my decision first):
1. The presentation is confusing, and at times self-contradictory. For example, in Section 1, the paper asserts that “the two systems differ in what they view as hard examples that appear ambiguous or uncertain“ but then proceeds to claim that AVH being a CNN-derived quantity (more on this in Con2) correlates well with HSF. In fact, [RRSS19] (heavily cited here) seems to suggest exactly the opposite that harder examples for humans are also hard for CNNs. This is not very surprising as high accuracies of these CNNs imply their agreement with human judgment: we are learning a dataset labeled by humans. (More on this in Que2)
2. AVH is a function of a particular CNN (architecture and parameter values) and the target class label _in addition_ to the input image. These dependencies make AVH a measure of the ambiguity of an image very problematic. Granted that the paper presents evidence that AVH correlates with HSF for a number of _trained_ models but they will be of different values.
3. The work is not well self-contained. HSF, the core quantity studied is not introduced with sufficient details. (See Que1 and Sug1)
Some possible mistakes/typos:
1. Feature vector x and class weight w in general do not lie on S^n. Indeed your definition of A(u, v) only relies on u, v being nonzero.
2. There is a missing { above Definition 1.
3. In Definition 1, “for any x” -> for any (x, y).
4. In References, [10] is a duplicate of [11].
5. The captions in Figures 5, 6, and 7 in Appendix A might be wrong. They say ||x|| whereas the y-axis in the plots is labeled AVH.
Questions (I listed the important ones first):
1. What is human visual hardness (HVH)? How is HSF related to HVH? Why is being selected from a pool of images (in the procedure described in [RRSS19]) a good measure of HVH?
2. Since the class logit is exactly <x, w_c>, with arccos being a decreasing function, I expect AVH to behave very much like the opposite of model confidence (Definition 2). And this seems to be confirmed in Table 1 (performing a confidence calibration on validation set might increase this further). I wonder how AVH is different from model confidence _qualitatively_ and consequentially what insights do we gain (or should we expect to gain) by studying AVH instead of model confidence?
3. Degradation levels (DL) are mentioned early on but the experiments and figures were not shown in the main text (deferred till Appendix). What is the rationale?
4. The middle row in Figure 3 has a small range of ~1e-4. Is that expected? Can you provide some simple arguments? The closeness of the initial and final values of AVH in the AlexNet plot also concerns me.
5. How is the visualization in Figure 1 generated? It is not immediately clear to me how the high dimensional space is projected into 2D. My concern is that though suggestive in the figure, the category weights w_c in general do not spread out evenly. Do they? I would suggest reporting the angular separation of the category weights (maybe by showing them in a CxC matrix).
6. In Figure 2, what happens to the dark stripes? Are there no data points with the specified range of HSF values?
Minor issues (factored little to none in my decision):
1. There are 60+ citations but their relevance to the current seems questionable in many cases. Many of them are accompanied by little or no technical comparison when they are mentioned. In particular, in Section 2 on the related work from psychology/neuroscience, little specifics are discussed to contextualize the current work.
2. Many arguments come across as (highly) speculative and imprecise. As a result, I find the reasoning and logical story diluted and hard to follow.
3. The comparison with feature norm seems poorly motivated. The other quantities, namely AVH and model confidence, both depend on the class label.
4. The term hardness has a rich history and connotation in the algorithmic analysis literature. I would suggest using a different term, as the hardness of a problem usually reflects some intrinsic aspects of its structure and not dependent on some algorithm.
Suggestions:
1. If DL is not important to the core results, it will help simplify and focus the presentation by leaving them out entirely.
2. Try to be more concise and more precise in the presentation. It might also benefit from more formalism wherever possible, and more procedural details, when human studies or notions is involved. The latter seems to be a lesson from [RRSS19] (in regard to reproducibility).
In summary, I do not recommend accepting the current article.
(To authors and other reviewers) Please do not hesitate to directly point out my misunderstandings. I am open to acknowledging mistakes and revising my assessment accordingly.


Gradient Episodic Memory (GEM) is an effective model for continual learning, where each gradient update for the current task is formulated as a quadratic program problem with inequality constraints that alleviate catastrophic forgetting of previous tasks. However, practical use of GEM is impeded by several limitations: (1) the data examples stored in the episodic memory may not be representative of past tasks; (2) the inequality constraints appear to be rather restrictive for competing or conflicting tasks; (3) the inequality constraints can only avoid catastrophic forgetting but can not assure positive backward transfer. To address these issues, in this paper we aim at improving the original GEM model via three handy techniques without extra computational cost. Experiments on MNIST Permutations and incremental CIFAR100 datasets demonstrate that our techniques enhance the performance of GEM remarkably. On CIFAR100 the average accuracy is improved from 66.48% to 68.76%, along with the backward (knowledge) transfer growing from 1.38% to 4.03%.

This paper presents three improvements to the previous GEM algorithm: choosing support examples better (the GEM paper used a random set), incorporating soft gradient constraints, and specifying the magnitude of the dot product in the gradient optimisation problem (in order to increase positive backward transfer). They then compare the original GEM algorithm (and other baselines) with their algorithm on two datasets, consider the case where there is less memory available, and provide ablation studies for their three improvements.
I recommend to reject this paper. I am struggling to see the improvements in the results, as each algorithm is only run once on each dataset, and all the numbers seem very close together (GEM vs new algorithm). There are also a significant number of hyperparameters added by this paper: how are these tuned? I also do not understand the reasoning behind the third idea ('positive backward transfer'). I do, however, like the other two improvements and see the reasoning behind them, however I still have some misgivings. I will now elaborate on these points.
Firstly, there are no standard deviations provided for the experiments. I believe this is especially important to do for the original GEM algorithm and the proposed algorithm because the two provide extremely similar metric values. Comparing values from the original GEM paper and this paper's run of GEM indicates to me that there is a significant chance that any improvements could be within error bars. For example, on CIFAR100, the authors' run of GEM gives 66.48%, the original GEM paper reports 67.83%, and the authors' method is 68.76%.
Secondly, this paper introduces a significant number of hyperparameters over the original GEM algorithm. How are these tuned? Is there a validation set? Does having to tune these hyperparameters slow down computation significantly (how much exactly)? It would be nice to see how much more computationally expensive the new method is compared with GEM: the authors claim "little computational burdens".
I like the soft gradient constraint idea introduced in this paper. It is more principled than the hack that the original GEM paper used. The method of choosing the support set also makes sense to me, however there are many hyperparameters in this idea. Additionally, the results (Table 3) for different memory sizes are confusing. It seems like smaller memory sizes lead to less improvement over random memory. Surely cleverly choosing memory should be more beneficial when constrained to smaller memory? This indicates to me that improvements need to still be made to the idea. I also do not follow the explanation given in Section 4.2.1 for the positive backward transfer improvement. Why should the magnitude of the inner products be specified? What do you mean by "cosine similarity in magnitude"?
As a final comment, I will say that there are many typos in this paper.