We propose RAPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder. Precisely, RAPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces. We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input. We devise two metrics aggregating those hidden activated values to quantify the novelty of the input. Through extensive experiments using diverse datasets, we validate that RAPP improves novelty detection performances of autoencoder-based approaches. Besides, we show that RAPP outperforms recent novelty detection methods evaluated on popular benchmarks.

UPDATE:
I acknowledge that I‘ve read the author responses as well as the other reviews.
I appreciate the improvements and clarifications the authors have made, especially adding an ablation study to see the benefits of adding additional layers. I updated my score to Weak Accept (6).
####################
This paper considers deep autoencoders (AEs) for the unsupervised novelty/anomaly detection task and proposes to extend the standard AE anomaly score, given by the reconstruction error between the input and output in the original data space, to also utilize the reconstruction errors of the hidden activations in the AE network. The proposed method, Reconstruction along Projection Pathway (RaPP), specifically compares the hidden activations of all encoder layers given by the original input $x$ with the activations of the same units given by feeding the reconstruction $\hat{x}$ back into the AE. Thus RaPP compares the activation statistics of the original input $x$ and its reconstruction $\hat{x}$ along the encoder projection pathway from original data space to latent code space. Two ways for aggregating those reconstruction errors to a final anomaly score are presented: (1) Simple Aggregation Along Pathway (SAP) which simply computes the sum of reconstruction errors, and (2) Normalized Aggregation Along Pathway (NAP) which computes the sum of reconstruction errors after normalization via Singular Value Decomposition (SVD). The paper conclusively presents experiments on eight datasets from various domains, in which SAP and NAP are compared to the reconstruction error baseline for vanilla AE, VAE, and AAE, as well as experiments on MNIST and Fashion-MNIST in which NAP is compared to state-of-the-art deep anomaly detectors.
Though this work is well presented and indicates promising results, I think the paper should not yet be accepted due to the following main reasons:
(i) The experimental evaluation indicates promising, but not yet convincing results;
(ii) The computational complexity of NAP seems to be a major limitation of RaPP which is not addressed in the text;
(iii) The added value/insights from the theoretical Section 4 (Motivation of RaPP) are not clear.
(i) I think the experimental section shows promising, but not yet convincing results. To judge the significance of results, I think the paper should address the following:
(ia) The experiments on the eight non-image datasets should include other baselines (e.g. OC-SVM, Isolation Forest) besides the standard AE reconstruction error. One should expect SAP and NAP to improve over the standard AE since both methods include the original data space reconstruction errors as well. Moreover, the advantage of deep approaches on such non-image datasets is less clear [7] why a comparison to well-known baselines should be given.
(ib) The main motivation for deep approaches to anomaly detection are large and complex datasets [6, 5, 4, 2]. I think the comparison to recent, state-of-the-art deep competitors should at least include another dataset more complex than MNIST or Fashion-MNIST, e.g. CIFAR-10 as reported in the previous works or MVTec [1].
(ic) I think the proposed method begs for an ablation study of subsequently adding the reconstruction errors of additional layers. This would clearly demonstrate the potential benefits of adding the hidden reconstructions.
(ii) The experiments indicate that a proper normalization of the hidden activation reconstruction errors is crucial for improving detection performance. NAP shows consistent improvements, whereas SAP often performs similar to the AE baseline. However, the current SVD normalization procedure on a matrix with dimensions number of samples × number of hidden encoder units seems extremely costly to me and appears to be a major limitation towards larger datasets or networks. Could you comment on this since this is not yet addressed in the manuscript. Have you tried using Batch Normalization (after activation) together with per-layer averaging? To me, this seems the natural first choice to normalize unit scores and to account for different layer widths. Do you apply SVD on mini-batches?
(iii) The additional insights from the theoretical Section 4 are not clear to me. I think the presented reconstruction property for the hidden layers follows somewhat directly per definition for symmetrically constructed deep autoencoders (specifically if weights would be shared in addition). For a theoretical contribution, on the other hand, the proof and proposition should be fully rigorous in my mind, i.e. stating all the necessary assumptions on the function class (e.g. you implicitly assume invertibility and thus some smoothness of the $g_i$'s which Conv+ReLU modules do not satisfy for instance). As of now, I think this section does not add to intuition, but on the other hand is not completely rigorous. Maybe I am missing something?
The overall presentation of the paper is good (clear writing and structure, polished Figures and Tables). The work is well motivated and properly placed in the literature. Maybe since the approach is rather simple (which I don’t find negative), the author felt the need to add some rigor to the paper, which I think would not be necessary for a significant contribution if the experimental results hold up against the additional baselines and more complex datasets as described in (i).
####################
*Additional Feedback*
*Positive Highlights*
1. Simple idea that does not require autoencoder modification or retraining that indicates improved anomaly detection results.
2. The work is well placed in the literature. The related work includes all relevant and recent major works on the subject matter.
3. I appreciate the evaluation on both anomaly/novelty detection setups, unimodal and multimodal.
4. Comparison to recent OC-NN [3], GPND [5], Deep SVDD [6], and GT [4].
5. The writing, structure and overall presentation is good.
*Ideas for Improvement*
6. Include additional baselines and more complex datasets as described in (i).
7. Address the computational complexity of RaPP as in described in (ii).
8. Maybe cut the methodical/theoretical parts in Section 3.2 and Section 4 a bit. I think they are rather straightforward. Maybe combine Figures 1+2 as well. Extend the experimental evaluation instead.
9. Report the AUROC standard deviations over the trials as well to better infer statistical significance of the results (defer to appendix if space is a constraint).
*Minor comments*
10. Section 2: “Unsupervised and semi-supervised learnings” » “Unsupervised and semi-supervised learning approaches”.
11. Section 2: “Variational Autoencoders (VAE) was reported ...” » “Variational Autoencoders (VAE) were reported ...”
12. Section 3.1: “Due to this representation learning property, the autoencoder has been widely used for novelty detection.” » emphasis on unsupervised learning property, specifically.
13. Section 3.1: “Although this approach has shown a promising result in novelty detection ...” » “Although this approach has shown promising results in novelty detection ...”
14. Section 3.1, last sentence: “... in more details.” » “... in more detail.”
15. Section 3.2: “Those are especially suited for the case of zero-knowledge to interpret identified hidden spaces, which commonly happens when modeling with deep neural networks.” Zero-knowledge case? Reference?
16. In Section 5.1: “Further setups are described in Section 5.1”?
17. Section 5.4.1: “Also, we showed the best score ...” » “Also, we show the best score ...”.
18. Section 5.2: “... maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively.” Why use different ratios?
####################
*References*
[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[3] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.
[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[5] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pages 6822–6833, 2018.
[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.


Saliency methods help to make deep neural network predictions more interpretable by identifying particular features, such as pixels in an image, that contribute most strongly to the network's prediction. Unfortunately, recent evidence suggests that many saliency methods perform poorly when gradients are saturated or in the presence of strong inter-feature dependence or noise injected by an adversarial attack. In this work, we propose a data-driven technique that uses the distributionpreserving decoys to infer robust saliency scores in conjunction with a pre-trained convolutional neural network classifier and any off-the-shelf saliency method. We formulate the generation of decoys as an optimization problem, potentially applicable to any convolutional network architecture. We also propose a novel decoy-enhanced saliency score, which provably compensates for gradient saturation and considers joint activation patterns of pixels in a single-layer convolutional neural network. Empirical results on the ImageNet data set using three different deep neural network architectures-VGGNet, AlexNet and ResNet-show both qualitatively and quantitatively that decoy-enhanced saliency scores outperform raw scores produced by three existing saliency methods.

Given a model and an image, the proposed method generates perturbations of the image, such that the model output in intermediate layers of the network does not change. Their method first generates such perturbed images, and then uses those to generate a saliency map in order to interpret the classification.

The idea is overall interesting, but its relationship with many other methods is not adequately examined. Significant portion of recent literature is ignored, especially studies on counterfactuals and decision boundaries in the context of image classification.

For numerical results, they compare their results with raw scores generated by three other methods, but the results are quite thin and disorganized with respect to adversarial robustness and interpretability.
Your proposed computational method possibly needs to be changed, too.

 Details are explained below:

Authors have cited the paper on "Sanity Checks for Saliency Maps", but have not used it to verify their own results, not with respect to saliency map, and not with respect to computational cost.

Several recent methods on saliency maps have not been considered, for example:

1. Certifiably robust interpretation in deep learning

2. Interpreting Neural Networks Using Flip Points
3. Explaining Image Classifiers by Counterfactual Generation

4. Counterfactual Visual Explanations

It's not clear how your proposed saliency scores relate to methods that consider counterfactuals and the decision boundaries. For any image, there is a counterfactual image closest to it. One can also compute the closest image on the decision boundary of the model. Such image can reveal which pixels should be changed in order to change or to keep the classification. The relationship between the current method and those methods needs to be established.

Assume that for an image, you compute the closest image to it on the decision boundary and let’s call the distance between them r (distance measured in l1 or l2 norm). Then, consider a ball centered at that image with radius r. All images inside that ball would have the same label as the original image and you can obtain all of them only by solving one optimization problem for finding closest point on the decision boundary. Would this be a less expensive computation compared to solving your proposed decoy optimization many times?
There are many hyper parameters in your optimization problem, and it seems that for each image, hyper-parameters should be tuned separately. However, if you seek the closest image on the decision boundary, your results would be independent of hyper-parameters.
As you have mentioned, your optimization problem seems quite hard to solve, but no details are given on how long it takes to solve it for a single image. You have mentioned sometimes you cannot solve it if c is not chosen well. This information needs to be reported. Is it solvable at all with a large mask? How many trials did it take to obtain your saliency maps?
Your optimization is nonlinear, non-convex, non-differentiable, …. . How did you deal with non-convexity? Do you arrive at the same solution each time you solve it? These are vital information to about an optimization method.
Proposition 1 is quite obvious. Its discussion seems obvious, too, given how you have defined the Z. In your proof, you have considered a single layer network, i.e. a simple linear transformation. It might be better to explain your proof for Proposition 1 as an observation and use it as the motivation on how you have defined Z.

Overall, you are taking a data-driven approach to interpret the classification of an image. The additional computational cost of your proposed method needs to be compared to other methods that are not data-driven. For example, how does your computational cost compare with the references above?
On page 12, one equation is referenced with ??.
At the beginning of section 3.4, you are using a standard optimization technique to augment a constraint as a penalty in the objective function. You can explain it as such, instead of saying we solve an alternative formulation. See Numerical Optimization by Nocedal and Wright.
The necessity to transform the variable via tanh, instead of performing projection is not explained. Does it make the process faster?
Strategy for choosing a good initial value for c (in the penalty term) and how to increase it is not explained well. How many iterations does it take to find the solution?
Any classification model is defined by its decision boundaries, so interpreting the classification without considering the decision boundaries seems inconclusive.
In summary, it would be best to compare your results with other methods, some of which are mentioned above. It would also be necessary to justify the additional data-driven cost that you are prescribing and to explain why your goals cannot be achieved otherwise with less expensive computation.


Pretrained Language Models (LMs) have shown excellent results in achieving human-like performance on many language tasks. However, the most powerful LMs have one significant drawback: a fixed-sized input. Due to this constraint, these LMs are unable to utilize the full input of long documents. In this paper, we introduce a new framework to handle documents of arbitrary lengths. We investigate the addition of different mechanisms to handle an extended input size, and we demonstrate how attention can be used for parameter updates based on the most discriminating segments of input. We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text. We show our method consistently outperforms state-of-the-art results reported in recent literature.

The paper presents a methods to combine multiple pre-trained language models using an attention mechanism in order to take the whole document into account. The effectiveness of the methods is evaluated on two long document classification tasks (Arxiv publication and patent classification) with state-of the art results.
Pros:
- the effectiveness of the methods is experimentally demonstrated using two relevant datasets
- the inverted wireless experiment clearly shows the interest of the attention combination strategy
Cons:
- the methods is very simple (combining multiple segment prediction to perform document classifications) making the contribution of this paper quite weak. In the own terms of the authors, the "main contribution [...] is to investigate the effectiveness of different combination strategies". I am not sure that it is sufficient for the ICLR standards.
- the paper is difficult to read and should be proofread to improve readability
Minor issues:
- Socher et al (2011) uses recursive neural networks (ReNN) and not recurrent neural networks (RNN). RNN are ReNN but restrained to a linear chain structure
- In Introduction: ... in the  domain extremely complex data that is language ... -> I'm not sure the sentence is correct
- In Introduction: the last sentence should be shorten and rephrased.
- many typos


Miscalibration -a mismatch between a model's confidence and its correctness -of Deep Neural Networks (DNNs) makes their predictions hard for downstream components to trust. Ideally, we want networks to be accurate, calibrated and confident. Temperature scaling, the most popular calibration approach, will calibrate a DNN without affecting its accuracy, but it will also make its correct predictions underconfident. In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated. When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-art calibrated models, also preserves the confidence of the model's correct predictions, which is extremely desirable for downstream tasks. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to theoretically justify the empirically excellent performance of focal loss. We perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (SST, 20 Newsgroup) datasets, and with a wide variety of different network architectures, and show that our approach achieves state-of-the-art accuracy and calibration in almost all cases.

Summary:
This paper studies the effect of the focal loss, proposed by Lin et al. in 2017 on network miscalibration, which appears when the network's confidence in its prediction does not match its correctness. The authors provide a theoretical explanation to the superior results of the focal loss for calibration. The temperature scaling technique of Guo et al. 2017 is applied (dividing the network's logits by a scalar learnt on a val set prior to softmax) to networks trained using the focal loss, with different options for the focal parameter, as well as the standard multi-class cross entropy and a few others. The experiments on CIFAR10/100 as well as two text dataset (20 Newsgroups, Stanford Sentiment Treebank) reach lower expected calibration error compared to the cross entropy (75% of relative improvement on cifar100 for instance).
The importance of the contribution will probably be discussed here. At first glance, it seems that the works build mainly on advances from Lin et al & Guo et al, but the authors do a promising job in combining the two.
Positive aspects:
- The paper is well written.
- Experiments on both image and text dataset demonstrate the superiority of the focal loss on several calibration metrics.
- The theoretical explanation is convincing.
Negative points:
- The importance of the problem is motivated by future assessments by downstream tasks but do not address this aspect in the experiments. In particular, as the images experiments are conducted on tiny images, an experiment on a real size image dataset would strengthen the paper.
- The policy that works best for defining the sample wise tuning of the focal parameter was hand-made but ultimately uses only 3 parameters so finally it is not so bad.
Minor:
- It'd be nice to illustrate the confidence improvements on a few qualitative examples, maybe in appendix.
- 10 pages is too much (given that were were given instructions to be more severe with long paper) table 6 and 3 could be merged for instance.
- The focal loss column results of table 1 should be the same as Table 5 (sample wise)?
- could specify what MMCE means
- clean the bibliography
I've read the other reviews and authors' responses. Experiences on Tiny ImageNet are better than CIFAR but still a little far from what I'd call real images but I understand it can be difficult to run experiments on ImageNet. Since the choice of gamma seems to be leading consistent results also on tinyIN, I find it less concerning.


We study neural-linear bandits for solving problems where both exploration and representation learning play an important role. Neural-linear bandits leverage the representation power of deep neural networks and combine it with efficient exploration mechanisms, designed for linear contextual bandits, on top of the last hidden layer. Since the representation is being optimized during learning, information regarding exploration with "old" features is lost. Here, we propose the first limited memory neural-linear bandit that is resilient to this catastrophic forgetting phenomenon. We perform simulations on a variety of real-world problems, including regression, classification, and sentiment analysis, and observe that our algorithm achieves superior performance and shows resilience to catastrophic forgetting.

This paper proposes a neural linear bandits algorithm that is resilient to catastrophic forgetting when using limited memory.
The proposed algorithm Alg. 2 is similar to Thompson sampling for linear contextual bandits, Alg. 1, but using the last layer activation vectors as a linear feature, and also a different way of updating noise parameter prior and posterior is used based on Bayesian linear regression Eq. (2).
Alg. 2 also works with limited memory of history data, therefore after every time, the memory is refreshed, likelihood matching is used to calculate new Phi to make the likelihood (mean and variance) of reward estimation the same as it for the old feature. For mean matching, minimizing MSE Eq. (3) is used and for variance, solving PSD problem Eq. (6) is used.
The complexity of this algorithm is analyzed. And experiments are conducted to show that the proposed method is resilient to catastrophic forgetting and can achieve good cumulative reward results.
The proposed method is reasonable and the results look promising. However, I found several weak points as follows.
1. As mentioned Bayesian linear regression Eq. (2) is used to update noise prior and posterior, but this update has no theoretical guarantees as mentioned. As an algorithm mainly works under bandit settings, this is kind of undesirable.
2. This algorithm works with neural network-based features, but it is in nature not scalable as shown in the complexity analysis (linear dependence on action number). The linear feature is just replaced by the last layer activation of NNs. From this perspective, the experimental results just justify again that the NN feature is somehow powerful, which is as expected.
3. The likelihood matching can deal with catastrophic forgetting with limited history memory, which looks good. But the fact that it actually works for linear feature (last layer activation) together with realization assumption weakens this contribution a lot. The authors find using Eq. (3) is better than the exact mean matching Eq. (5), and there is no explanation for this, which kind of shows the proposed likelihood matching probably is not a good way when using full NNs rather than just linear features (last layer). On the other hand, the SDP seems also can only work under linear feature settings, and is not promising to be generalized to fully update for NNs.
Overall, this is a reasonable paper. However, on the one hand, as an algorithm mainly works under bandit settings, it is a lack of theoretical support. On the other hand, the linear feature setting weakens the contribution of likelihood matching to deal with catastrophic forgetting with limited memory. There are some questions of the proposed mean matching, and the matching is not able to generalize.


Deep neural networks are known to be vulnerable to adversarial perturbations. In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems. From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets. This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust. The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently. Experiments show that our method effectively improves deep model's adversarial robustness.

Summary:
The goal of this paper is to train neural networks (NNs) in a way to be robust to adversarial attacks. The authors formulate training a NN as finding an optimal controller for a discrete dynamical system. This formulation allows them to use an optimal control algorithm, called method of successive approximations (MSA), to train a NN. The authors then show how constraints can be added to this optimization problem in order to make the trained NN more robust. They show that the resulted constraint optimization problem can be formulated as a semi-definite programming and provide some experimental results.
Comments:
- Although the problem studied in the paper is important and the approach is interesting, it seems the paper has been written in rush and in my opinion is not ready for publication. The writing is not good. The introduction and related work sections are incomplete and not very informative. It is not clear what has been done before and what is the contribution of this paper. The main technique/algorithm of the paper has not been explained clearly that someone can easily understand and implement it. The experimental results are not convincing.
- There are strong claims in the paper such as "experiments show that our method effectively improves deep model's adversarial robustness", this is too strong, given the quality of the experiments of the paper. Or "the constraint optimization problem can be formulated as a semi-definite programming (SDP) problem and hence can be solved efficiently", to the best of my knowledge, SDP solvers are limited to small problems and cannot solve the large problems efficiently.
- The area of making NNs robust to attacks is a very active area and there are many attacks and solutions out there, which require more comprehensive empirical studies of any new method. I do not see this in the paper.
- Overall, I think this paper requires a major revision in order to be evaluated better and to be more useful for the community.


We propose an approach for sequence modeling based on autoregressive normalizing flows. Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques. We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models. Results are presented on three benchmark video datasets, where flowbased dynamics improve log-likelihood performance over baseline models.

Summary
The paper proposes to combine the video modeling approaches based on autoregressive flows (e.g. Kumar’19) with amortized variational inference (e.g. Denton’18), wherein an autoregressive latent variable model optimized with variational inference is extended with an autoregressive flow that further transforms the output of the latent variable model while allowing to compute exact conditional probability. This is motivated with a physical intuition, where a dynamics model can benefit from decorrelating the inputs, and it is demonstrated that layers of autoregressive flows can represent derivatives of the original signal. In a proof-of-concept experiment, it is shown that using a layer of autoregressive flow improves NLL of a latent variable model.
Decision
The paper presents an interesting method and tackles an important problem. At the same time, the properties of the proposed method are not well exposed and the experimental evaluation is incomplete. Moreover, the motivation of the paper is confusingly disconnected from the proposed model. I rate this paper as borderline, but am hopeful that some of the issues will be clarified during the discussion period.
Pros
- The paper is well-motivated and tackles a significant problem.
- The proposed method is novel.
- The paper is well-written.
Cons
- The experimental evaluation is incomplete and does not expose the properties of the method fully. Comparisons to prior art are missing. (see below)
- The motivation is disconnected from the proposed model. The introduction of the paper motivates a model that hierarchically decorrelates a sequence of frames to arrive at a fully factorized model, which is later motivated with a physical example. However, the method proposed in the paper is instead a single layer of autoregressive flow on top of a powerful latent variable model! This is expressed in the title, but only glossed over in the abstract and introduction. The writing has to be updated to coherently focus on the contribution of the paper.
Questions (ordered by decreasing importance)
1. In table 1, quantitative results are reported for the introduced methods. It is shown that introducing autoregressive flows achieves better likelihood and better generalization. However, quantitative comparisons with published methods that were evaluated on these datasets are missing, such as Denton’18 and Kumar’19. A quick calculation shows that Kumar et al. achieves a log-likelihood of -0.43 in Table 1 when converted to this paper’s metric, although it is possible my conversion is incorrect. Is the presented model competitive with previously published results?
2. No qualitative generation results are presented. Since the model achieves a high likelihood it is likely to do well on one-frame prediction, and possibly would even work on autoregressive multi-step prediction. Is the model capable of generation of diverse and plausible video?
3. The paper has a lengthy section 3.1 that convincingly explains that decorrelating latent variables in time is important for sequence modeling. However the proposed approach in fact produces latents that are correlated in time! Since the prior over latent variables is conditioned on past frames, the model can in fact learn a correlated representation and still achieve optimal likelihood. Moreover, the position of both the digit and the robot arm could be seen in what should be the decorrelated image in Fig 4. Is there solid quantitative (or even qualitative) evidence that the model learns a ‘more decorrelated’ representation beyond the fact that it copies the background and that the likelihood improves? The evaluation in this paper does not convince me that the model learns a temporally decorrelated representation.
4. Were modern techniques beyond affine flows considered, such as from Kingma’18, Kumar’19? Two layers of affine flows are likely insufficient to model the complexity of these data, which makes the comparison to the purely flow-based models somewhat unfair.
5. It is stated that the paper is “the first to demonstrate flows across time steps for video data”, however, the related work by Kumar et al. proposes a somewhat similar model in which conditional flows are used to model video data. Do Kumar et al. not “demonstrate flows across time steps”?
Minor comments
1. Eq (10) and (12) seem to be inconsistent. Perhaps x_t = x_t-1 + u_t-1 was meant in eq (10)?
2. Line before eq(14): it not true that u_t-1 = x_t-1 - x_t-2. It would be true if the deterministic x_t = x_t-1 + u_t-1 model was assumed instead of the gaussian N(x_t; x_t-1 + u_t-1, Sigma). It is possible that eq(14) is still correct as the variance of Gaussians is additive.
3. The following work uses autoregressive flows for modeling temporal dynamics and should be cited: Rhinehart’18,19
Rhinehart et al, Deep Imitative Models for Flexible Inference, Planning, and Control
Rhinehart et al, PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings
--------------------- Update 11.19 -----------------------
The newly provided experiments support some of the claims of the paper. In particular, I appreciate the plot showing that the proposed method successfully learns a more decorrelated representation over time, and the provided qualitative samples from the model. The authors also clarified my questions about motivation. At the same time, the proposed method is not shown to compare well to state-of-the-art approaches. I am leaning towards accepting the paper, but I believe the method would have a much larger impact if its properties were more fully exposed.
== comparison with Denton&Fergus'18 (SVG) ==
When trained with beta=1, as the authors suggest for comparison, this method is known to perform poorly. There are two possible ways of alleviating this: 1) to train with the modified objective as in the paper but evaluate the true lower bound on the likelihood, or 2) interpret the beta as the fixed variance of the decoder distribution. Given the results the authors have provided, I believe the latter option will lead to SVG outperforming the proposed approach.
== Correlation plot ==
Thanks for performing this experiment! While measuring correlation only captures linear dependencies, which is likely mostly the background image, this plot shows that the model indeed learns to (linearly) decorrelate the frames in the sequence.
== Samples ==
Thanks for providing samples from the model! While the performance on BAIR is not quite convincing, the MNIST samples look very good.
= Kumar et al. comparison ==
The author's response convinces me that the proposed model is significantly different from Kumar et al. in scope, as Kumar et al simply use a per-frame normalizing flow encoder coupled with a sequential prior.
== eqs. 10, 12 ==
The authors' response cleared my confusion, the equations are correct.


Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals. The standard way of learning in such models is by estimating the conditional intensity function. However, parameterizing the intensity function usually incurs several trade-offs. We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times. We draw on the literature on normalizing flows to design models that are flexible and efficient. We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form. The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data. Code and datasets are available under tinyurl.com/ifl-tpp.

The authors propose a new paradigm for learning models for point processes which circumvents the need to explicitly model the conditional intensity. This utilizes recent work on Normalizing flows and upends a long-standing paradigm. The paper is a true tour-de-force and the authors make a very convincing case for why instead of modelling conditional intensity, one could (and should) model the distribution of times directly.
The authors do an extensive literature review and place their work in the context very well. The writing is very lucid and the paper was a pleasure to read. The illustrations are also helpful and the extensive experiments complement the discussions very well. The appendix is also very easy to read and has been judiciously separated.
The two primary axes they justify their model is by first comparing it against conditional intensity-based methods in general and show that:
 1. Their approach has universal approximation property (thanks to a result from DasGupta (2008)).
 2. The likelihood function can be evaluated efficiently (thanks to it being a mixture of log-normals).
 3. The expected next action time can be evaluated efficiently.
 4. Sampling from their model is also easy.
Then they show that all related work fails either one or more of these conditions. Finally, they also address the common concerns of why point processes were historically always taught using intensity functions. A bit of historical context here is that in absence of modern computers and when Cox-Hazard models still needed to be worked out by hand, the conditional intensity functions indeed were much simpler to handle and afforded desirable properties. However, with modern approaches towards modelling probability distributions, those reasons are no longer valid. Further, some of the recent research which has built on the conditional intensity functions has lost much of the advantages anyway because of the use of deep neural networks. Hence, at this point, one might as well move directly to modelling the probability (as the authors have done) instead of taking a detour via the intensity functions. The experimental results for prediction seem to justify taking this route as modelling intensity function does not seem to out-perform the intensity-free models (more on the metrics later). Finally, the authors also show that their model, thanks to its generative nature, can also be used for several ancillary tasks: Sequence embedding, missing data imputation and learning with conditional information. The provided code is easy to read through and execute.
Hence, I do not have any hesitation in recommending the paper for publication.
As a side-note, it is notable that in (Tabibian, 2016), a mixture model is proposed, but I believe it is not for the final probability but instead for the intensity itself to make the learning problem tractable.
Some ways of improving the paper:
 - Theorem 1 can be made more rigorous by making the role of parameters like 'K' more explicit. e.g. does the theorem imply that there 'exist' K, \mu_k, \s_k, or does it say that for any K, \mu_k, and \s_k, such 'w_k' can be found, etc.?
 - Using NLL for comparing models in the experiment results is a bit unusual. Could a different metric (e.g. MAE) be employed?
 - A key discussion missing in the paper is that of the complexity of training the model. RMTPP, for example, can be trained very efficiently while Neural-Hawkes is difficult to train due to a Monte-Carlo sampling embedded in the calculation of the likelihood. Empirical results will also add to better placing the Intensity free method against other methods.
Citations:
 Tabibian, Behzad, et al. "Distilling information reliability and source trustworthiness from digital traces." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.


Vision-and-Language Navigation (VLN) requires an agent to follow naturallanguage instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are extremely useful in navigating new environments that the agent does not know about previously. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations which contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gap between seen and unseen on multiple datasets (i.e., 8.6% to 0.2% on R2R, 23.9% to 0.1% on R4R, and 3.74 to 0.17 on CVDN) and achieve competitive unseen results to previous state-of-the-art models.

This paper has two main contributions. First, the authors perform an extensive study to understand the source of what they refer to as 'environment bias', which manifests itself as a gap in performance between environments used for training and unseen environments used for validation. The authors conclude that of the three sources of information provided to the agent (the natural language instruction, the graph structure of the environment, and the RGB image), the RGB image is the primary source of the overfitting. The second contribution is to use semantic information, compact statistics derived from (1) detected objects and (2) semantic segmentation, to replace the RGB image and provide input to the system in a way that maintains state-of-the-art performance but shrinks the performance gap between the seen and unseen data.
This paper has some pretty exhaustive treatment diagnosing the source of the agent's 'environment bias' (which, as I discuss below, I believe is more accurately referred to as 'overfitting') in Sec. 4. To me, this is this highlight of the paper, and some interesting work; the investigation of the behavior of the system is interesting and informative. It provides a framework for thinking about how to diagnose this behavior and identify its source. The authors use this rather extensive study to motivate the need for new features (semantic features) to replace the RGB image that their investigation finds is where much of this 'environment bias' is located. Unfortunately, it is here that the paper falls flat. The authors proposal methods perform nominally better on the tasks being investigated, but much of the latter portion of the paper continues to focus on the 'improvement' in the metric they use to diagnose the 'bias'. As I mention below, the metric for success on these tasks is performance on the unseen data, and, though an improvement on their 'bias' metric is good anecdotal evidence their proposed methods are doing what they think, the improvements in this metric are largely due to a nontrivial decrease in performance on the training data. Ultimately, this is not a compelling reason to prefer their method. I go into more details below about where I think some of the other portions of the paper could be improved and include suggestions for improvement.
High-level comments:
- I am uncertain that 'bias' is the right word to describe the effect under study. In my experience, environment bias (or, more generally, dataset bias) usually implies that the training and test sets (or some subset of the data) are distinct in some way, that they are drawn from different distributions. The learning system cannot identify these differences without access to the test set, resulting in poor performance on the 'unseen' data. In the scenario presented here, the environments are selected to be in the train/test/validation sets at random. As such, the behavior described here is probably more appropriately described as 'overfitting'. The shift in terminology is not an insignificant change, because using 'bias' to describe the problem incorrectly suggests that the data collection procedure is to blame, rather than a lack of data or an overparamatrized learning strategy; I imagine that more data in the training set (if it existed) could help to reduce the gap in performance the paper is concerned with. That being said, I imagine some language changes could be done to remedy this.
- Perhaps the biggest problem with the paper as written is that I am not convinced that the 'performance gap' between the seen and unseen data is a metric I should want to optimize. This metric is instructive for diagnosing which component of the model the overfitting is coming from, and Sec. 4 (devoted to a study of this effect) is an interesting study as a result. However, beyond this investigation, reducing the gap between these two is not a compelling objective; ultimately, it is the raw performance on the unseen data that matters most. The paper is written in a way that very heavily emphasizes the 'performance gap' metric, which gets in the way of its otherwise interesting discussion diagnosing the source of overfitting and some 'strong' results on the tasks of interest. The criteria should be used to motivate newer approaches, rather than the metric we should value for its adoption. This narrative challenge is the most important reason I cannot recommend this paper in its current state.
- Using semantic segmentation, rather than the RBG image, as input seems like a good idea, and the authors do a good job of motivating the use of semantics (which should show better generalization performance) than a raw image. However, the implementation in Sec. 6.3 raises a few questions. First (and perhaps least important) is that 6.3 is missing some implementation details. In this section, the authors mention that 'a multilayer perceptron is used' but do not provide any training or structure details; these details should be included in an appendix. More important is the rather significant decrease in performance on the seen data (11% absolute) when switching to the learned method. Though the performance on the unseen data does not change much, it raises some concerns about the generalizability of the learning approach they have used: in an ideal world with infinite training data, the network would perfectly accurately reproduce the ground truth results, and there should be no difference between the two. Consequently, the authors should comment on the discrepancy between the two and the limits of the learned approach, which I worry may limit its efficacy if more training data were added.
Smaller comments:
- I do not fully understand why the 'Touchdown' environment was included in Table 1, since the learned-semantic agent proposed in the paper was not evaluated. The remainder of the experiments are sufficient to convince the reader that this gap exists, and I would recommend either evaluating against the proposed technique or removing this task from the paper.
- Figure captions should be more 'self-contained'. Right now, they describe only what is shown in the figure. They should also describe what I, as a reader, should take away or learn from the figure. This is not always necessary, but in my experience improves readability, so that the reader does not need to return to the body of the text to understand.
- The use of a multilayer perceptron for the Semantic Segmentation learned features, trained from scratch, stands out as a strange choice, when there are many open source implementations for semantic segmentation exist and could be fine-tuned for this task; a complete investigation (which may be out of scope for the rebuttal period) may require evaluating performance of one of these systems.


Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model. In this paper, we propose the Fréchet Joint Distance (FJD), which is defined as the Fréchet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric. We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for cGAN benchmarking and model selection.

Summary:
This paper extends the Fréchet Inception distance (FID) to the conditional distribution. To this end, the authors use an additional embedding for the condition variables (class, image, text), and concatenate to the data embedding. The proposed metric, name the Fréchet joint distance (FJD), captures three desired properties of conditional generative models: sample quality, conditional consistency, and sample diversity. The authors demonstrate that the proposed metric indeed captures the properties using a synthetic (dSprite) dataset, and shows reasonable values for real datasets.
Pros:
- FJD is an intuitive extension of FID for conditional generative models.
- FJD can be applied to various types of conditions (e.g., image and text), which cannot be done by prior work (e.g., [1]).
- The paper is easy to read and experimental details are clearly stated.
Cons:
1. FJD is a straightforward extension of FID.
FJD simply follows the FID formula but concatenates the condition embedding to the original data embedding. It is a straightforward extension of FID and suffers from the design choice problems due to the concatenation, as stated below.
2. FJD requires many design choices and not theoretically justified.
As FJD requires an additional embedding function h, balancing parameter \alpha, and merging function g, it raises a burden of design choices. While the authors give some suggestions, they are not theoretically justified. Also, one may use the statistical distances [2] between data distribution p_data(x,c) and model distribution p_g(x,c) to evaluate conditional generative models in a principled way, e.g., measure the KL-divergence using the density ratios [3]. The advantage of FJD over such metrics is unclear, as stated below.
3. The advantage over the prior work is not clear.
FJD and FID show the same trend in all reported experiments (Table 2, 3, 4), hence the advantage of FJD is unclear. Also, one may measure the FID score on conditional distributions, i.e., \sum_c FID( p_data(x|c), p_g(x|c) ). It also captures the desired three properties and would be a strong baseline for FJD. Besides, while the authors aim to design a single metric to stand the models in a line, identifying the trade-offs of models may also be useful. For example, Improved PRD [4] provides the precision-recall trade-offs of generative models, which provides some insights for the models.
[1] Ravuri and Vinyals. Classification Accuracy Score for Conditional Generative Models. NeurIPS 2019.
[2] https://en.wikipedia.org/wiki/Statistical_distance
[3] Uehara et al. Generative Adversarial Nets from a Density Ratio Estimation Perspective. arXiv 2016.
[4] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.


Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises. In this work, we show that highly-structured semantic hierarchy emerges as variation factors for synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation 1 .

The paper presents a visually-guided interpretation of activations of the convolution layers in the generator of StyleGAN on four semantic abstractions (Layout, Scene Category, Scene Attributes and Color), which are referred to as the "Variation Factors" and validates/corroborates these interpretations quantitatively using a re-scoring function. The claim of the paper is that there is a hierarchical encoding in the layers of the StyleGAN generator with respect to the aforementioned "Variation Factors".  Figure 3(a) illustrates how these "Variation Factors" emerge in the layers of the StyleGAN generator.
The basic GAN architecture used in this work is that of StyleGAN. However, details on the architecture of this particular GAN are missing, including in the Appendix. How many Convolution layers are present in its generator? Not everyone is aware of StyleGAN architecture -- A better illustration of their architecture in the main paper and its correspondence with the layer levels (bottom, lower, middle, top) is desired, mainly because the paper is built upon this. The dataset used to train the StyleGAN model is not clear either. In Appendix, Table 1 tabulates the training details, but nowhere is it clearly mentioned if the N=500,000 latent codes are sampled from a GAN model that was trained on a mixture of datasets (i.e., bedroom, living rooms, kitchen etc.) or individual datasets. As well, the unit of training time in Table 1 of the Appendix seems to be M; is it Million or Minutes? Both of them seem unrealistic units for training a GAN.
Since the training dataset is not clear, my understanding of the method is that a range of datasets are used to produce the results, especially for the effect where the transition of Semantic Category results is studied. As a first step, StyleGAN model trained on "bedroom" scenes from LSUN dataset is used to randomly sample codes from the learned distribution, which are further passed through the generator to obtain the respective image mappings. Off-the-shelf image classifiers are employed on each of the images to classify them to one of the four "Visual concepts", which is nothing but the aforementioned four semantic abstractions. Here, I would like to encourage the authors to use a consistent terminology -- The four semantic abstractions have been referred to as "Variation Factors" (page 3), "Candidate concepts" (page 6), "Visual concepts"(page 12), "Semantics" (page 8) interchangeably throughout the literature, which is confusing.  Then, 2000 top positive examples and 2000 top negative examples identified by the image classifiers are used to train a linear SVM, i.e., a binary-SVM all the four scene abstractions ("Varying Factors"), and the separation boundary is obtained. I assume the separation boundary is only obtained once, and not after every layer of the generator. Otherwise, it would not make much sense.
With the separation boundary (in the form of a normal vector) known for each of the four scene semantics, different feature activations are obtained by moving the latent code towards/away from the separation boundary. A scoring function is obtained to quantify (Equation 1) how the corresponding images vary in a particular semantic aspect when the latent code is moved from the separation boundary.  As per the last line of Paragraph 2 on page 4, a ranking of such scores using this function is used to understand the most relevant latent semantics. Does this mean that initially, a large set of semantics is used to observe whether the output of GAN is manipulated by probing each of them? Or are only four scene semantics chosen to begin with? And what happens when there is a tie? And, what value of K makes this metric more accurate? Any lower bound? Please explain. More questions on the effect of lamda later below.
In the next step, the authors sample a latent code from the learned distribution and pass it through
every layer of the GAN generator. The output code y is varied along the boundary of the SVM classifier. This is repeated at every layer of the GAN generator and the same lamda is used to perturb the resulting output code from the separation boundary. The results are visualized in Fig 3(c). The claim here is that with the same perturbation of the resulting codes (lambda=2) at the output of different GAN layers, the change in the visualized output demonstrates what kind of, if any, semantic is being captured by different layers of GAN. This is also claimed to have been validated through the "re-scoring" function. I am not very clear on this.
I request the following experiment:
1) Within just a single layer (be it bottom, lower, middle or top), how does the output change when the output code of that layer is perturbed in all directions? This is to see the effect (by visualizing) of the range of lamda values on the output at all the layers. Do you discover any changes weakening your claim?
2) I would like to see the visualizations of the latent codes at the separation boundaries, just to see how well the binary-SVM performs and whether or not, non-binary information is lost/unaccounted for.
On Page-5, see the fourth line from the bottom (going up): how do we know the desired output apriori? Are the four semantic abstractions decided based on the desired output? This takes us back to a question I asked earlier.
Moreover, Layout variation is just view-point variation. So I think it will be appropriate to call it "view-Point Variation" rather than "Layout Variation". This is because Layout is associated with spatial arrangement of objects in a scene, with functionality goals.
One last question I have is: What is so special about StyleGAN that it was used as the guiding architecture in this work? How generalizable is this approach to other kinds of GANs other than PGGAN and BigGAN (or rather, why is this approach relatable to StyleGAN, PGGAN and BigGAN alone)?
The paper has grammatical errors  (sentences are not well written), typos (ex; "manipulabe" on page-8 which should be "manipulatable") and is not polished. I also suggest the authors to change the title of the paper, which right now, is a bit odd; if you decide to keep it, there should not be a "the" before "Deep" in the title.
All in all, the paper is interesting but lacks persuasiveness.
I may jump my score if the authors address all the aforementioned questions and concerns convincingly, and work on the presentation.


This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called concentrated random vectors. Further exploiting the fact that Gram matrices, of the type G = X X with X = [x 1 , . . . , x n ] ∈ R p×n and x i independent concentrated random vectors from a mixture model, behave asymptotically (as n, p → ∞) as if the x i were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.

This paper provides a formal proof that the data produced by a GAN are concentrated vectors. The proof is rather intuitive: a GAN essentially consists of a Lipschitz function that takes as input a latent vector drawn from a Gaussian distribution and therefore one can use Lipschitz concentration bounds to prove that the transformed output satisfies a concentration property. Overall, I think this is a very intuitive idea and I fail to see the added value of such an observation. I provide detailed feedback below, I would really like to get a sensible answer regarding the novelty aspect.
Gaussian assumption
It is of course very common to sample the latent vectors from a Gaussian distribution, although one could potentially use a different distribution (uniform, gamma, …). To what extent could these results extend to other distributions?
Prior work
The paper does not appropriately discuss prior work that impose a Lipschitz constraint while training, see e.g.:
Gulrajani, Ishaan, et al. "Improved training of wasserstein gans." Advances in neural information processing systems. 2017.
Roth, Kevin, et al. "Stabilizing training of generative adversarial networks through regularization." Advances in neural information processing systems. 2017.
Theorem 3.3
I would like further clarifications regarding this theorem. This theorem is essentially a concentration bound for the resolvent of the gram matrix G around its mean. The expectation is already computed in (Louart & Couillet, 2019) so the contribution in this theorem seems to be in showing that the result only depends on the first and second order statistics. You claim this is a surprising result, although it does not seem so surprising to me given that this is an asymptotic result. Could you comment on deriving a non-asymptotic result instead?
Added value
As mentioned earlier, I do not see what particular insight is this paper bringing. There are some obvious connections that one could make, e.g. implications such as robustness to adversarial samples depending on the Lipschitz constant, or perhaps improve generalization property, but some of these connections are already made in prior work (see comment above). What particular insight do we get from your analysis?
Experiments
I think a more detailed study regarding the effect of various regularization schemes would be valuable. One could for instance compare various networks with/without batchnorm, resnet connections, ...


Numerous machine reading comprehension (MRC) datasets often involve manual annotation, requiring enormous human effort, and hence the size of MRC data remains significantly smaller than that of those available for unsupervised learning, limiting the generalization capability. To overcome this issue, a new approach, which can generate synthetic question-and-answer data from large corpora such as Wikipedia, has been recently proposed. Such synthetic data can be utilized as additional data to pre-train the main MRC model before fine-tuning it using real, existing MRC data. However, the quality of generated questions and answers is still far from being satisfactory, so previous work introduced a pretraining technique for the question generator by pre-training on the generation of the next sentence that follows a paragraph. However, the next sentence that follows a paragraph may have little relevance to the questions or answers from within the paragraph, and thus it is not the ideal candidate for pre-training question generation. In response, we propose a novel method called Answer-containing Sentence Generation (ASGen). Our approach is composed of multiple stages, involving two advanced techniques, (1) dynamically determining K answers from a given document and (2) pre-training the question generator using the task of generating the answer-containing sentence. We evaluate the question generation capability of our method by comparing the BLEU score with existing methods and test our method by fine-tuning the MRC model using the downstream MRC data after training on synthetic data. Experimental results show that our approach achieves outperforms existing methods achieving new state-of-the-art results on SQuAD question generation and increases the performance of the state-of-the-art MRC models across a range of datasets such as SQuAD-v1.1, SQuAD-v2.0, KorQuAD, and QUASAR-T with no architectural modifications to the original MRC model.

This paper proposes a pretraining technique for question generation, where an answer candidate is chosen beforehand, and the objective is to predict the answer containing sentence given a paragraph excluding this sentence and the target answer candidate. The intuition of this method is that question generation requires to generate a sentence which contains the information about the answer while being conditioned on the given paragraph. In particular, the paper compares its approach to Devlin’s presentation (https://nlp.stanford.edu/seminar/details/jdevlin.pdf according to the references; is it not a published work?) which uses next sentence generation for pretraining, that is less related to the downstream question generation task.
The proposed pretrained model is then used to finetune on a standard question generation task, which is then used to generate synthetic QA pairs for data augmentation in QA. The proposed method is evaluated on four datasets (SQuAD v1, SQuAD v2, KorQuAD, QUASAR-T) and have shown substantial performance gain.
The strength of this paper is clear to me: the idea in the paper is new and interesting, and they provide a good intuition of why the proposed learning objective is helpful compared to the previous methods.
However, I have several concerns as follows.
First, the performance improvements are marginal (less than 1.0 on three datasets, except QUASAR-T, which is actually interesting given it is under transfer learning setting).
Second, there is a very related work, “Dong et al. Unified Language Model Pre-training for Natural Language Understanding and Generation NeurIPS 2019” which also proposes a new pretraining approach for question generation and data augmentation for question answering. Not only this submission did not discuss this paper (it was posted in May 2019, which is substantially ahead of ICLR deadline), the result is substantially worse in both BLEU score of question generation and EM/F1 of end QA performance after data augmentation on SQuAD v2.
Third, the paper does not discuss any work in question generation overall, although the task of question generation was studied for decades, including question generation as data augmentation for question answering. I believe this paper should discuss previous work in question generation and compare the performance with them.
Some of the work on question generation.
- Heilman & Smith. Good Question! Statistical Ranking for Question Generation. NAACL 2010.
- Wang et al. Learning to ask questions in open-domain conversational systems with typed decoders. ACL 2018.
- Zhao et al. Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Network. EMNLP 2018.
- Kim et al. Improving Neural Question Generation Using Answer Separation. AAAI 2019.
Some of the work on question generation for question answering.
- Du et al. Learning to ask: Neural question generation for reading comprehension. ACL 2017.
- Song et al. A Unified Query-based Generative Model for Question Generation and Question Answering. AAAI 2018.
- Tang et al. Learning to Collaborate for Question Answering and Asking. NAACL 2018.
- Sachan & Xing, Self-training for jointly learning to ask and answer questions. NAACL 2018.
- Tang et al. Question Answering and Question Generation as Dual Tasks. AAAI 2018.
- Lewis et al. Unsupervised Question Answering by Cloze Translation. ACL 2019.
Fourth, it looks like the paper uses the development set for both development and the final evaluation. The paper should either use a portion of the train set for development and treat the original development set as the test set, or just show the final model’s performance on the hidden test set by submitting to the leaderboard. (Especially, for Table 3, did authors reimplement BERT+NS and evaluate on the dev set? Because Devlin’s presentation only shows the result on the test set from the official leaderboard.)
(This is not a weakness of the paper, but more a subjective opinion about the paper's claim) The paper claims answer-containing sentence generation is close to question generation. However, unlike question generation where core information about the answer is given as input text, answer containing sentence generation should generate the information about the answer itself with no given information. Consider the first example in Table 5. Question generation requires to read “The racial makeup of Fresno was 245,306 (49.6%)” and generate question “What percentage of the Fresno population is White?” However, in the proposed pretraining technique, the generation model is supposed to generate “The racial makeup of Fresno was 245,306 (49.6%)” with no information. In fact, I think the fact that answer candidate is given makes it closer to question generation task, rather than “generating answer-containing sentence” is the key.
(By the way, I am giving 3 as a rating although my actual rating is closer to 4 or 5, because 4 or 5 is blocked in the review system. I am happy to increase the score after rebuttals.)
--------------------------------------------------------------------------------------------------------------------------------------------
Now, here are clarification questions.
Regarding Section 2.1
1) Did you attempt to predict ‘number of answer candidates’ by regression or classification? Which objective function did you use? Based on the current description, it looks like it’s neither regression nor classification which makes me confused.
2) I believe the second last formula in this section should say “#{...}=k” instead of “#{...}”?
3) Have you compared with thresholding instead of predicting the number of answer candidates?
4) What is the objective of training the span selection model? There are multiple candidate spans given a single sentence, and hopefully you do not want to discourage candidates except only one.
5) How was this model trained (both predicting the number and predicting the span)? Was it trained on QA datasets such as SQuAD? Then, did you train this model & go through preprocessing separately for each dataset? (Except QUASAR-I which uses synthetic data from SQuAD v1.)
6) I believe that the most popular approach in question generation literature is selecting named entities & noun phrases using off-the-shelf tools, and wonder if authors have compared their method with this baseline.
Regarding Section 2.2
1) Although this section is the most important section in the paper, the description is a bit confusing to me. My understanding is that, if the initial paragraph has three sentences, <Sentence 1>, “The racial makeup of Fresno was 245,306 (49.6%).”, and <Sentence 3>, and the answer prediction model (described in Section 2.1) chooses “49.6%”, the question generation model is given “<Sentence 1> 49.6% <Sentence 3>”, and the model is supposed to generate  “The racial makeup of Fresno was 245,306 (49.6%).”. Is it correct?
2) Did you insert any special token to indicate whether it is the sentence or the target answer candidate, or which sentence is before or after the target sentence?
3) Have you tried the formulation of the masked language model, instead of the proposed model?
Regarding Section 3
Is preprocessing the same for all Golub et al 2017, NS and AS?
Regarding Section 4
1) I am curious why the improvement in BLEU-4 score (in Table 2) is significant but the improvement in the end task (in Table 3) is marginal. Do you have any intuition on it?
2) For Figure 5(a), do you have results with BERT without any data augmentation as well? I wonder if using too small number of SQuAD annotations makes question generation inaccurate, which makes synthetic data be bad quality and hurts the performance compared to not using any synthetic data.
3) How many synthetic QA pairs are used for Table 3? Curious since the model gets around 92.5 with the largest amount of data shown in Figure 5(b) but the number reported in Table 3 is 92.8.

**** Update on Nov 10 **** Increasing the score to 6.


With the rapid growth of data, distributed stochastic gradient descent (DSGD) has been widely used for solving large-scale machine learning problems. Due to the latency and limited bandwidth of network, communication has become the bottleneck of DSGD when we need to train large scale models, like deep neural networks. Communication compression with sparsified gradient, abbreviated as sparse communication, has been widely used for reducing communication cost in DSGD. Recently, there has appeared one method, called deep gradient compression (DGC), to combine memory gradient and momentum SGD for sparse communication. DGC has achieved promising performance in practice. However, the theory about the convergence of DGC is lack. In this paper, we propose a novel method, called global momentum compression (GMC), for sparse communication in DSGD. GMC also combines memory gradient and momentum SGD. But different from DGC which adopts local momentum, GMC adopts global momentum. We theoretically prove the convergence rate of GMC for both convex and non-convex problems. To the best of our knowledge, this is the first work that proves the convergence of distributed momentum SGD (DMSGD) with sparse communication and memory gradient. Empirical results show that, compared with the DMSGD counterpart without sparse communication, GMC can reduce the communication cost by approximately 100 fold with no loss of generalization accuracy. GMC can also achieve comparable (sometimes better) performance compared with DGC, with an extra theoretical guarantee.

Gradient sparsification is an important technique to reduce the communication overhead in distributed training. In this paper, the authors proposed a training method called global momentum compression (GMC) for distributed momentum SGD with sparse gradient. Following existing gradient sparsification techniques such DGC, GMC is also built up on the memory gradient approach; the major distinction between GMC and existing techniques is that GMC keeps track of global gradient to maintain the memory gradient, while the existing technique keeps track of worker-local gradients for memory gradient. The primary contributions in the paper are as the following:
1. The authors propose GMC, a training method for distributed momentum SGD with sparse gradient communication. It uses global gradient (but still achieve sparse communication) to maintain the gradient memory while existing approaches such as DGC use worker-local gradient to do so.
2. The authors prove the convergence rate of GMC for 1. strongly convex and smooth functions 2. convex functions and 3. Non-convex Lipschitz smooth functions. This is the first work on proving the convergence rate of distributed momentum SGD using sparse communication techniques based on memory gradient.
3. Empirically, the authors show that GMC can empirically attain the same model accuracy as conventional distributed momentum SGD with ~100x reduction in communication overhead. It can also match the performance of DGC at the same communication compression rate.
I think in general the ideas and efforts of the authors in proving the convergence rate of distributed momentum SGD with *gradient sparsification* is interesting and important. However, I have the some questions and concerns on validating the claims in the paper. I currently give weak reject but I am happy to raise the score if the authors can clarify or improve in their rebuttal / future drafts. The primary questions and concerns (critical to the rating) are:
1. One important claimed advantage of GMC over existing method is that it uses global gradient for memory gradient, while existing methods such as DGC uses local-work gradient to do so. But I did not find convincing support of this advantage in the paper: Empirically, in the experiment results, I don't think GMC demonstrate better performance than DGC in a statistical meaningful way; instead they are basically demonstrating matching performance. Theoretically, I am not sure if only the global gradient enables the proof of convergence rate while the worker-local gradient cannot. My preliminary feeling is that by bounding the gradient variance, it should also be possible to prove a rate for DGC using worker-local gradient; this is because the difference between the global gradient and the local gradient might be bounded via the gradient variance.
2. In the experiments, the authors focus on momentum SGD for image classification tasks. To better support the versatility and efficacy of GMC, it would be interesting to include some experiments for other domains (e.g. using the STOA transformer style models for NLP tasks). In these models, momentum-like components are also used in the optimizer (e.g. Adam for fairseq for machine translations), it will be interesting to see if the efficacy of GMC also empirically transfer to these settings.
Minor questions (influencing the rating in a secondary way)
1. Regarding the assumptions in the paper, I think assumption 2 need some validation / support to show that it is a proper one. My preliminary feeling is that assumption 2 is intuitive as the sparsification procedures only zero out small values so that the error introduced in the gradient is small and bounded. But it should be more convincing to empirically show the magnitude of u comparing to the magnitude of gradient g in Equ. 8.
2. I notice that the experiments uses conventional momentum SGD for a few epochs as warm up, is there any specific reasoning on using this warmup approach instead of the sparsity level warmup as used in DGC?
3. In the experiments, GMC does not use the factor masking trick while DGC uses. If it is for demonstrating the benefits of global gradient for gradient memory, I think it is more proper to also include the results of DGC without factor masking? In this way, this question can be directly answered in an ablation study way by eliminating the possible contribution of using/not using factor masking.
NITS to improve the paper (not related to the rating):
1. The last contribution bullet forgets to mention that it is about comparing to DGC.
2. In algorithm 1, it is clearer to mention how the mask m is generated (e.g. based on magnitude).
3. In the second paragraph in section 3.2, the vector inner product is not properly written between coefficients and w.
4. Above theorem 1, in the text, the condition for the discussion on the two cases are confusing.
5. In the definition of CR in the first paragraph of section 5, why the summation starts from 5. The text describes as warm up with 5 *epochs* while in the equation it is saying warm up with 4 *steps*.


Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable. Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients. Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

*Summary*
This paper provides the analysis of three algorithms in the context of minmax convex concave games: Simultaneous stochastic subgradient method, Simultaneous gradient with optimism and Simultaneous gradient with anchoring. These three algorithm are first analysed in continuous time with an ODE perspective and then leverage these intuitions and techniques to analyze the discrete time versions.
I think that these contributions are of interest of ICLR community, however, I have some concerns regarding the presentations of the results.
*Decision*
I vote for a weak accept that could move to an accept if the authors improve the presentation of the paper:
The paragraph “Regularized dynamics and convergence’ in Section 3.1 in quite hard to follow. This subsection is basically the proof of the convergence of the continuous version of  GD-O. Stating the result before the proof would help the reader to understand where the authors want to go.
Very same point for The subsection 4.1
For the stochastic version of SimGD-A a new parameter $\epsilon$ is introduced without any comment or description on why it is necessary.
A FID above 40 for MNIST is very far from standard results (that are below 1). Thus I am not convinced by the practical advantage of Anchored Adam on these models that have performances results very far from the standard ones.  (for instance between 20 and 25 for CIFAR10 is very reasonable). It is maybe because you do not use convolutional layers in your architecture. I think that using a DGAN architecture (for instance the one from the pytorch tutorial https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) would give the expected results (i.e. FID close to 0).
*Questions*
- When you cite Lassale Principle you mention that if $z_\infty$ is a limit point of $z(t)$ then starting at $z_\infty$, you stay at a constant distance to $z_*$. But in the bilinear example you give any point in the circle is not a limit point (size no dynamics converge to it). I guess when you said limit point you meant adherent point ?
- Is the condition $\epsilon$ only necessary for the proof ? Does $\epsilon =1$ work in practice ?
If no, what is the best value for $\epsilon$ ?


The Triplet Loss approach to Distance Metric Learning is defined by the strategy to select triplets and the loss function through which those triplets are optimized. During optimization, two especially important cases are easy positive and hard negative mining which consider, the closest example of the same and different classes. We characterize how triplets behave based during optimization as a function of these similarities, and highlight that these important cases have technical problems where standard gradient descent behaves poorly, pulling the negative example closer and/or pushing the positive example farther away. We derive an updated loss function that fixes these problems and shows improvements to the state of the art for CUB, CAR, SOP, In-Shop Clothes datasets.

Summary: The authors propose a new loss function as well as an adjoining visualization for improved performance of hard negative / easy positive mining for deep triplet metric learning. The authors note that under the NCA loss, if one selects an easy positive / hard negative and computes the gradient with respect to this example, this can lead to the negative example also being pulled closer to the anchor which is undesired. Similar phenomena can also be observed for easy positive / semi-hard negative mining as well. Motivated by this, the authors begin by designing a visualization to make this issue with NCA loss more apparent. Then they design what they refer to as an “entanglement factor” to quantify this issue more precisely. Using the desired dynamics of the gradients for the easy positive / hard negative mining and integrate to form what they refer to as the “second order loss.” Using this loss, they compare against the standard NCA loss on several datasets, showing modest performance gains. They also compare against a variety of other deep triplet embedding frameworks and show competitive results.

Comments, questions, and concerns:
- Overall the paper is well written and clear.
- The authors do a good job selecting reasonable baselines upon which to compare their method.
- It is worth noting, though not stated in the paper, that this trade-off between pushing away hard negatives and pulling in easy positives has actually previously been considered in the linear metric learning setting, though the terminology was different then. See the “Large Margin Nearest Neighbors” algorithm by Weinberger and Saul, for example which trains on “target neighbors” and “imposters.”
- Is projecting back to the hypersphere actually an issue? If the modelling assumption is truly that the learned representation should lie on the hypersphere, then this amounts to projected gradient descent and is a standard tool. In fact, projecting back to the sphere is correct under the model. If magnitude is truly important, then enforcing that the data be on the sphere is incorrect.
- The claim of a “systematic characterization for triplet selection strategies…” seems overly broad. It seems more correct to state this with respect to NCA loss specifically.
- NCA loss is just logistic loss penalizing the difference of similarities. The second order loss is the logistic loss penalizing (1/2 times) the squared difference of similarities, hence the cross term. Might be good to state this explicitly.
- The figures throughout are difficult to read due to the small font size.
- The y-axis in figure 4 is very different for each plot, which makes the effect seem much larger than it is.
- There are no error bars on plots or a discussion about how much variance a practitioner should expect running these experiments. If every single training run ever takes the same amount of time, who cares, but otherwise, we have one sample from each distribution and it’s hard to infer from that.
- Additionally, in Table 1, since there are no confidence regions, it is difficult to know if when the algorithm does perform well if these differences are significant. Especially important since on several datasets, the model does not perform as well as others.


Artificial neural networks (ANNs) are very popular nowadays and offer reliable solutions to many classification problems. However, training deep neural networks (DNN) is time-consuming due to the large number of parameters. Recent research indicates that these DNNs might be overparameterized and different solutions have been proposed to reduce the complexity both in the number of parameters and in the training time of the neural networks. Furthermore, some researchers argue that after reducing the neural network complexity via connection pruning, the remaining weights are irrelevant and retraining the sub-network would obtain a comparable accuracy with the original one. This may hold true in most vision problems where we always enjoy a large number of training samples and research indicates that most local optima of the convolutional neural networks may be equivalent. However, in non-vision sparse datasets, especially with many irrelevant features where a standard neural network would overfit, this might not be the case and there might be many non-equivalent local optima. This paper presents empirical evidence for these statements and an empirical study of the learnability of neural networks (NNs) on some challenging non-linear real and simulated data with irrelevant variables. Our simulation experiments indicate that the cross-entropy loss function on XOR-like data has many local optima, and the number of local optima grows exponentially with the number of irrelevant variables. We also introduce a connection pruning method to improve the capability of NNs to find a deep local minimum even when there are irrelevant variables. Furthermore, the performance of the discovered sparse sub-network degrades considerably either by retraining from scratch or the corresponding original initialization, due to the existence of many bad optima around. Finally, we will show that the performance of neural networks for real-world experiments on sparse datasets can be recovered or even improved by discovering a good sub-network architecture via connection pruning.

This submission studies losses at local minima of a set of neural networks trained on an XOR-like synthetic dataset, finds that local minima are of varying quality, and proposes a network pruning method to find better local minima. The pruning method is evaluated on XOR-like datasets as well as real-world datasets.
The use of an XOR-like dataset to study loss landscapes is interesting, making for a controlled and analyzable setting to carry out the study. The way the authors set it up, the XOR-like problem involves nuisance variables that naturally introduce suboptimal local minima into the loss landscape (this is my observation as a reviewer -- I am not sure if the authors were aware of this). I am unsure if Section 2 of the paper was intended as a core contribution or as a motivation for the pruning algorithm proposed in Section 3. Given the set-up’s simplicity, a short theoretical argument (maybe even a theorem) about the quality and number of local minima one would expect to find could have been more concise and compelling than the empirical analysis from the paper. The findings from Section 2 may not be surprising enough to warrant two full pages.
Section 3 proposes a network pruning method to find better local minima. The authors cite a paper by Adrian Barbu as the inspiration for their pruning algorithm with annealing, and use it “to improve the capability of NNs to find a deep local minimum even when there are irrelevant variables”. The cited paper by Barbu as well as https://arxiv.org/pdf/1805.01930.pdf (also by Adrian Barbu, not cited, maybe because it appeared) explore feature selection and regularization with (nearly) the same annealed pruning algorithm in some detail. I would be grateful if the authors could highlight the differences between their work and Barbu’s.
I vote to “weak reject” this paper. The paper discusses interesting ideas, but other ICLR submissions present deeper and more novel material, and there appears to be some (unintentional, I believe) overlap with already-published work. I recommend that the authors cite and discuss https://arxiv.org/pdf/1805.01930.pdf , and possibly submit the paper at a less competitive conference.
Further comments / questions / advice
=================================
- It would be helpful if the authors made more clear what they consider the key contributions of their paper. If contributions build directly on earlier work, it’s helpful to highlight the differences.
- Section 4.2 states that datasets were “carefully selected” in what sounds like a case-by-case basis, probably with the goal of finding data sets on which CPNA outperforms networks trained with vanilla gradient descent methods. This process would have selection bias and surface data sets on which CPNA outperforms. I could be grateful if the authors could clarify if this was indeed the process, or if a less biased criterion was used. For example, one could have chosen data sets on which a 1-layer fully connected neural network achieves between 50% and 90% F-1.
- A reader of the paper might wonder for what data sets they should use CPNA in order to train network that achieves low out-of-sample loss. I could be grateful if the authors could comment on this. Following up on the previous point: it would be great the authors could include data sets where CPNA does not outperform.
- Could the authors include information on how long training takes for the experiments from Table 3?
- https://openreview.net/pdf?id=HkghWScuoQ should probably be cited
- https://arxiv.org/pdf/1805.01930.pdf should definitely be cited


Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

This paper proposes a new approach to learning contextualised cross-lingual representations based on a three-step procedure called: transform-extract-reorder. The proposed model is tested on zero-shot cross-lingual transfer for one language pair: English-Chinese, where the gains are reported over the initial multilingual BERT model which does not rely on any bilingual signal. I like the main modeling idea, and the actual model is described quite nicely and in an intuitive way. However, I believe that much more work is needed in terms of proper experiments including: a) additional strong and insightful baselines; b) experiments with more language pairs; c) experiments across other transfer tasks. The paper also does not include nor compares to all relevant previous work. My main comments and remarks are as follows:
*Related work and Novelty* The topic of learning contextualised cross-lingual representations has become quite popular since the rise of ELMo and BERT. However, the authors cite only one paper that proposed such contextualised xling representations (the work of Schuster of al.). However, they never compare to that model, which is quite weird given the fact that the main goal of all xling representation models is to enable cross-lingual transfer in the first place. A comparison to the cited work of Schuster et al., plus additional comparisons to other (non-cited) models are required to clearly understand the main empirical benefits of the proposed three-step framework. Other relevant papers which are not cited nor compared against: Mulcaire et al. (NAACL 2019); Aldarmaki and Diab (NAACL 2019).
The authors should really stress the key novelty of their approach in comparison to the existing literature (which is mostly based on simpler projection-based methods).
- Note that there are more papers on the subject getting published soon such as Mulcaire et al. (CoNLL 2019) and Liu et al. (CoNLL 2019); while making comparisons to models introduced in those papers is not required, it would be nice to also briefly mention that latest work in the related work section once it gets published.
*Comparisons to XLM* Based on the limited set of results reported in the paper, it seems that the proposed model cannot match the performance of the XLM model (Lample and Conneau, NeurIPS 2019). The authors try to justify the usage of their method over XLM by emphasising the fact that their method produces language-independent xling representations, which XLM cannot do. However, there are multiple issues here: 1) it is not clear why exactly this requirement of language-independence is needed for a task such as zero-shot XNLI; 2) the paper does not demonstrate the usefulness of that language-independence property empirically at all. To me, it just seems like an unsuccessful attempt to make a conceptual distinction between the proposed model and XLM given the fact that XLM seems to significantly outperform the proposed model.
Page 8 ("Training cost and beyond"). The authors claim that 250k pairs of parallel sentences is enough to learn a near-optimal cross-lingual model and that the model saturates with more parallel sentences. I don't see it as a positive aspect of the model as claimed by the authors. This means that the model cannot optimally encode knowledge about variety of cross-lingual contexts during its training, i.e., it stops learning earlier than expected. Why does that happen? A plot showing transfer learning results of XLM versus the proposed model versus some static word embedding model that also exploits parallel data would be very useful. The authors also do not even speculate why their model saturates already with 250k sentence pairs. Given the limited set of results (only one task and only language pair) it also remains unknown how general this phenomenon is and whether the same point is hit with some other language pairs and with other tasks. This definitely requires further and more thorough investigation.
*Other Comparisons and Experiments*
As mentioned, the paper is very limited when it comes to proper thorough evaluation. Most experiments focus on intrinsic/internal evaluations of different steps of the proposed framework, or some probing tests, which can also be seen simply as diagnostic experiments. The actual downstream experiments are conducted on only one language pair and for only one (zero-shot) transfer task. This is definitely not sufficient to draw any generalisable claims, and it prevents us to dig deeper into other interesting aspects of the model. I would suggest the authors to maybe run the model across the same range of tasks as done in the XLM work of Lample and Conneau, and definitely for more languages (ideally diverse target languages).
Also, besides XLM and translation-based baselines reported by multilingual BERT's github page, there is actually no other baselines, which really makes it hard to put this work in context, and understand its usefulness. For instance, static cross-lingual word embeddings could be added to ESIM and used to evaluate on XNLI as well (see, e.g., the work of Glavas et al. (ACL 2019)). It would be interesting to report the benefits of replacing such static vectors with truly contextualised representations. Also, as mentioned before, comparisons to other models that learn contextualised cross-lingual representations are definitely something that should be included in the paper.
*Other Comments*
- Based on the result from Table 4, it seems that, in order to enable fully language-independent representations, one must sacrifice some of the monolingual performance, as the numbers drop from T over T+E to T+E+R variant. Is the same pattern visible for other languages monolingually? Why does this happen?
- As mentioned before, the whole emphasis is of language-independence is somewhat oversold in the whole paper without providing sufficient empirical evidence that this is crucial for transfer performance.