One of the most surprising and exciting discoveries in supervising learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it has been observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). In this paper, we perform an exhaustive study of different aspects of overparameterization in unsupervised learning via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out loglikelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that, when learning using methods such as variational inference, larger models can significantly increase the number of ground truth latent variables recovered.

The paper “aims to be a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings. ” The author’s empirical study is comprehensive, and to my knowledge the most detailed published work on this to date. Specifically, the authors empirically study
- the ability of networks to recover latent variables
- the effects of extreme overparameterization
- the effects the training method (e.g. batch size)
- latent variable stability over the course of training
In line with the findings for supervised settings, the authors find that overparameterization is often beneficial, and that overfitting is a surprisingly small issue. This is an interesting and useful observation, particularly since it at first sight appears to be in disagreement with some earlier work (the authors suggest explanations for the differing observations).
As the authors point out (and I agree), the paper constitutes a compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods.
The authors perform studies on a range of different real-world and synthetic datasets.
The paper is well-written, well-structured, and easy to follow. Relevant literature has been cited. The appendices contain a wealth of details that will make this work reproducible.
Decision: weak accept. The paper contains some new insights, but its contributions are not quite as substantial (e.g. lack of precise mathematical statements) or surprising as those in stronger ICLR papers.
A small gripe: the authors promise “ a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings”. I would argue that “making precise” is too strong for what the paper actually delivers. I suggest rewording this.


An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove the consistency of the method under general conditions, provide a detailed error analysis, and demonstrate strong empirical performance on benchmark tasks, including off-line PageRank and off-policy policy evaluation.

In this paper the authors proposed a framework for off-policy value estimation under the scenario of infinite horizon RL tasks. The new proposed method utilize the variational representation of $f$-divergence, which quantifies the difference between $\mathcal{T}\tau$ and $\tau p$, where $\tau$ is the parametric density ratio between the unknown behavior policy data and the target policy. If only if $\tau$ is the true density ratio, the loss $\mathcal{D}_{f}(\mathcal{T}\tau || \tau p) = 0$.
Compared with prior work (Nachum et. al 2019), the new proposed framework can generalize the undiscounted case $\gamma = 1$, and the derivation for the new algorithm is quite simple and easy to follow. The experimental results show the advantage of the proposed methods over baseline methods such as model-based, DualDice etc, for both discrete and continuous cases. Moreover, I have two specific questions:
- The choice of $f$-divergence. Although the author mentioned the difficulty of using the dual representation of KL divergence, it would be nice to have an ablation study that shows the effectiveness of various $f$-divergence (Personally I think Jensen-Shannon Divergence may be also a good choice).
-The authors should also have a discussion that similar idea can be generalized to more general  distribution metrics such as Integral Probability Metrics, specifically wasserstein-1 distance (similar to wasserstein-gan) or maximum mean discrepancy (Maybe it is unnecessary to conduct experiments, some discussion should be enough to clarify the relationship. I think there is a concurrent submission using MMD metrics).

Overall I think this is a good paper and I recommend for acceptance.
Reference Papers:
- Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems. 2016.
- Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein gan." arXiv preprint arXiv:1701.07875 (2017).
- Nachum, Ofir, et al. "DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections." arXiv preprint arXiv:1906.04733 (2019).
- Anonymous, “Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning”, submitted to ICLR 2020.


This paper is concerned with the robustness of VAEs to adversarial attacks. We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as β-TCVAE (  (2018)). This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

This paper examines adversarial attacks to a VAE. It is known that by small norm perturbations on the conditioning input x of a VAE can dramatically change the generated output. This paper
empirically illustrates that alternative objectives can improve robustness, in the sense of previously proposed adversarial attacks such as (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)). This paper is concerned with the stability of reconstructions and their quality, and proposes Seatbelt-VAE to remedy some of the shortcomings in the original VAE objective.
The key idea of the Seatbelt-VAE is introducing a conditionally Gaussian chains (of length L) in the encoder and decoder distributions of a VAE. This is a plausible and sensible idea. Then the authors evaluate the robustness of reconstructions under various output attacks.
The methodological part of the paper is quite well written and easy to follow, despite the fact that it is somewhat overloaded with too many abbreviations. The experimental section is harder to read as the motivations and its organization is not clearly stated. Overall, this section feels as if it is too hastily written, many results put into appendix without much discussion. The organization can be much more improved.
The disentanglement achieved by this novel representation is characterized only anecdotally and by contrasting the resulting objectives to a beta-VAE. It would have been much more informative to illustrate and discuss further the representations learned by such a conditionally Gaussian architecture. Figure 6 and 7 partially try to achieve this by showing the interplay of depth L and the inverse-dispersion parameter beta but I found it hard to interpret this results, for which an entire page is devoted.
In the experiments, the ELBO is reported for various methods. I would argue that the ELBO is not a very representative proxy for robustness. For example VAE ELBO and beta-VAE ELBO are both lower bounds of the true marginal likelihood and it is possible that beta-VAE is much lower while attaining a higher robustness in the sense of being resilient to suitably defined attacks.
The authors claim that there are no clear classification tasks for the datasets -- but this is not accurate as both celeb-a has clear classification tasks in the form of predicting attributes. It would have been really quite informative if adversarial accuracy on downstream tasks would have been reported. Relying on qualitative results in Figure 1 is only providing partial evidence about the approach.
Robustness to independent noise, as the authors have, is a good experiment to have -- however typical adversarial examples may be quite structured and such a randomized strategy may not give an accurate indication about the nature of the representation.
Overall, the paper is quite promising but I feel that one more iteration maybe needed.


Quantizing weights and activations of deep neural networks results in significant improvement in inference efficiency at the cost of lower accuracy. A source of the accuracy gap between full precision and quantized models is the quantization error. In this work, we focus on the binary quantization, in which values are mapped to -1 and 1. We introduce several novel quantization algorithms: optimal 1-bit, ternary, 2-bits, and greedy. Our quantization algorithms can be implemented efficiently on the hardware using bitwise operations. We present proofs to show that our proposed methods are optimal, and also provide empirical error analysis. We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed optimal quantization algorithms.

This paper proposes to quantize the weights of neural networks that can minimize the L_2 loss between the quantized values and the full-precision ones. The authors propose solutions for optimal 1-bit/ternary/2-bit quantization, as well as a greedy algorithm to approximate the optimal k-bit quantization.  Experiments are performed on image classification data set ImageNet using ResNet-18.
First of all, the authors should explicitly state in the paper that the optimality is in terms of what?  Indeed the authors obtain the solution of (1) quantization with a scaling parameter, and (2) in terms of minimizing quantization error (L_2 loss between the quantized value and the full-precision one), which is quite restricted to be a universally optimal one.
Since the number of weights and activations are limited in the network, it is not appropriate to formulate quantization error the weights and activations in the format of continuous distribution in (4) and (8).
It is also not clear to me why this paper begins with rank-1 quantization but ends up with scaled quantization. What kind of assumption are used in this approximation, and can the optimality still be guaranteed?
One of my main concerns is the novelty of this paper. Many of the solutions in the paper have already been discovered in literature. For instance, the optimal 1-bit solution in (5) was already obtained in Binary-Weight-Network [1] in 2016. The optimal ternary solution (i.e., the ternarization threshold should be 1/2 of the scaling parameter) in (12)  was also already obtained in Corollary 3.1 in  [2] in 2018, as a special case when curvature information is dropped.
Yet another concern is about the experiments. Since the proposed optimal binarization has the same solution as BWN, where does the performance gain in Table 2 come from? Moreover, some of the recent quantization methods are not compared. For instance, in PACT [3], 2-bit weight&acitvation quantization already achieves 64.4% top-1 accuracy of Resnet18 on ImageNet, while the proposed method achieves the same accuracy with full-precision activation and 2-bit weight (Table 2). In addition, according to Table 2, the proposed method also can not beat LQ-Net.
[1] Rastegari, Mohammad, et al. "Xnor-net: Imagenet classification using binary convolutional neural networks." ECCV, 2016.
[2] Hou, Lu, and James T. Kwok. "Loss-aware weight quantization of deep networks." ICLR 2018.
[3] Choi, Jungwook, et al. "Pact: Parameterized clipping activation for quantized neural networks." arXiv preprint arXiv:1805.06085 (2018).


Neural architectures for set regression problems aim at learning representations such that good predictions can be made based on the learned representations. This strategy, however, ignores the fact that meaningful intermediate results might be helpful to perform well. We study two new architectures that explicitly model latent intermediate utilities and use non-additive utility aggregation to estimate the set utility based on the latent utilities. We evaluate the new architectures with visual and textual datasets, which have non-additive set utilities due to redundancy and synergy effects. We find that the new architectures perform substantially better in this setup.

This paper studies non-additive utility aggregation for sets. The problem is very interesting. Choquet Integral is used to deal with set input. The authors propose two architectures. The two architectures, though not novel enough, are towards representing “non-additive utility”.
However, the experimental comparison is not fair, the description of the model (e.g. how Choquet is integrated into the model and help to learn “intermediate meaningful results”) is not clear, some claims are not true.
First, the authors claim that they are the first to combine Choquet integral with deep learning. However, there are a few, though not many, works in the literature trying to combine Choquet integral with deep learning. For example, “Fuzzy Choquet Integration of Deep Convolutional Neural Networks for Remote Sensing” by Derek T. Anderson et al.
Second, the authors claim they are using/motivated by Choquet integral, but do not have any (appendix) sections to explain how this mathematical tool is really integrated into their models. How do you guarantee that the representation learned by the neural network still obeys the property of Choquet integral? What is your loss or your algorithm? These need to be further clarified.
Third, the comparison to baseline and “DeepSet” is not fair. According to the illustration, it seems that you first obtain “features/representations”. Then the representations are fed to the four architectures you listed in figure one. RNN-based approaches are with better “complexity” comparing to your sum baseline and “Deepset” approach. So, I have some doubts about the experimental results.


Existing Lipschitz-based provable defences to adversarial examples only cover the 2 threat model. We introduce the first bound that makes use of Lipschitz continuity to provide a more general guarantee for threat models based on any p norm. Additionally, a new strategy is proposed for designing network architectures that exhibit superior provable adversarial robustness over conventional convolutional neural networks. Experiments are conducted to validate our theoretical contributions, show that the assumptions made during the design of our novel architecture hold in practice, and quantify the empirical robustness of several Lipschitz-based adversarial defence methods.

Summary:
This paper presents interesting theoretical contributions towards measuring the robustness of Lipschitz constrained neural networks. The empirical results are fairly limited and lack explanation in places and appropriate comparisons to existing work. The paper is missing references to several key pieces of related work tackling similar problems. I identified (fixable) issues with the theoretical results in the paper and felt that overall the paper was rushed and difficult to read in places as a result.
Further, this paper exceeds the recommended 8 page limit with ill-formatted references.
Overall
1) Claim that threat models beyond l2 has not been explored before is false. See Anil et al. for networks implementing an L-infinity Lipschitz constraint [1] and Huster et al. [2] for a discussion of infinity norm constrained architectures (the latter is cited).
2) In this work you claim through empirical evidence that adversarial training has only a small effect on the upper bound of the Lipschitz constant. This is contested by e.g. [3] which focused on measuring the Lipschitz constant of neural networks tightly and highlighted the impact of adversarial training.
3) Ignoring issues discussed below, this paper presents an interesting theoretical result on providing stochastic robustness guarantees for Lipschitz constrained neural networks.  However, the paper does not discuss existing baselines which provide similar stochastic guarantees e.g. [4].
4) There are several (fixable) issues present in the theoretical results. First, notation is inconsistent throughout with the loss function $l$ taking either one or two arguments in different places (e.g. Eqn. 2 vs. Proposition 1). In fact, in Proposition 1 the loss function on the left hand side of the inequality takes two arguments not one (one of which is vector valued).
From the proof, it is clear that the authors mean to make the loss a function of the margin (which should be discussed clearly). Further, Proposition 2 presents the wrong terms in the sum (so that E[L] != adversarial risk). L should instead consider the sum over the maximum change in the margin for the dataset and use Proposition 1 after Eqn 11 as described. Further, this proof is a straight forward application of McDiarmid's inequality and does not need to define a Doob martingale (in fact, it is not clear that one is defined here as no filtration is presented). I also believe that the presented bound is computed incorrectly, as the maximum change in the sum value through modifying a single element would be B/n . I believe the stated result can be made rigorous and correct, however in its current form there are mistakes.
5) I do not understand the proposed training strategy. It seems as though standard cross entropy training is used but with a modified Lipschitz constraint which allows the output of the vector-valued function to have different Lipschitz constants at each index. If this is the case, could you please elaborate on what you mean by the argument that subnetworks associated with each class can achieve higher accuracy?
6) In section 4.1 you introduce the methods use to constrain the Lipschitz constant of the network. The first of which is a regularization scheme which provides no provable guarantees on the Lipschitz constant itself. Further, you describe spectral normalization as a regularization technique which is inaccurate --- it is typically implemented as a projection. Spectral normalization is also unable to tightly enforce the required Lipschitz constant (see e.g. [1]). Finally, I would like to ask how the authors compute the Lipschitz constants of the networks used throughout the experiments. Is the product of the linear layer p-norms computed after training? If so, this would give a loose upper bound (see [3]).
7) I am also concerned by the range of perturbation size used to measure the theoretical robustness of the networks -- a maximum perturbation size on CIFAR-10 of 4/255 is used which is very small compared to e.g. [4] which searches up to a radius 1.5.
I would consider raising my score if the issues present in the theoretical results are addressed by the authors.
Minor:
- In introduction: "so-called adversarial examples, appear to humans as normal images" may be considered to strict a definition. For example, images which look like random noise can force high classification confidence in classifiers and are often described as adversarial examples. Similarly, the thread model defined in equation (2) is presented as the general adversarial objective but is only a special case. One should be also be concerned with threat models violating a p-norm constaint.
- Page 5, typo "This means the the"
References:
[1] Sorting out Lipschitz function approximation, Cem Anil, James Lucas, and Roger Grosse
[2] Limitations of the Lipschitz constant as a defense against adversarial examples, Todd Huster, Cho-Yu Jason Chiang, and Ritu Chadha
[3] Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks, Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George J. Pappas
[4] Certified Adversarial Robustness via Randomized Smoothing, Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter


Network representation learning (NRL) is a powerful technique for learning lowdimensional vector representation of high-dimensional and sparse graphs. Most studies explore the structure and metadata associated with the graph using random walks and employ an unsupervised or semi-supervised learning schemes. Learning in these methods is context-free, because only a single representation per node is learned. Recently studies have argued on the sufficiency of a single representation and proposed a context-sensitive approach that proved to be highly effective in applications such as link prediction and ranking. However, most of these methods rely on additional textual features that require RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identify multiple contexts of a node. In this study, without requiring additional features nor a community detection algorithm, we propose a novel context-sensitive algorithm called GAP that learns to attend on different parts of a node's neighborhood using attentive pooling networks. We show the efficacy of GAP using three real-world datasets on link prediction and node clustering tasks and compare it against 10 popular and state-ofthe-art (SOTA) baselines. GAP consistently outperforms them and achieves up to ≈ 9% and ≈ 20% gain over the best performing methods on link prediction and clustering tasks, respectively.

Summary:
The proposed paper adapts Attentive Pooling Network (ATN) for graphs, by noting that the order of
neighbors of any node in a graph does not matter unlike the neighboring words in a sentence (for
which APN was developed). A simple modification of removing the layer which encodes higher order
sequential properties in APN, eg n-gram like statistics, the APN is adapted to GAP. This allows
encoding context from the neighboring nodes for a specific pair to be compared. Applications on Link
Prediction and Node Clustering are demonstrated on three benchmark datasets.
Detailed comments:
- The paper is simple and easy to read and the modification of APN to GAP is also small while being
well motivated and empirically important.
- There are some confusions in the writing: p4 first paragraph says that "In principle one can learn
using all pairs of nodes, however that is not scalable, and hence we restrict learning between pairs
in E." However from and around Eq2 it appears that r_s.r_t^- is also computed, where (s,r^-) is not
an edge.
- Also the statement below Eq2 "The goal is to learn, in an unsupervised fashion..." is not correct
  specially for link prediction task, as having a training graph (with edges) is having annotations.

- Although, the removal of the `encoding' step from APN makes sense, it would also be possible that
  the encoding operation is learned to be an identity function automatically if the training data is
  presented appropriately. I would assume that learning in such a case might take longer, and might
  not achieve the performance achieved by GAP. An ablation experiment could be reported to support
  GAP further where encoder is kept in APN, but at the training same pair is presented multiple
  times with the order of the neighborhoods randomized.

- Similar to the experiment above, the order of the nodes could be arbitrarily fixed by some simple
  logic. Eg order the neighboring nodes sorted by the feature similarity with the current node. This
  can then be fed to APN and a baseline could be reported.
- I do not directly work with Link Prediction or Node Clustering tasks, so am not familiar with the
  literature and the baselines reported. However, it might be that many baselines do not use a
  discriminative objective. Perhaps just using the discriminative objective gives a lot of boost and
  using the context from neighbors is not very important? From Fig2 and Fig3a it seems that
  neighborhood size does not make a big difference? This should also be ablated. One experiment
  could be to use just the current feature and have a small MLP on it, and learn using the proposed
  discriminative objective. If any other experiment can be designed in similar lines, it should be
  included as well.
- Why are run time comparison given on simulated data and not real data?
- The experimental setup for training edge selection should be detailed more. In particular, I
  would recommend that true generalization in terms of nodes and edges should be ensured, by having
  no test node (i) appear in the train set and (ii) has an edge which connects it to a training
  node. If this is not used then the alternative scheme used should be explained and argued for.
I am not a direct expert in the area and felt that the paper was somewhat lacking. I am putting my initial rating
to the conservative side. I am very open to revising the rating based on the author responses and the other
reviewers' comments.


Modern machine learning models are often trained on examples with noisy labels that hurt performance and are hard to identify. In this paper, we provide an empirical study showing that a simple k-nearest neighbor-based filtering approach on the logit layer of a preliminary model can remove mislabeled training data and produce more accurate models than some recently proposed methods. We also provide new statistical guarantees into its efficacy.

This paper proposes a k-NN method for identifying corrupted labels, and then applies this k-NN in the representation space of a deep neural net rather than the original feature space. Overall the paper is well written and the results look quite convincing
The theory appears to be important (if somewhat straightforward-looking) contributions of existing k-NN theory to the corrupted labels setting, based on the key quantity the authors defined as S_k, the minimum k-NN spread.
Since the theory highly depends on this quantity, the authors should after Definition 2 justify why they chose to base their results around S_k, and why intuitively it is the right quantity.
- I encourage another experiment where the label corruption is not completely at random, that is it depends on the values of x itself.
- In particular I encourage a synthetic experiment where the authors look at corruptions with varying S_k (minimum k-NN spread) values, to empirically verify the their theory holds and this really is a meaningful quantity to consider in the label-corruption setting.
- Why don't the authors show the results for vanilla deep kNN trained on the full (noisy + clean) dataset in their experiments. This seems important to ascertain the benefits that might be attributed to simply switching to kNN.  Or is deep kNN generally worse that the original model trained on the full dataset?
- Why didn't the authors show the original model trained on the full dataset?
Is it because it always does worse than all the baselines considered in the paper?
I would expect it sometimes does much better than Control (eg. when noise rates are low), and this is the straightforward approach must practitioners would use.
- Why don't the authors present the accuracy of the k-NN method at identifying corrupted datapoints vs the other methods that aim to explicitly identify the corrupted datapoints?
In general, it seems the authors did not compare other filtering baselines, which would be more related to their method, for example:
Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
Northcutt et al. (2017). https://arxiv.org/abs/1705.01936
- It would be valuable to the scientific community if the authors can comment on:
Rolnick et al. (2018). Deep Learning is Robust to Massive Label Noise. https://arxiv.org/pdf/1705.10694.pdf
- Some similar looking ideas have been proposed in:
Gao et al. (2018). On the Resistance of Nearest Neighbor To Random Noisy Labels
https://pdfs.semanticscholar.org/4227/918020c15b719c415e93eb63d436583f1745.pdf
Parvin et al. (2010). A Modification on K-Nearest Neighbor Classifier.
https://globaljournals.org/GJCST_Volume10/7-A-Modification-on-K-Nearest-Neighbor-Classifier.pdf
so the authors should contrast their method/analysis against those papers.
- In Thm 1: "w.r.t. X" should be "w.r.t. x" (lower case)


Semantic parsing is the problem of deriving machine interpretable meaning representations from natural language utterances. Neural models with encoder-decoder architectures have recently achieved substantial improvements over traditional methods. Although neural semantic parsers appear to have relatively high recall using large beam sizes, there is room for improvement with respect to onebest precision. In this work, we propose a generator-reranker architecture for semantic parsing. The generator produces a list of potential candidates and the reranker, which consists of a pre-processing step for the candidates followed by a novel critic network, reranks these candidates based on the similarity between each candidate and the input sentence. We show the advantages of this approach along with how it improves the parsing performance through extensive analysis. We experiment our model on three semantic parsing datasets (GEO, ATIS, and OVERNIGHT). The overall architecture achieves the state-of-the-art results in all three datasets.

This paper proposes a reranking architecture with a LogicForm-to-NaturalLanguage preprocessing step for semantic parsing. The authors experiment their method on three datasets and get the state of the art results.
The proposed method is natural. But using neural models to rank (or rerank) is a long-existing technique, regardless of the chosen parametrization of the reranking model. This paper chose BERT. See section-2.6 of this tutorial for more details about using neural models to rank: https://www.microsoft.com/en-us/research/uploads/prod/2017/06/INR-061-Mitra-neuralir-intro.pdf.
Overall, I think the paper is not ready to publish for the following reasons.
1. The method relies much upon manual designs that seem hard to generalize.
By converting the logic forms to natural languages, the authors can leverage paraphrase datasets and pre-train the critic as a paraphrase model. However, the way they convert the logic forms is different for each dataset and they have to manually design rules for each logic form.
2. It is not clear how certain experimental designs were made.
The authors chose to not rerank if the candidates' scores are too low or high but close. Such choice and associated thresholds seem arbitrary: how were they actually found out? Were they tuned on a development set? How does the method work if the candidate with the highest score is always picked: in the end, this is what the model is supposed to learn, correct?
Other designs include beam size, whether or not to use a pretrained model, etc. How were such decisions made? Tuned on a development set?
3. The results are not sound enough. Given the issued pointed out in 1 and 2, I am not sure if the results are really sound as the authors claimed.
For example, what if the authors don’t use a LogicForm-to-NaturalLanguage conversion? What is the result if we directly learn to match input and logic forms?
Moreover, the authors better answer questions in 2 so I can gauge if their hyper-parameters were chosen in the principled ways. Once those are answered, a significant test had better be done since the improvement seems small.
4. Claiming Shaw et al. 2019 in table-3 as ``our methods’’ is wrong. It is clear that Shaw et al. (2019) didn't experiment on OVERNIGHT dataset, but setting up the baseline on a dataset should not be classified as ``our method’’.
Moreover, I have some comments on the model and experiments. These are not weakness, but I think some work in this direction may help improve the paper.
1. The model architecture should be better justified. In its current form, the two arguments (input query and output sequence translated from a logic form) are interchangeable. Why so? Why isn’t an asymmetric architecture more natural? How can the authors use a pair of logic forms as negative examples (in figure-2)? Why do the authors use the Quora dataset in particular?
2. The error analysis might be better to be a bit more quantitative. Its current form doesn’t seem to give insight on how the proposed method really helps. What the authors can do is: you can sample some sentences from the test/development set and count how many comparative words are misused in the original model, among which how many are corrected by reranking.


Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks. Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy. Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2018) can significantly improve validation accuracy on image classification tasks. However, this approach is not computationally practical for large-scale problems. In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss. The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization. In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network. Compared to AutoAugment, this leads to about 12× reduction in computing cost and 11× shortening in time overhead on ImageNet. We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art. On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model. On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

This paper proposes a technique called Adversarial AutoAugment which dynamically learns good data augmentation policies during training. An adversarial approach is used: a target network tries to achieve good classification performance on a training set, while a policy network attempts to foil the target network by developing data augmentation policies that will produce images that are difficult to classify. To train the policy network, each mini-batch is augmented multiple times with different policies sampled from the policy network. Each augmented mini-batch is passed through the target network to produce a corresponding training loss, which is used as the training signal for the policy network. Experimental results are shown for CIFAR-10, CIFAR-100, and ImageNet datasets, where Adversarial AutoAugment outperforms competing methods (AutoAugment and Population Based Augmentation) on a variety of model architectures.
In its current state, I would tend towards rejecting this paper. The overall structure, the figures, and the experimental results are very nice, but there are two major issues that are holding it back. First, I am skeptical that the policy network is actually learning useful policies. Secondly, there are many grammatical errors in the paper which hamper readability. Upon reading the first paragraph of the paper, my initial impression was already quite negative, simply due to the number of grammatical errors. Fixing these would strengthen the paper considerably, and I would increase my score accordingly if properly addressed.
Primary Concerns:
1) One of my main concerns with this paper is this line here: "To guarantee the convergence during adversarial learning, the magnitude of all the operations are set in a moderate range". Since the policy network has no incentive to select transformations that the target network can still learn from, I assume that the ranges are required so that the policy network cannot choose to apply extreme transformations which always fool the target network, such as setting brightness to 0 to make the entire image black. How are acceptable ranges determined? If cross validation is required, then this becomes very much like the original hand-tuning of data augmentations that we wanted to avoid in the first place.
Additionally, I think it would be useful to to see a plot of the magnitude of each transformation versus training epoch, similar to Figure 4a in [1]. If the policy network simply learns to use the most extreme augmentations available in order to fool the target network, then this may indicate that the gain in performance is from tuning the magnitude ranges, and not from the policy network selecting good policies.
2) The paper could benefit greatly from some revision of the grammar. I would recommend either having a friend or colleague read it over, or even using an automated grammar checking program, such as Grammarly. For example, in the first paragraph alone there are several sentences that could be improved:
"Massive amount of data promotes the great success" -> "Massive amounts of data have promoted the great success"
"when more supervised data available" -> "when more supervised data is available"
"or better data augmentation method adapted" -> "or a better data augmentation method is adopted"
"which can automated learn" -> "which can automatically learn"
"there still requires tens of thousands of GPU-hours consumption." -> "tens of thousands of GPU-hours of computation are still required"
Things to improve the paper that did not impact the score:
3) In the first paragraph it is claimed that data augmentation policies have weak transferability across different tasks and datasets. I do not fully agree with this claim, since papers such as AutoAugment [2] have shown that learned policies are highly transferable to new datasets, and augmentation strategies such as CutMix have been shown to be highly effective for a variety of tasks.
4) No citation for the original GAN paper, despite multiple mentions of adversarial learning, and GANs themselves.
5) There is no computation time comparison with PBA, which is about 1000x faster than Autoaugment, and therefore roughly 100x faster than Adversarial AutoAugment.
6) CIFAR-10 results are not necessarily state-of-the-art. The 1.36% error rate claimed by the paper is surpassed by work in [3], which achieved 1.33% using a combination of AutoAugment and mixup.
7) Citation for the CIFAR-10 dataset incorrectly refers to the Adam optimizer paper [4].
References:
[1] Daniel Ho, Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. ICML, 2019.
[2] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. "Autoaugment: Learning augmentation policies from data." CVPR (2019).
[3] Wistuba, Martin, Ambrish Rawat, and Tejaswini Pedapati. "A Survey on Neural Architecture Search." arXiv preprint arXiv:1905.01392 (2019).
[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015
EDIT: The authors have addressed the majority of my concerns, and as such I have increased my score from a 3 to a 6.


Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, the pre-trained language models are usually computationally expensive and memory intensive, so it is difficult to effectively execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we firstly propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large "teacher" BERT can be well transferred to a small "student" TinyBERT. Moreover, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT. TinyBERT 1 is empirically effective and achieves more than 96% the performance of teacher BERT BASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them.

This paper proposes a new knowledge distillation method for BERT models. A number of modifications to the vanilla knowledge distillation method of Hinton et al (2015) are proposed. First, authors suggest adding L2 loss functions between alignment matrices, embedding layer values and prediction layer values. Second, authors propose run knowledge-distillation twice, once with the original pre-trained BERT model as teacher, and then again with task specific fine-tuned BERT as a new teacher. Third, authors emphasize the use of data augmentation for successful knowledge distillation. In Table 2, authors claim a significant lift across GLUE benchmarks with respect to other baseline methods with comparable model size.
While the main contribution of this paper is the proposal of empirically useful techniques than theoretical development, the empirical results reported in this paper are somewhat puzzling.
First of all, GLUE benchmark scores reported in Table 2 don't seem to be consistent with Table 1 of Sun et al (2019) for BERT-PKD ( https://arxiv.org/pdf/1908.09355.pdf ) or DistilBERT ( https://medium.com/huggingface/distilbert-8cf3380435b5 ). Indeed, BERT-PKD in Sun et al seems to significantly outperform TinyBERT on QNLI (89.0 vs 87.7) and RTE (65.5 vs 62.9), and the gap between BERT-PKD and TinyBERT on other tasks are much smaller if we take numbers reported in the original paper.
In Table 6, ablation studies with different distillation objectives are reported. Quite surprisingly, without Transformer-layer distillation (No Trm) the performance drops quite significantly. This is unexpected, because baselines such as Sun et al and DistilBERT do not use the Transformer-layer distillation but much more competitive to full TinyBERT than TinyBERT without Transformer-layer distillation. Would there be a reason why TinyBERT is so critically dependent on Transformer-layer distillation? Similarly, the removal of data augmentation (Table 5, No DA) is so detrimental to the performance of the model that it makes me to suspect whether the most of gain is from successful data augmentation. Indeed, 'No DA' row of Table 5 is very close to the performance BERT-PKD in Table 4, although the number of layers is different (4 vs 6).
In order for the research community to understand the contribution of proposed techniques more thoroughly, I suggest authors to conduct ablation studies with the simplest baseline. That is, rather than starting with the full TinyBERT model, start with a simple but competitive baseline like BERT-PKD, and only add one technique (DA, GD, Transformer-layer distillation) at a time so that readers shall understand what technique is the most important to be added to the baseline, and also whether some of the proposed techniques should always be used in combination.
---
After Author Rebuttal: authors have addressed all of my concerns quite clearly. Additional experiments which targeted a specific design choice at a time made me much more convinced that the techniques proposed in this paper are useful not only for this particular context but also more broadly applicable.


Few-shot models have become a popular topic of research in the past years. They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class. Such models are trained on a wide range of classes and their respective examples, learning a decision metric in the process. Types of few-shot models include matching networks and prototypical networks. We show a new way of training prototypical few-shot models for just a single class. These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples. The difficulty here lies in the fact that no relative distance to other classes can be calculated via softmax. We solve this problem by introducing a "null class" centered around zero, and enforcing centering with batch normalization. Trained on the commonly used Omniglot data set, we obtain a classification accuracy of .98 on the matched test set, and of .8 on unmatched MNIST data. On the more complex MiniImageNet data set, test accuracy is .8. In addition, we propose a novel Gaussian layer for distance calculation in a prototypical network, which takes the support examples' distribution rather than just their centroid into account. This extension shows promising results when a higher number of support examples is available.

This paper looks at the problem of few-shot classification in the regime when only a single class is present. The task at hand is as follows: given a number of support images of a previously unseen class (not present during training) and a single unlabeled image, we need to decide if this image belongs to the class or not. While previous approaches would explicitly construct negative examples to contrast the positive ones with during training, the authors bypass this by using batch norm in the last layer, which, on average, centers embedding feature vectors at 0, defining effectively the embedding for the negative class. In addition, the authors look at modelling the distribution of support image embeddings to improve the performance of their model.
Overview:
I like the general idea and find the paper easy to follow. Using the centering effect of batch norm to effectively bypass the need to define the universal negative class embedding is an interesting approach. I think it is especially compelling due to the special role of the \vec{0} point in the embedding space, where all embedding vectors of length 0, regardless of their direction, meet. I also appreciate that the authors used multiple training runs and their averages as their results, rather than the best result.
I have a few points that I believe should be clarified:
-- Point 1 --
Are you assuming that the trainable offset in the batch norm will be close to \vec{0}, or are you identically setting it to \vec{0}? Since your definition of the “garbage” class relies on this, it might be worth checking what the actual learned parameters in the batch norm end up being, and if they are small, possibly fixing them to 0?
-- Point 2 --
I didn’t get what distance metric in particular you were using to determine if the unlabeled example X is closer to the centroid of the class c or the origin 0. If I understood it correctly, you effectively look at length(c-X) > length(X-0) as your condition determining if X belongs to c. What metric did you use as your length measure?
-- Point 3 --
When using the Gaussian approximation estimating the distribution of the support images in the embedding space, how does it affect the condition  length(c-X) > length(X-0)? In “Gaussian Prototypical Networks for Few-Shot Learning on Omniglot” by Stanislav Fort you cite, they incorporate the covariance of the Gaussian S in the length function, effectively using length(c,X,S). Their Euclidean metric uses the covariance matrix is a metric tensor to weight different axes differently. How exactly do you use yours?
-- Point 4 --
When comparing to the previous algorithms in the single-class (one-way) regime, you use a random image from one (at random) of the C_train - 1 remaining training classes as your negative example / example of the other class. Is that a fair comparison? How many images of those classes do you use? If it were the case that for a problem with the support size of 1 (or a few), sampling from these C-1 other classes might provide very unrepresentative embedding vector for the negative class. Might it be fairer to look at the average embedding vector over more images from the negative classes?
-- Typos --
= 2 typos on page 4 at the bottom: “*Is* is then taken into account *explictly* in the distance metric.”
= 1 typo on page 5 at the bottom: “standard deviation between runs is generally very low (¡.01). Each train”
-- Conclusion --
Overall, I think the problem and solution proposed are interesting. The authors provide a large enough set of comparisons to existing algorithms, and explain their approach well.


Machine learning (ML) algorithms, especially deep neural networks, have demonstrated success in several domains. However, several types of attacks have raised concerns about deploying ML in safety-critical domains, such as autonomous driving and security. An attacker perturbs a data point slightly in the concrete feature space (e.g., pixel space) and causes the ML algorithm to produce incorrect output ( a perturbed stop sign is classified as a yield sign). These perturbed data points are called adversarial examples, and there are numerous algorithms in the literature for constructing adversarial examples and defending against them. In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space representing the environment that produces input for the ML model. For example, an attacker can change the background of the image to be cloudier to cause misclassification. We present an algorithm for constructing SAEs that uses recent advances in differential rendering and inverse graphics. The generated SAEs are easily realizable in the physical world, and can be used to help the model adapt better to different domains.

This paper studies the problem of semantic adversarial attacks with a differentiable de-rendering and rendering pipeline. More specifically, this paper proposed a variant of FGSM (Goodfellow et al. 2015) and PGD (Madry et al. 2017) by extending the traditional Lp-bounded adversarial attacks in the rich semantic space. It considered a list of semantic parameters including color, weather, foliage, rotation, transformation, and object shape. To facilitate back-propagation and improve the quality of rendering, this paper re-implemented the differentiable equivalents of several image manipulation operations based on the previous work (Yao et al. 2018). For experimental evaluations, this paper selected the object detector SqueezeDet (We et al. 2016) as the target model for attack on the virtual KITTI dataset. Experiments demonstrated that the generated semantic adversarial examples (SAEs) can attack the SqueezeDet (see Table 1 and Table 2). By re-training with augmented data by the proposed method, the robustness of SqueezeDet (see Table 3) can be further improved.
Overall, this is an okay paper with incremental technical novelty and clear presentation. Reviewer has several concerns regarding the experiments.
(1) This paper only conducts experiments on virtual KITTI dataset, a synthetic benchmark for object detection and semantic segmentation. In general, studying the adversarial examples in the synthetic domain seems not a significant contribution. Reviewer would like to know the performance on the real dataset such as Cityscape (used in Yao et al. 2018) and other challenging indoor datasets such as ADE20K. At least, reviewer would like to know whether re-training on adversarial examples help to improve the performance on real dataset?
(2) The conclusion is largely based on the 1547 semantic adversarial examples generated, while there are more than 4K synthetic images in the dataset. This seems contradicts against the flexibility of generating semantic adversarial examples described in the paper (e.g., single parameter modifications). Reviewer suspects the proposed differentiable rendering pipeline is not very effective so that generating SAEs requires quite a bit exhaustive search over the parameter space. Please comment on the average running time for generating a semantic adversarial example. How does that compare to generating a pixel-based adversarial example?
(3) While several different quantitative analyses have been conducted, this paper only provides two examples as the qualitative result (see Figure 2). It would be more convincing if this paper provides more such examples in the appendix. In addition, ablation studies on semantic parameters are currently missing. Furthermore, reviewer wonders if it is possible to report the FID score and make sure the generated adversarial examples have the same distribution as ground-truth images.
(4) SqueezeDet is the only model used in the paper. Please also consider other models and report the attack performance and transferability. In a high-level, reviewer would like to know whether the proposed differentiable rendering method generalizes to other tasks including semantic segmentation and depth prediction.
(5) The following paper is related (see Figure 5 of MeshAdv paper), but not even mentioned here. Reviewer would like to see the comparison between the proposed method and the MeshAdv baseline.
-- MeshAdv: Adversarial Meshes for Visual Recognition. Xiao et al. In CVPR 2019.


Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -from deforestation, to human rights violations -that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image. Because the ground truth SR image is typically misaligned with the estimation SR image, the authors proposed to learn the shift with a neural network 'ShiftNet' in a cooperative setting with HighRes-net. The experiments were performed on the ESA challenge on satellite images, showing good results.
Overall, I found this paper interesting, and the method described is both clever and efficient. While some points need to be clarified, I am in favor of accepting this paper to ICLR.
Positive aspects:
- the paper is very clear and easy to read, with nice figures.
- a sensitivity analysis on many different parameters or types of inputs are made, which makes this paper an interesting research paper. For example, the tests on the type of reference image to stack at the beginning are very interesting.
- While I am not an expert on super-resolution, I do see a clever algorithm, that can be for example used with different number of input views.
- The end-to-end framework is also quite interesting as it allows to be spread easily across the very large satellite images users, with the code aldready publicly available.
- Lastly, the results are good wrt to the state-of-the-art, as the algorithm was proposed during a 2019 challenge and was in the top ones on the private leaderboard.
Remarks and clarifications:
- After looking at the challenge website, it is quite strange to find that the results values in Table 1, in terms of public and private scores, differ from the actual leaderboards, even if the metric is still the same. Thus, I was not able to understand (i) if the authors really participate to the challenge (ii) what method they presented during the challenge as 3 or 4 are shown here (iii) what was their ranking in the real leaderboard. It is important to be precise. From what I see, they might be second on the private score (the one that matters), if so it can be clarified also in the abstract where the word 'topped' was used.
- cPSNR metric : (i) can you please explain the acronym; (ii) it was the competition metric, so it means it is not 'we use' but 'the challenge used' - which will also show that it was not a choice to satisfy your results, but a standard metric.
- Lanczos interpolation: can you please explain in a small sentence in what this interpolation differs from the others?
- ShiftNet: what is this network? We only know tha it is adapted from HomographyNet, but we don't have any information about how it is composed. We just know from App. 1 that it has a very large number of parameters (34M). Why is that? Why a rigid registration needs such a large number of parameters?
- median anchor image: this is one of the interesting points of the paper. Can you please just clarify that you take a naive median image? Such as: for pixel (i,j), med(i,j)= med( LR1(i,j), LR2(i,j) ,...) .
- You are saying 'each imageset is padded to 32 views'. What do you mean? I thought your network was able to be used with different numbers of views. How did you pad your imageset?
- you list 'several baselines': for me, a baseline is something to compare to - usually, it is even something easy, such as the ESA baseline. But in your list, you are mixing baselines, other state-of-the-art methods (some of which you don't beat, so it's not a 'baseline'...), and your methods. It is very difficult to know which ones are yours. Please make different lists and/or identify yours in the table 1.
- What is the Ensemble method? It appears that in the paper, you explain the 'HighRes-net + shiftNet' method, but actually, the 'Ensemble' is the better one, while not clearly described. I don't understand what outputs are averaged - I thought HighRes-net and shiftNet were performed simultaneously.
- How did you select the hyperparameters of your model?
Scientific questions:
- Is it possible to have views of different sizes as inputs? Or views with missing parts?
- Usually, we have a high resolution image that is different in terms of type of acquisition (like a different type of satellite, but also true in medical images for ex.). Is your network ready for that? I mean: can we constrain the method even if it is not the same type of acquisition as ground truth? In that case, is that possible to have a super-resolution of the type of the input LR images?
Typos:
- is comprised of -> is composed of
- in Table 7, the bold number should be the beta=infinity as it is the best one. It will be clearer, even if of course, a good train score does not mean a good method because of overfitting.


Answering questions that require multi-hop reasoning at Web-scale requires retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question. This paper introduces a new graphbased recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Our retriever trains a recurrent neural network that learns to sequentially retrieve evidence documents in the reasoning path by conditioning on the previously retrieved documents. Our reader ranks the reasoning paths and extracts the answer span included in the best reasoning path. Experimental results show state-of-the-art results in three opendomain QA datasets, showcasing the effectiveness and robustness of our method. Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points.

Summary
========
This paper introduces a graph-based recurrent retrieval model for retrieving evidence documents in a multi-hop reasoning question answering task. The main idea is that (1) the graph formed by Wikipedia links between passages can be used as constraint for constructing reasoning chains, and (2) the joint encoding of the question and current passage can be used to retrieve a subsequent passage in the reasoning chain. The paper describes a model for implementing the above retrieval system, and how they jointly train with a reading comprehension model. They demonstrate the effectiveness of the system on HotPotQA, showing improvements over previously published models, and SQuaD-Open, showing competitive results.
Overall Comments
===============
The paper is an interesting, but incremental, improvement to the area of question answering. Overall, there are two main concerns about this work. First, while the results are somewhat strong, the ideas presented are small variations on existing systems. For example, Godbole et al 2019 and Ding et al. 2019 both explore using graphical structural to constraint iterative, multi-hop, retrieval. Also, Feldman et al 2019, describe an encoder based approach to encode question and paragraph context for iterative retrieval. Asides from smaller modeling differences (choice of RNN, training regime, BERT reader, etc.) to account for the difference in results, the main difference seems to be the joint training of the retrieval system with the reader. Secondly, the paper lacks clarity on some formal definitions and definition of the graph, making it hard to understand the content precisely.
Detailed Comments
================
Below are some detailed comments about specific parts of the paper, in order of importance:
1. One important limitation of this technique is the reliance on a linked documents for constructing the retrieval system. It is not clear from the paper how much of the results are obtained from constraining the set of retrieved passages (after the initial retrieval) to Wikipedia links. And whether, for example, substituting Wikipedia links with links derived from an off-the-shelf entity linking system would suffice.
2. Given that the retrieval model is restricted to link structure in Wikipedia that induces the proposed retrieval graph, I assume that there are “reasoning paths” that do not exist in the graph, given Wikipedia’s policy of avoiding adding redundant links within a Wikipedia page. It would have been informative to conduct an “Oracle” experiment: that is, given the initial  set of retrieved nodes and the graph structure, are there *any* paths that provide the correct answer and reasoning chain? That is to say, what is the upper-bound performance on the proposed system given the currently induced Wikipedia graph?
3. In Section 3, and even later on in the paper, it was not clear what “E” denotes. It never seems to be defined, and is used interchangeably with “graph node”, “wikipedia page”, “wikipedia paragraph” and “reasoning path”. Are these the same thing? It would be much clearer to define what E means, and perhaps separate the different concepts (node, passage, reasoning path) properly.
4. In Section 3, it seems that ‘q’ is not defined. Is it the question?
5. In Section 3.1, it is not clear what the graph actually contains. Does it contain all the paragraphs from Wikipedia? Just the paragraphs with links? The first paragraph of every Wikipedia page? What granularity of the wikipedia page becomes an individual node in the graph?
6. In Section 3.1.1., the representation of the starting retrieval (i.e., time-step = 0), h_0, is not defined. Later in the section, the paper mentions the use of TF-IDF for the initial set of nodes, instead of the learned retrieval model. This seems a bit unusual design decision without further explanation. Particularly when taking the results in Table 4, showing TF-IDF based retrieval performs worse that the learned retrieval system from the proposed model.
7. In Section 3, C_{t} (the candidate set of paragraphs) is not defined. This is an important set to define. Is it the set of paragraphs derived from Wikipedia links, starting from the current node?
8. In Section 3.1.2, “Loss function”, the term g_{r} is not defined.
9. In Section 4.4 “Analysis on reasoning path length”, it would have been useful to see the performance of the model with different path lengths. This analysis is somewhat common on multi-hop reasoning tasks, and should be included.
10. Typo in Section 4.4: “..., and out model is likely too terminate …”  should be “ likely to terminate “


We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages a strong language model to provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on a clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to a fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, we further improve the WER on the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self-supervised approach.

Overview:
This paper is dedicated to proposing a self-supervised objective, local prior matching (LMP), for speech recognition. This approach can take advantage of vase quantities of unlabeled speech data. What' more, the objective is simple to implement and theoretically well-motivated. In the paper, based on a supervised pretrained model, it then finetunes 360 hours with unlabeled data and LPM reduces the WER consistently. They also conduct extensive ablation experiments to show the effect of their self-supervised approach.
Strength Bullets:
1. I think this self-supervised learning objective (LMP) is very novel. The motivation that the source of indirect supervision on processing unlabeled speech comes from prior knowledge about the world and the context of the speech makes sense to me. The author combines the Bayesian method to build the model which is aligned with the motivation. They also provide clear and well-organized derivations.
2. The paper performs extensive ablation studies over all components, including beam size, mixing ratio, LPM weights, model update strategies, model initialization, length filtering and choice of language models. It provides convincing evidence of the effect of each component.
3. It provides interesting experiments results to study the relationship between the amount of unlabelled data for self-supervision and final performance. And LPM can surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data.
Weakness Bullets:
1. The paper only evaluates their method on LibriSpeech dataset. Although this dataset is popular, one or two more datasets will be more convincing.
2. For bayesian based methods, it is well known that it performs badly in high dimensional space. The reason is that we can not sample enough data points to obtain a good posterior estimation. Could you provide more analysis about the quality of posterior with different amount of sampled data?
3. For the experiment between the amount of unlabelled data for self-supervision and final performance, it would be better the author can provide a curve with more results.
Recommendation:
I think it is a good paper. The proposed approach is useful. This is a weak accept.


As the performance of computer systems stagnates due to the end of Moore's Law, there is a need for new models that can understand and optimize the execution of general purpose code. While there is a growing body of work on using Graph Neural Networks (GNNs) to learn static representations of source code, these representations do not understand how code executes at runtime. In this work, we propose a new approach using GNNs to learn fused representations of general source code and its execution. Our approach defines a multi-task GNN over low-level representations of source code and program state (i.e., assembly code and dynamic memory states), converting complex source code constructs and data structures into a simpler, more uniform format. We show that this leads to improved performance over similar methods that do not use execution and it opens the door to applying GNN models to new tasks that would not be feasible from static code alone. As an illustration of this, we apply the new model to challenging dynamic tasks (branch prediction and prefetching) from the SPEC CPU benchmark suite, outperforming the state-of-the-art by 26% and 45% respectively. Moreover, we use the learned fused graph embeddings to demonstrate transfer learning with high performance on an indirectly related algorithm classification task.

Using Deep Learning and especially, GNNs seems to be a popular area of research. I am no
expert at optimizing code performance, so please take my review with a grain of salt. The algorithmic contributions of the paper are as following:
(a) GNN that combines static code and dynamic execution trace.
(b) Binary encoding of features leads to better performance in comparison to categorical and scalar representations.
The results show that the proposed method outperforms existing methods on standard benchmarks in the program execution community.
From a machine learning stand point, the contributions are straightforward and the results make sense. I have the following questions:
(I) Authors argue that binary representations are better because of their hierarchical nature. They mention that they can generalize even if not all combinations of bits are seen, but a subset is seen in a manner that every bit has been flipped a couple of times. I don’t agree with this reasoning, as seeing the individual bits flip has no guarantee that a NN would generalize to a new combination of bits unless the distance in the binary code makes sense. Is there some special way in which the binary code is constructed?

(ii) Transfer learning experiments: Its unclear to me if the comparison presented in the paper is a fair one. Comparison is made against Ben-Nun et al. pre-training on LLVM IR. I am not sure how different is LLVM IR dataset from the algorithm classification dataset. If the dataset is very different, then obviously a lot of pre-training will only result in modest performance gain. What happens with Ben-Nun method is pre-trained on the same dataset as the proposed method? Also, what is the difference in performance between the cases when the proposed method is applied to algorithm classification with and without pre-training?
Overall, the paper is a application of GNN to optimizing code execution. The technical innovations are domain-specific and do not inform the general machine learning community. Given lack of expertise in the area of program execution, I cannot judge the significance of the performance improvements reported in the paper.
Given my current concerns, I cannot recommend acceptance. I might change my ratings based on the review discussions and the author’s responses to the above questions.


Stochastic gradient descent (SGD), which trades off noisy gradient updates for computational efficiency, is the de-facto optimization algorithm to solve largescale machine learning problems. SGD can make rapid learning progress by performing updates using subsampled training data, but the noisy updates also lead to slow asymptotic convergence. Several variance reduction algorithms, such as SVRG, introduce control variates to obtain a lower variance gradient estimate and faster convergence. Despite their appealing asymptotic guarantees, SVRG-like algorithms have not been widely adopted in deep learning. The traditional asymptotic analysis in stochastic optimization provides limited insight into training deep learning models under a fixed number of epochs. In this paper, we present a non-asymptotic analysis of SVRG under a noisy least squares regression problem. Our primary focus is to compare the exact loss of SVRG to that of SGD at each iteration t. We show that the learning dynamics of our regression model closely matches with that of neural networks on MNIST and CIFAR-10 for both the underparameterized and the overparameterized models. Our analysis and experimental results suggest there is a trade-off between the computational cost and the convergence speed in underparametrized neural networks. SVRG outperforms SGD after the first few epochs in this regime. However, SGD is shown to always outperform SVRG in the overparameterized regime.

This paper examines the tradeoffs between applying SVRG and SGD for training neural networks by providing an analysis of noisy least squares regression problems as well as experiments on simple MLPs and CNNs on MNIST and CIFAR-10. The theory analyzes a linear model where both the input $x$ and label noise $\epsilon$ follow Gaussian distributions. Under these assumptions, the paper shows that SVRG is able to converge to a smaller neighborhood at a slower rate than SGD, which converges faster to a larger neighborhood. This analysis coincides with the experimental behavior applied to neural networks, where one observes when training underparameterized models that SGD significantly outperforms SVRG initially, but SVRG is able to attain a lower loss value asymptotically. In the overparameterized regime, SGD is demonstrated to always outperform SVRG experimentally, which is argued to coincide with the case where there is no label noise in the theory.
Strengths:
I liked how the authors distinguished between the underparameterized and overparameterized regimes in the analysis and experiments. This allowed them to observe different behavior between the two regimes when comparing SVRG and SGD. I also found the authors' setting of analyzing noisy least squares problems to be interesting because of its potential usefulness for both analytically and empirically understanding certain forms of DL phenomena. The introduction is also well-written.
Weaknesses:
One aspect that I found unclear about the paper is its definition of the SVRG algorithm. In the analysis, the paper examines the expected risk least squares problem, and (if I understand correctly), considers the version of SVRG where the snapshot gradient is sampled i.i.d. over a large batch randomly from the true distribution. This is in contrast to the original SVRG method, which was designed for the empirical risk (or finite-sum) problem, where the set of datapoints is fixed. This coincides with the experiments, where the full training set is used to evaluate the snapshot gradient. Is this the correct interpretation of the theoretical and experimental results?
If so, how does this theoretical version of SVRG compare to a stochastic gradient method with large batch size? Does the theoretical behavior and insights exhibited by SVRG differ significantly from the theoretical behavior of SGD with larger batch size?
In addition, is the noisy least squares regression model with a diagonal data covariance equivalent to a separable quadratic problem? If so, it may not be surprising that the expected second moment of each parameter would evolve independently from each other, as noted at the end of Section 2.
I also found some of the theorems and proofs difficult to follow. This is partly due to some inconsistent notation: what is $B$ (vs $b$) (pg. 4)? Is $A = M$ (pg. 4)? What does $\circ$ denote in the exponents in the Appendix? What is the meaning of the constants defined in Definition 1? Some further explanation of the theoretical results (such as the meaning of those constants and more directly comparing the bounds for SVRG and SGD) would help with interpreting their results, particularly Theorem 4.
Along these lines, is it true that the rate of convergence for SGD is faster than the rate for SVRG? The constants made this difficult to tell, and no explanation was provided (although this was claimed in the Experiments section).
Most steps in the proof were also left unexplained, which made it difficult to follow without knowledge of certain properties of multivariate Gaussians. Some necessary assumptions were also missing from the definition of the model; in particular, the paper did not specify the relationship between $\epsilon_i$ and $x_i$ (which I assume are independent).
The experiments could also certainly be reinforced with some larger scale experiments on some larger datasets (such as ImageNet). One could see that the results became much more messy in the case of the underparameterized CNN on CIFAR-10 for example, and I wonder if this phenomena still holds with much larger datasets.
Some additional typos:
- Should use \citep for the Johnson & Zhang reference at the end of page 3
- SVRG Dynamics and Decay "R"ate in page 5
Overall, although the paper provides an interesting observation and direction in contrasting the underparameterized and overparameterized regimes when comparing SVRG and SGD for training DNNs, in my opinion, the paper needs some additional refining, particularly in terms of clarity with respect to the theory and notation, and perhaps some more experiments. If I understand the theoretical and empirical SVRG algorithms correctly, I'm not currently convinced that the paper provides substantially more theoretical insight than before due to differences between the theoretical and empirical SVRG methods applied in this paper and the theoretical algorithm's similarity to large-batch SGD. The observation in the underparameterized regime, for example, has been highlighted in prior work even with logistic regression (particularly due to the cost of evaluating a full gradient), and a theoretical comparison of small-batch SGD and large-batch SGD neighborhood results for strongly convex problems (see Bottou, Curtis, and Nocedal (2018), for example) would lead to similar conclusions. Because of these reasons, I do not recommend this paper for publication at this time.


In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class. Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes. While both few-shot classification and out-of-distribution detection are popular topics, their combination has not been studied. In this work, we propose tasks for outof-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets. Then, we propose two new methods for this task and investigate their performance. In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

This work investigates a new problem setting that combines few-shot image classification and out-of-distribution detection. The main procedure for the task still follows a standard few-shot classification task, but in each episode, the data from a different distribution may be presented together with the query images. The evaluation focuses on the performance of out-of-distribution detection, which relies on a scoring function to assign a high score for the normal query images while having a low score for the out-of-distribution images. The paper evaluates with three different scoring functions on four few-shot classification datasets and nine out-of-distribution datasets.
We recommend a weak rejection, although the proposed new problem setting might be interesting to the community, due to three major concerns. The first is insufficient description for reproducing the evaluation. For example, the text mentions “All results are evaluated using 1000 test episodes” without information of how the in-distribution data and out-of-distribution data (especially the OOS data) is chosen in each episode. Second, the experiments and discussions do not provide enough insights for readers to understand the impact of combining the two problems. Some questions should be addressed and are listed below. The last is the writing style which has a noticeable fraction of content not directly related to the proposed problem setting. Such a style can confuse readers and require additional passes of reading to understand.
Some questions to be addressed for the second concern:
1. What’s the impact of doing out-of-distribution under the few-shot setting? Is it harder than a normal out-of-distribution detection setting? How does the “N-way X-shot” setting affect the difficulty of the problem?
2. The paper proposes to use -MinDist and LCBO for the scoring functions instead of the methods that are commonly seen in a standard out-of-distribution detection paper. Why not use those previous methods (ex: ODIN, Mahalanobis, ensemble strategies, etc.) for the evaluation? If those previous methods do not work well with the proposed new setting, what are the possible causes?
Examples of the third concern include: (1) The MAML in Section 2 has a whole paragraph that could be summarized in a few sentences. (2) Figure 3 draws the schematics of out-of-distribution detection, but its connection to the proposed setting is not clearly described. (3) The introduction mentioned semi-supervised learning and continual learning, which does not strengthen the argument of why few-shot learning and out-of-distribution detection should be combined. (4) Lastly, the FS-SSL paragraph in the experiment section has no conclusion.


Deep neural networks have achieved great success in classification tasks during the last years. However, one major problem to the path towards artificial intelligence is the inability of neural networks to accurately detect samples from novel class distributions and therefore, most of the existent classification algorithms assume that all classes are known prior to the training stage. In this work, we propose a methodology for training a neural network that allows it to efficiently detect outof-distribution (OOD) examples without compromising much of its classification accuracy on the test examples from known classes. Based on the Outlier Exposure (OE) technique, we propose a novel loss function that achieves state-of-the-art results in out-of-distribution detection with OE both on image and text classification tasks. Additionally, we experimentally show that the combination of our method with the Mahalanobis distance-based classifier achieves state-of-the-art results in the OOD detection task.

This paper proposes to tackle the problems of out-of-distribution (OOD) detection and model calibration by adapting the loss function of the Outlier Exposure (OE) technique [1]. In OE, model softmax outputs are encouraged to be uniform for OOD samples, which is enforced through the use of a KL divergence loss function. The first proposed modification in this paper is to replace KL divergence term with an L1 penalty. The second change is the addition of an L2 penalty between the maximum softmax probability and the model accuracy. Experimental results demonstrate that adding these two components increases performance over OE on standard OOD benchmarks for both vision and text domains, and also improves model calibration.
Although this paper presents some good quantitative results, I tend towards rejection in its current state. This is mainly due to the limited comparison to alternative methods, and the lack of ablation study. If these were addressed I would consider increasing my score.
Things to improve the paper:
1) Currently, one of the most commonly used benchmark methods for OOD detection is the Mahalanobis distance based confidence score (MD) [2], which, as far as I am aware, is state-of-the-art among published works. The authors claim that they do not compare to this work because it is a post-training method, and, presumably, the techniques should be doubly effective when combined. However, we do not have any proof that this is actually the case. Therefore, I think it is important to verify that the two techniques are indeed compatible, and if not, then direct comparison with MD would still be necessary.
2) In the case of confidence calibration, there is no comparison made with other calibration techniques, such as temperature scaling [3]. I think it would be good to included these for reference.
3) Since two distinct components are being added to the loss function, I think it is important to include an ablation study to identify how much each component contributes to improvements in OOD detection and confidence calibration.
Minor things to improve the paper that did not impact the score:
4) With regards to the confidence calibration loss, there is similar work by [4] which also optimizes the output of the model to make sure confidence predictions are close to the true accuracy. It may be worth citing if you think it is relevant.
Additional questions:
5) How are lambda1 and lambda2 tuned? I could not find this information in the paper.
6) How sensitive is model performance to the setting of the lambda hyperparameters? It would be nice to see a plot of lambda versus the OOD detection metrics.
7) Have you evaluated how this method performs at detecting adversarial attacks? I do not think the paper will suffer without these results, but they are certainly relevant and of interest to practitioners in this area.
References:
[1] Hendrycks, Dan, Mantas Mazeika, and Thomas G. Dietterich. "Deep anomaly detection with outlier exposure." ICLR (2018).
[2] Lee, Kimin, Kibok Lee, Honglak Lee, and Jinwoo Shin. "A simple unified framework for detecting out-of-distribution samples and adversarial attacks." NeurIPS, (2018).
[3] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1321-1330. JMLR. org, 2017.
[4] Corbière, Charles, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. "Addressing Failure Prediction by Learning Model Confidence." NeurIPS (2019).
### Post-Rebuttal Comments ###
I would like to thank the authors for their hard work during the rebuttal period. I think the current version of the paper is much improved over the previous version. The choice to remove the claims about calibration definitely improves the focus of the paper. The addition of the  Mahalanobis distance experiments and the ablation study also significantly strengthen the paper. However, as the other reviewers have pointed out, the novelty of the paper is quite limited since the majority of the gains come from simply swapping the KL-divergence penalty with an L1 penalty. Despite simplicity, this single change yields a significant improvement in performance for the OE algorithm, which is noteworthy. As a result, I will increase my score from a weak reject (3) to a very, very weak accept (more like a 5 than a 6).


Batch Normalization (BN) is a well-known technique used in training deep neural networks. The main idea behind batch normalization is to normalize the features of the layers (i.e., transforming them to have a mean equal to zero and a variance equal to one). Such a procedure encourages the optimization landscape of the loss function to be smoother, and improve the learning of the networks for both speed and performance. In this paper, we demonstrate that the performance of the network can be improved, if the distributions of the features of the output in the same layer are similar. As normalizing based on mean and variance does not necessarily make the features to have the same distribution, we propose a new normalization scheme: Batch Normalization with Skewness Reduction (BNSR). Comparing with other normalization approaches, BNSR transforms not just only the mean and variance, but also the skewness of the data. By tackling this property of a distribution, we are able to make the output distributions of the layers to be further similar. The nonlinearity of BNSR may further improve the expressiveness of the underlying network. Comparisons with other normalization schemes are tested on the CIFAR-100 and ImageNet datasets. Experimental results show that the proposed approach can outperform other state-of-the-arts that are not equipped with BNSR.

The paper proposes to add an extra nonlinearity function in batch normalization, between the normalization and affine scaling. The nonlinearity is a power function x ** p for x >= 0 and - (-x) ** p for x < 0 (python pseudo-code), where p is a constant, and the authors propose to keep it fixed to 1.01. The intuition behind is reducing skewness of activations, and the modification is evaluated on CIFAR and tiny ImageNet datasets. The authors also experiment with the part at which to insert the nonlinearity.
I propose reject mainly due to insufficient experimental evaluation. The authors choose small datasets on which trained networks have large variance, and report a single accuracy value for each network, so it is not possible to judge the effectiveness of the method. Regarding the skewness reduction itself, it is not very convincing too, because the authors use a very small value for p (1.01), and because after affine layer ReLU removes most of the negative part, so perhaps the negative part of the nonlinearity is not needed.
I would suggest the following experiments to improve the paper:
 - try various values of p
 - try removing negative part of the nonlinearity
 - report mean+-std results of 5 runs for each experiment on small dataset
 - include experiments on full scale ImageNet
 - include more network architectures, or at least have more configurations of ResNet.
Would be very helpful if authors included code in pytorch or tensorflow for the proposed modification. It is surprising that such a small addition increases epoch time from 86 s to 119 s.
Also, there are some minor grammar mistakes. There is an undefined figure reference on page 6, and it is not clear what figures 6 and 7 are supposed to show, since the colors of histograms are never explained.


We consider the problem of domain generalization, namely, how to learn representations given data from a set of domains that generalize to data from a previously unseen domain. We propose the Domain Invariant Variational Autoencoder (DIVA), a generative model that tackles this problem by learning three independent latent subspaces, one for the domain, one for the class, and one for any residual variations. We highlight that due to the generative nature of our model we can also incorporate unlabeled data from known or previously unseen domains. To the best of our knowledge this has not been done before in a domain generalization setting. This property is highly desirable in fields like medical imaging where labeled data is scarce. We experimentally evaluate our model on the rotated MNIST benchmark and a malaria cell images dataset where we show that (i) the learned subspaces are indeed complementary to each other, (ii) we improve upon recent works on this task and (iii) incorporating unlabelled data can boost the performance even further.

In this work, the authors propose a domain invariant variational autoencoder for domain generalization problem. Specifically, the data is assumed to be constructed from three independent variables, one for the domain, one for the class and one for the residual variations. The method can be used for both unsupervised and semi-supervised cases. Experimental studies on rotated MNIST dataset and a malaria cell images dataset verify the effectiveness of the proposed method.
The paper is well-written and easy to follow. The proposed generative model is simple and technically sound. However, I have the following concerns.
(1)	It is not clear how the problem setting, i.e., domain generalization, matters in DIVA. In another words, the proposed DIVA is not specific to domain generalization problem, but can be used for domain adaptation, multiple source transfer learning etc. Actually, I find that the authors compare with DA, which is a conventional domain adaptation method, in the experiment. Moreover, the experimental setups in section 4.1.3 is a multi-source transfer setting, and DIVA can be well be applied. In this sense, I am not very convinced on the claim of the contribution that DIVA is proposed for domain generalization. For me, DIVA is a more general method.
(2)	With point 1, more related works on VAE for domain adaptation need to be discussed.
(3)	The idea of constructing data from disentangled latent variables is not new, see a latest work [ref1]. The main difference of DIVA from [ref1] is the residual variation variable. Two baselines are necessary for comparisons: (1) [ref1], and (2) DIVA without the residual variation variable. Actually, the semantic meaning of z_x is not well discussed. Even in the experimental studies figures 2 and 3, it is hard to tell what Z_x actually represents.
(4)	Regarding the 4.1.2, why DA is selected as a baseline? How does DA deal with the multiple domains? Can any other domain adaptation methods, e.g., [ref2], or multiple source transfer methods, e.g. [ref3], be compared?
(5)	In the right side of Table 1, the improvements seem to be very marginal considering the variance. This makes the ability of DIVA to use unlabelled data less convincing.
(6)	Regarding 4.1.3, it seems that the domain similarity plays an important role in the performance, comparing the results of M_{30} with M_{60}. Without the labelled M_{60}, which is very similar to the target M_{75}, the performance degenerates dramatically. The current DIVA treats all the domains equally, is it possible to have a weighted form of DIVA that distinguishes the contributions of different domains?
(7)	What is the task of the malaria cell images experiments? Is it to classify the parasitized and uninfected cells? For a given patient, it makes more sense that all the cells belongs to one category, either infected or healthy. How is the class distribution for a person (in this case a domain)? Is it very unbalanced?
(8)	For figure 3, It is hard to judge the cell images parasitized or uninfected without domain knowledge, can you give the label for each image? Again, the semantic meaning of Z-x is hard to tell. I am not convinced by the shape of the cell for Z-x.
(9)	For 4.2.2, why and how DA is compared? As far as I know, it is for unsupervised domain adaptation. Moreover, the improvements are quite marginal.
Some minor comments:
(1)	Page1, first para, 3rd line, “present” -> “presented”.
(2)	Page2, first para, 2nd line, “Y” - > “Y denotes”
(3)	Some references lack of page information
The paper should be self-contained. I would suggest the authors move some paragraphs in appendix to the paper, for instance, 5.1.1, 5.2.2, and 5.2.3.
Overall, the paper is presented with extensive empirical evaluations, but less theoretical justification. The significance of the paper is moderate as the key idea of learning disentangled latent variables has been studied, and the paper lacks of evidence to show the pure benefits of introducing Z_x as well as the comparison with the related work [ref1].
[ref1] Learning Disentangled Semantic Representation for Domain Adaptation
[ref2] Conditional Adversarial Domain Adaptation
[ref3] Multiple Source Domain Adaptation with Adversarial Learning