Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design three pre-training tasks: Masked Language Modeling (MLM), ImageText Matching (ITM), and Masked Region Modeling (MRM, with three variants). Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking. We also conduct a thorough ablation study to find an optimal setting for the combination of pre-training tasks. To answer this question, we introduce UNiversal Image-TExt Representations (UNITER), a largescale pre-trained model for multimodal embedding. We adopt Transformer (Vaswani et al., 2017)  as the core of our model, to leverage its elegant self-attention mechanism designed for learning contextualized representations. Inspired by BERT (Devlin et al., 2019), which has successfully applied Transformer to NLP tasks through large-scale language modeling, we pre-train UNITER through three pre-training tasks: ( As shown in Figure 1, UNITER first encodes image regions (visual features and bounding box features) and textual words (tokens and positions) into a common embedding space with Image Embedder and Text Embedder, then applies a Transformer module to learn generalizable contex-

# 1. Summary
The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).
I am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below.
Strengths:
* State-of-the-art results on several downstream vision-language tasks
* Empirical work to investigate different ways to perform conditioned masking

Weaknesses:
* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method.
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

# 2. Clarity and Motivation
The paper reads quite well, although some points need to be improved:
* "Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.
* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?
* Sec. 3.2 ITM: "an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?
* "The scoring function is denoted as s" -> please indicate in the main text what function you used
* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?
* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.

# 3. Novelty
The novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.
# 4. Experimentation
The main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.
The evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.


Original "male" Adversarial "female" Adversarial "female" Adversarial "female" Adversarial "female" Asymmetric dataset We analyze conditions on the data distribution under which the Bayes-Optimal classifier may be vulnerable to adversarial attacks and present realistic image datasets for which the BayesOptimal classifier can be calculated efficiently. When the data distribution satisfies certain asymmetries, the Bayes-Optimal classifier is vulnerable (b), but when the distribution is symmetric, the optimal classifier is robust and adversarial attacks are perceptually meaningful (c). Our experiments with these datasets show that CNN training consistently fails to find a robust classifier even when the optimal classifier is robust (d), while large-margin methods often succeed (e).
Adversarial attacks on CNN classifiers can make an imperceptible change to an input image and alter the classification result. The source of these failures is still poorly understood, and many explanations invoke the "unreasonably linear extrapolation" used by CNNs (Goodfellow et al., 2018) along with the geometry of high dimensions. In this paper we show that similar attacks can be used against the Bayes-Optimal classifier for certain class distributions, while for others the optimal classifier is robust to such attacks. We present analytical results showing conditions on the data distribution under which all points can be made arbitrarily close to the optimal decision boundary and show that this can happen even when the classes are easy to separate, when the ideal classifier has a smooth decision surface and when the data lies in low dimensions. We introduce new datasets of realistic images of faces and digits where the Bayes-Optimal classifier can be calculated efficiently and show that for some of these datasets the optimal classifier is robust and for others it is vulnerable to adversarial examples. In systematic experiments with many such datasets, we find that standard CNN training consistently finds a vulnerable classifier even when the optimal classifier is robust while large-margin methods often find a robust classifier with the exact same training data. Our results suggest that adversarial vulnerability is not an unavoidable consequence of machine learning in high dimensions, and may be the result of the specific distributions of commonly used datasets or of suboptimal training methods used in current practice.

This paper proposes studying adversarial examples from the perspective of Bayes-optimal classifiers. They construct a pair of synthetic but somewhat realistic datasets—in one case, the Bayes-optimal classifier is *not* robust, demonstrating that the Bayes-optimal classifier may not be robust for real-world datasets. In the other case, the Bayes-optimal classifier is robust, but neural networks fail to learn the robust decision boundary. This demonstrates that even when the Bayes-optimal classifier is robust, we may need to explicitly regularize/incentivize neural networks to learn the correct decision boundary.
The contribution of the two datasets (the symmetric and asymetric CelebA) is, in my opinion, an extremely important contribution in studying adversarial robustness and on their own these datasets warrant further study. Previously, all studies of this sort had to be done with small-scale classifiers and simplistic datasets such as Gaussians. The paper also definitively proves that there are realistic datasets where the Bayes-optimal classifier is non-robust, which goes against quite a bit of conventional wisdom in the field and opens up many new paths for research. However, there are a few (in my opinion) critical concerns that currently bar me from strongly recommending acceptance of the paper. I outline these below.
1. Prior work: the paper seems to ignore a plethora of prior work around studying adversarial robustness and understanding its roots. For example, a few very closely related works are as follows:
   - Adversarial examples are not Bugs, they are Features (https://arxiv.org/abs/1905.02175): Ilyas et al (2019) demonstrate that adversarial perturbations are not in meaningless directions with respect to the data distribution, and in fact a classifier can be recovered from a labeled dataset of adversarial examples. While not in conflict with this work, it does closely relate and discuss many of the same issues discussed in this work, so relating them would be fruitful.
   - A Discussion of Adversarial Examples are not Bugs they are Features (https://distill.pub/2019/advex-bugs-discussion/): Nakkiran (2019) actually constructs a dataset (called adversarial squares) where the Bayes-optimal classifier is robust but neural networks learn a non-robust classifier due to label noise and overfitting. Interestingly, they also construct a dataset where they Bayes-optimal classifier is robust and neural networks *do* learn a robust classifier (adversarial squares sans label noise). While I think the datasets presented in this work are much more interesting and certainly more realistic, this work should be put in context.
    - Excessive Invariance causes Adversarial Vulnerability (https://arxiv.org/abs/1811.00401v3): Jacobsen et al offers an explanation for adversarial examples based on the fact that NNs are not sensitive to many task-relevant changes in inputs, which seems to tie in nicely to the discussion in this paper, as under the presented setup the Bayes-optimal classifier will certainly exploit (and be somewhat sensitive) to such changes.
    - Adversarially robust generalization requires more data (https://arxiv.org/abs/1804.11285): Schmidt et al show a setup where many more samples are required for adversarial robustness than for standard classification error. And it seems to have very relevant connections to your work.
    - In general this list is not comprehensive either: there are many relevant connections to the robustness-accuracy tradeoff (https://arxiv.org/abs/1901.08573, https://arxiv.org/abs/1805.12152), and other works.
2. Discussion/interpretation of the results:
    - Sufficient vs necessary: While the experimental design and results are both of very high quality, I am slightly confused about the interpretation of the results. First, if my understanding of the paper is correct, the experiments show that (a) the Bayes-optimal classifier can be non-robust in real-world settings, and (b) even when the Bayes-optimal classifier is robust, NNs can learn a non-robust decision boundary. In particular, (b) indicates that it may be *necessary* to design regularization methods that steer NNs towards the correct decision boundary—it says nothing about whether these regularization methods will be *sufficient*, which the paper seems to suggest, e.g. in the abstract "our results suggest that adversarial vulnerability is not an unavoidable consequence of machine learning in high dimensions, and may often be a result of suboptimal training methods used in current practice." In fact, if real-world datasets end up being like the asymmetric dataset, then the results of this paper would actually indicate the *opposite* of the above statement. It is unclear on what basis one can say that real-world datasets are more like the symmetric case or the asymmetric case. I believe a more measured conclusion (perhaps that we *need* more regularization methods, but even then we may not be able to get perfect robustness and accuracy) would better fit the strong results presented in the paper.
    - CNN vs Linear SVM: I am confused about why we would expect a CNN to be able to learn the Bayes-optimal decision boundary but not the Linear SVM. The paper justifies the adversarial vulnerability of the Linear SVM by arguing that the Bayes-optimal classifier is not in the Linear SVM hypothesis class, which makes sense. The RBF SVM, for small enough bandwidth can express any function and is convex, so no argument needs to be made about its ability to find the Bayes-optimal classifier. For CNNs, however, it is unclear if the Bayes-optimal classifier lies in the hypothesis class (there are "universal approximation" arguments but these usually require arbitrarily wide networks and are non-constructive)—couldn't it be that the CNNs used here is in the same boat as the Linear SVM (i.e. the Bayes-optimal decision boundary is not expressible by the CNN?)
3. Experimental setup:
    - One somewhat concerning (but perhaps unavoidable) thing about the experimental setup is that all the considered datasets are not perfectly linearly separable, i.e. the Bayes-optimal classifier has non-zero test error in expectation, and moreover the data variance is full-rank in the embedded space. This is in stark contrast to real datasets, where there seem to be many different ways to perfectly separate say, dogs from cats, and the variance of the data seems to be very heavily concentrated in a small subset of directions. I am concerned that these properties are what drive the Bayes-optimal classifier for the symmetric dataset to be robust (concretely, if 0.01 * Identity was not added to the covariance matrix of the symmetric model and the covariance was left to be low-rank, then any classifier which was Bayes-optimal along the positive-variance directions would be Bayes-optimal, and could behave arbitrarily poorly along the zero-variance directions, still being vulnerable). This concern does not make the contribution of the symmetric dataset less valuable, but a discussion of such caveats would help further elucidate the similarities and differences of this setup from real datasets.
    - It is unclear if what is lacking from the NN is explicit regularization, or just more data. In particular, with such low-variance directions, at standard dataset sizes the distributions generated here are most likely statistically indistinguishable from their robust/non-robust counterparts (you can see hints of this in the fact that the CNN gets . While completely alleviating this concern may once again be quite difficult/impossible, it could be significantly alleviated by generating training samples dynamically (at every iteration) instead of generating a dataset in one shot and training on it. It would be very interesting to see whether these results differ at all from the one-shot approach here.
4. A suggestion rather than a concern and not impacting my current score: but it would be very interesting to see what happens for robustly trained classifiers on the symmetric and asymmetric datasets.
Overall, this paper is a very promising step in studying adversarial robustness, but concerns about discussion of prior work, discussion of experimental setup, and conclusions drawn, currently bar me from recommending acceptance. I would be more than happy to significantly improve my score if these concerns can be addressed in the revision and corresponding rebuttal.


We propose a novel inherently interpretable machine learning method that bases decisions on few relevant examples that we call prototypes. Our method, ProtoAttend, can be integrated into a wide range of neural network architectures including pre-trained models. It utilizes an attention mechanism that relates the encoded representations to samples in order to determine prototypes. The resulting model outperforms state of the art in three high impact problems without sacrificing accuracy of the original model: (1) it enables high-quality interpretability that outputs samples most relevant to the decision-making (i.e. a sample-based interpretability method); (2) it achieves state of the art confidence estimation by quantifying the mismatch across prototype labels; and (3) it obtains state of the art in distribution mismatch detection. All this can be achieved with minimal additional test time and a practically viable training time computational cost.

This paper presents a sample-based self-explaining method for image classification. The basic idea is adopt the attention mechanism to learn the relation between the latent representation of the query sample and training samples, and identify the training samples with higher similarity as the prototype. The classification decision is based on the label consistency between the identified prototypes (with the relation score in attention mechanism as the weight of different prototypes in determining the  label agreement)
The proposed model is intrinsically interpretable since the prototypes with higher weights can play as the decision explanation. And the authors have conducted experiments to show that such self-explaining mechanism based on attention model can achieve comparable classification accuracy with original black-box models.
The presentation of the paper is clear and easy to follow. But I have several concerns regarding the choice of prototypes and the evaluation of the interpretation:
1) According to Eq. (2), it seems that all training samples are used as the prototypes (but with different weights). Why not just use the top few prototypes? Would this such setting introduce a lot of noise, since many training samples are from different classes?
2) Since one focus of the paper is to provide interoperation of the classification model, some more experiments are needed to evaluate how well the interoperation is. For example, some crowdsourcing experiments to check if the provided prototypes can help human users correctly guess the model prediction.
3) I think the authors should also compare with the black-box model when we use the attention mechanism as a post-hoc interpretation. One straightforward baseline is that use the black-box model for classification, and pick the top "prototypes" with the highest similarity in the latent representation. Such comparison can help to validate if incorporating attention mechanism in the model design can provide better quality prototypes.


We focus on the problem of black-box adversarial attacks, where the aim is to generate adversarial examples using information limited to loss function evaluations of input-output pairs. We use Bayesian optimization (BO) to specifically cater to scenarios involving low query budgets to develop query efficient adversarial attacks. We alleviate the issues surrounding BO in regards to optimizing high dimensional deep learning models by effective dimension upsampling techniques. Our proposed approach achieves performance comparable to the state of the art black-box adversarial attacks albeit with a much lower average query count. In particular, in low query budget regimes, our proposed method reduces the query count up to 80% with respect to the state of the art methods.

This paper proposed a BO-based black-box attack generation method. In general, it is very well written and easy to follow. The main contribution is to combine BO with dimension reduction, which leads to the effectiveness in generating black-box adversarial examples in the regime of limited queries.  However, I still have some concerns about this paper.
1) The benefits of BO? It seems that the step of dimension reduction is crucial to make BO scablable to high-dimensional problems. I wonder if the gradient estimation-based attack methods can apply the similar trick and yield the similar performance. That is, one can solve problem min_{\delta} attack_loss( x, y, g(\delta) ) by estimating gradients via finite-difference of function values, where g(\cdot) is the dimension-reduction operator, and \delta is the low-dimensional perturbation. Such a baseline is not clear in the paper, and the comparison with (Tu et al., 2019) is not provided in the paper.
2) Moreover, in experiments, it seems that only query-efficiency was reported. What about distortion-efficiency for BO-based attack? For $\ell_\infty$-attacks, the other $\ell_p$ norms can be used as distortion metrics. I wonder what perturbation does the BO method converge to. It was shown in (https://arxiv.org/pdf/1907.11684.pdf, Table 1) that BO usually leads to larger \ell_1 and \ell_2 distortion.
3) It might be useful to show the convergence of BO in terms of objective value versus iterations/queries. This may give a clearer picture on how BO works in the attack generation setting.
4) Minor comment: In related work "Bayesian optimization has played a supporting role in several methods,
including Tu et al. (2019), where ...." However,  Tu et al. (2019) does not seem using BO and ADMM.
############ Post-feedback ##########
Thanks for the clarification and the additional experiments. I am satisfied with the response, and have increased my score to 6.


Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains RESNET on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches. Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings. Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and RESNET-50 training with very little hyperparameter tuning. In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance. By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1).

This paper proposes a learning rate adaptation mechanism, called LAMB, for large-batch distributed training. The goal is to stabilize the training as the batch size increases. The idea is simple and straightforward -- there should be a layerwise learning rate adjusted by normalizing the layer weights and gradients at each layer so that layers with larger weights take larger learning steps, and vice versa. The authors perform empirical studies on BERT-large and ResNet to conclude that LAMB can scale up training batch size while still being able to converge in time with comparable accuracy.
Strengths:
+ Demonstrate the scalability of large-batch training (up to 64K) on BERT-Large with comparable accuracy.
+ A leap from the prior work LARS that demonstrates the layer-wise learning rate adjustment scheme also works with Adam for NLP tasks.
+ The re-warmup technique for stabilizing the second phase of mixed sequence training is neat.
Weaknesses:
- Although the authors' analysis is based on a large set of models and clearly outperforms the prior work LARS, it is still hard to assess the generality of the obtained results. The authors made an effort to show evaluation results on MNIST and CIFAR-10, but they are much less challenging tasks.
- Technical novelty over LARS seems to be incremental, where a large portion of the work is essentially applying LARS to Adam and demonstrate its effectiveness on BERT and ResNet.
Overall, the LAMB technique seems to be simple to apply yet very useful in practice for large scale training. The work can potentially help the practitioner to scale-out large model training to hundreds or even thousands of GPUs/TPUs with good scalability. Moving forward, the authors are encouraged to report LAMB optimization results on transformer-based models such as GPT, RoBERTa, and XLNet.
Question:
Does the training take advantage of FP16 half-precision training?
How does the training process handle overflow and NaN gradients?
What is the significance of the range [α_l, α_u] in Theorem 2 and Theorem 3, and how to choose the value for them in practice?


Efficiently learning to solve tasks in complex environments is a key challenge for reinforcement learning (RL) agents. We propose to decompose a complex environment using a task-agnostic world graphs, an abstraction that accelerates learning by enabling agents to focus exploration on a subspace of the environment. The nodes of a world graph are important waypoint states and edges represent feasible traversals between them. Our framework has two learning phases: 1) identifying world graph nodes and edges by training a binary recurrent variational autoencoder (VAE) on trajectory data and 2) a hierarchical RL framework that leverages structural and connectivity knowledge from the learned world graph to bias exploration towards task-relevant waypoints and regions. We thoroughly evaluate our approach on a suite of challenging maze tasks and show that using world graphs significantly accelerates RL, achieving higher reward and faster learning.

This paper proposes a novel approach to hierarchical reinforcement learning approach by first learning a graph decomposition of the state space through a recurrent VAE and then use the learned graph to efficiently explore the environment. The algorithm is separated into 2 stages where in the first stage random walk and goal conditioned policy is used to explore the environment and simultaneous use a recurrent binary VAE to compress the trajectory. The inference network is given the observation and action and the reconstruction is to, given the hidden state or hidden state+observation, reconstruct the action taken. The approximate posterior takes on the form of a hard Kumaraswamy distribution which can differentiably approximate a binary variable; when the approximate posterior is 0, the decoder must reconstruct the action using the hidden state alone. The nodes of the world graph are roughly states that are used to reconstruct the trajectories in the environment. After the graph is constructed, the agent can use a combination of high-level policy and classical planning to solve tasks with sparse reward.
Personally, I quite like the idea of decomposing the world into important states -- it is closely related to the concept of empowerment [1] which the authors might want to take a further look into. I believe extracting meaningful abstraction from the environment will be a key component for general purpose RL agent. One concept I really like in the paper is using the reconstruction error as the reward for the RL agent, which has some flavors of adversarial representation learning. Further, I also really like the idea of doing structured exploration in the world graph and I believe doing so can help efficiently solve difficult tasks.
However, I cannot recommend accepting this paper in its current draft as there might be potential major technical flaw and I also have worries about the generality of the algorithm. My main concerns are the following:
    1. The ELBO given in the paper is wrong -- the KL divergence should be negative. I want to give the paper the benefit of doubts since this could be just a typo and some (very rare) researchers use a reverse convention; however, this sign is wrong everywhere in the paper including the appendix yet the KL between Kuma and beta distributions uses the regular convention. I tried to check the source code provided by the authors but the code only contains architecture but not the training objective, training loops or environments. As such, I have to assume that the ELBO was wrongfully implemented, unless the author can provide the full source code, or, if the ELBO is indeed incorrectly implemented, rerun the experiments with the correct implementation.
    2. The proposed method for learning the graph does not have to be a VAE at all. The appendix shows that the paper uses a 0.01 coefficient on the KL, which is an extremely small value for VAE (in fact, most VAE’s have beta larger than 1 for disentanglement). Thus, I suspect the KL term is not actually doing anything; instead, the main reason why the model worked might be due to the sparsity constraints L_0 and L_T. In other words, the model is simply behaving like a sequence autoencoder with some sort of hard attention mechanism on the hidden code, which might explain why the model still worked well even with the wrong ELBO. To clarify, I think this is a perfectly acceptable approach for learning the graph and it would still be very novel, but the manuscript should be revised accordingly to reflect this. If the (fixed) VAE is important, then this comparison (0 KL regularization) would be a nice ablation regardless.
    3. Algorithm 1 requires navigating the agent to the key points from \mathcal{V}_p. This assumption is quite strong. When the transition dynamic is deterministic and fully reversible like the ones considered in the paper, using the reverse of replay buffer can indeed take the agent back to s_p, but in settings where the transitions are stochastic or the transitions are non-linear or non-reversible, how should the algorithm be used?
    4. It is not clear how \mathcal{V}_p are maintained. If multiple new nodes are added every iteration, wouldn't there be more than necessary nodes in \mathcal{V}_p? It seems to me some pruning criteria were used unless the model converged within small number of iterations? Are the older ones are discarded in favor of newer ones?
    5. How are the actions sequences “normalized”?
    6. In what way are the Door-Key environment stochastic? It seems like the other environments also have randomness, so is the only difference the lava pool?
I believe the propose method is sound, so if the revision can address either 1 or 2, I am willing to raise my score to weakly accept. If the revision in addition addresses 3, 4, 5, 6 in a reasonable manner, I am willing to raise my score to accept.
=======================================================================
Minor comments that did not affect my decision:
    - I think mentioning the names of the environment in the abstract might be uninformative since the readers do not know what they are a priori.
Reference:
[1] Empowerment -- An Introduction, Salge et al. 2014


Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

In this paper, the authors investigate non-autoregressive translation (NAT). They specifically look into how using different auto-regressive translation (AT) models for knowledge distillation impacts the quality of NAT models. The paper is well organised and the experiments are sound and interesting, shedding light on an aspect of NAT models that's been rather dismissed as secondary until now: the impact of varying AT knowledge distillation.
First, so as to better please those out there who are more "state-of-the-art" inclined, I suggest the authors to better emphasise their improvements. Results obtained by their analysis can lead to improvements as stated in the last paragraph of Section 4. This could be better stressed in the introduction and it would make the main take-away messages from the paper stronger.
On a more general note, I would like to know how robust these models are. I understand this is a problem NAT and machine translation papers have in general, but I would still like to suggest the authors train each model from scratch multiple times using different random seeds, and report mean and variance of their results (i.e. BLEU). Although I understand that training each AT model 4 times and each NAT model 4x4 times (multiple times for each AT model trained) is unfeasible, you could still report mean and variance separately for AT and NAT, and simply choose one out of the 4 trained AT models per architecture to perform distillation. I recommend training each model at least 3 times and reporting mean BLEU and variance.
I would also recommend using other MT metrics (e.g. chrF, METEOR, BEER, etc.), since BLEU is a comparatively weak metric in terms of correlations with human judgements. For translations into German, look into characTER, chrF and BEER. For more information on the different metrics available, how they correlate with human judgements, and which better apply to different target languages / language pairs, please refer to the WMT evaluation/metrics shared tasks. [1]
I have a few general comments and suggestions:
- In Section 3.1, when introducing the NAT model (simplified version of Gu et al., (2019)), be more specific. I would like to see an actual description of what the modifications consist of in more detail: is there a sentence length prediction step? what happens when source length is different from source length? etc.
- In Section 3.2, right after Equation (3), the authors refer to "x and y" in an inverted manner, please fix it.
- In Section 4.3, you mention that two of the decoding methods used involve "random sampling", which I find misleading. You probably mean sampling according to the model distribution p(y_t | y_{<t}) for all t, which is not random. I suggest you simply remove the word "random" and mention that you use "sampling" and "sampling within the top-10 candidates". Also, when you sampling using the top-10 candidates, do you simply re-normalise the probability mass to include only the top-10 candidate tokens?
- As a suggestion, Tables and Figures could be more self-contained. There is always a compromise between conciseness and clarity, added by the fact that the page limit makes things even harder. However, I would recommend including more information in Table/Figure captions, especially in Figure 3 and Tables 3 and 4. Try to at least mention the training/evaluation data (dev or test?). Ideally one should understand it from reading the abstract and carefully reading the caption.
- In Appendix A, you mention that you select models according to validation loss. Is that really the case? If so, why? I am not sure whether validation loss (i.e. word cross-entropy on the validation set) should correlate so well with translation quality (or sentence-level metrics such as BLEU).
[1] http://www.statmt.org/wmt19/metrics-task.html


We make the following striking observation: fully convolutional VAE models trained on 32×32 ImageNet can generalize well, not just to 64×64 but also to far larger photographs, with no changes to the model. We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale the VAE-based 'Bits-Back with ANS' algorithm for lossless compression (Townsend et al., 2019) to large color photographs, and achieving state of the art for compression of full size ImageNet images. We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression results 1 .
of magnitude, resulting from vectorization.
We release an open source implementation based on 'Craystack', a Python package which we have written for general prototyping of lossless compression with ANS.

Summary:
This paper focuses on lossless source compression with bits back coding for hierarchical fully convolutional VAEs. The focus/contribution is three-fold: 1. Improve the compression rate performance by adapting the discretization of latent space required for the entropy coder ANS. The newly proposed discretization scheme allows for a dependency structure that is not restricted to a Markov chain structure in the encoder model q(z|x) and in the generative part of the model p(x,z). This is in contrast with bit-swap[1], which requires a markov chain structure. The dependency structure that is allowed in the proposed method is widely known to perform better than a markov chain structure, which can explain why it improves significantly over Bit-swap [1] (another hierarchical VAE compression algorithm that uses bits back coding.) 2. Increasing compression speed by implementing a vectorized version of ANS, and heaving an ANS head in the shape of a pair of arrays matching that of the latent variable and the observed variable. The latter allows for simultaneous encoding of the latent with the prior distribution and the image with the decoder distribution. 3. Showing that a model trained on a low-resolution imagenet 32 dataset can generalize its compression capabilities to higher resolution datasets with convincing results.
Decision: Accept.
This paper is clearly written, makes clear claims and supports these claims with convincing experiments. The contributions are of practical use and I expect future work to benefit from this paper.
Supporting arguments for decision:
The paper is well motivated; off the shelf compression algorithms such as PNG are also not trained on every dataset separately, and cross-dataset generalization is important if this model should be used in practice for many different images from different datasets and of different resolutions.
The paper clearly supports the main claims. It improves upon the previous bits-back coding-based hierarchical VAE [1]. The only hypothesis that is not checked is the one that hypothesizes that the lower bpd for higher resolution images is due to the lower ratio of edge pixels versus non-edge pixels, but this is not a dealbreaker from my point of view.
I would like the authors to revise their statement of state of the art compression performance on page 7 directly below table 2. “The fact that HiLLoC achieves state of the art compression rates relative to the baselines even under a change of distribution is striking, and provides strong evidence of its efficacy as a general method for lossless compression of natural images”. This is sentence should be made more nuanced as the proposed model only improves on Bit-Swap, but is still significantly outperformed by Local bits back coding (LBB [2]), and in the case of cifar-10 also by integer discrete flows (IDF [3]). On the other hand, it would be useful to still state that LBB is trained on every dataset separately, as well as IDF. Note also that in [3], a model that is trained on Imagenet32 and evaluated on the other datasets is also reported (see table 1 in [3]). It would be beneficial for the author to include the scores of this model, as the proposed method seems to perform slightly better at generalizing to new datasets.
Because of the buffer of initial bits required by bit-back coding, the compression/decompression of several data points has to be sequential if one wants to amortize this cost over several data points. Compression methods that don’t rely on bits-back coding, such as IDF [3], do not have this issue and can compress/decompress data points in parallel. Since this influences the practical usability of the model, it would be transparent to mention this.
My final main question is on the equivalence of evaluation methods of Bit-Swap and Hilloc on imagenet. The Bit-Swap paper states: “For MNIST, CIFAR-10 and Imagenet (32 × 32) we report the bitrates, shown in Table 5, as a result of compressing 100 datapoints in sequence (averaged over 100 experiments)...”. This means that Bit-Swap is not evaluated on the full test set of Imagenet 32 (as this contains 50000 images), as opposed to Hilloc. Do the authors think this is a problem?
Furthermore, in the case of “full” Imagenet, Bit-swap uses a subset of 100 images for evaluation and crops them to a multiple of 32 pixels in height and width, so that bit-swap can compress patches and the result is the average of patches for on image. Hilloc appears to take 500 random images and does not state anything about cropping. Could the authors comment on this?
Additional feedback to improve paper (not part of decision assessment):
- In the introduction, first paragraph: “ the method can achieve an expected message length equal to the variational free energy, often referred to as the evidence lower bound (ELBO) of the model. “ → “ the method can achieve an expected message length equal to the variational free energy, often referred to as the negative evidence lower bound (ELBO) of the model. “
- Section 3.2, last paragraph: It is not clear if in practice the latent and image are actually encoded in parallel as the author states that this is “in theory” possible.
- Page 4: “... we found that most of the compute time for our compression was spent in neural net inference, …” I assume you mean “inference” in any part of the encoder or decoder, and not specifically approximate inference of the encoder network. Perhaps clarify this to avoid confusion?
- Section 4: When referring to the ResnetVAE by Kingma et al, it would be appropriate to also cite [4], as this is very similar to resnetVAE’s and was released earlier.
[1] F. H. Kingma, P. Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICML), 2019.
[2] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with Flows via Local Bits-Back Coding. arXiv e-prints, 2019.
[3] Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer Discrete Flows and Lossless Compression. arXiv e-prints, 2019.
[4] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems (NIPS), 2016.


This paper proposes a novel generative model called PUGAN, which progressively synthesizes high-quality audio in a raw waveform. PUGAN leverages on the recently proposed idea of progressive generation of higher-resolution images by stacking multiple encode-decoder architectures. To effectively apply it to raw audio generation, we propose two novel modules: (1) a neural upsampling layer and (2) a sinc convolutional layer. Compared to the existing state-of-the-art model called WaveGAN, which uses a single decoder architecture, our model generates audio signals and converts them in a higher resolution in a progressive manner, while using a significantly smaller number of parameters, e.g., 20x smaller for 44.1 kHz output, than an existing technique called WaveGAN. Our experiments show that the audio signals can be generated in real-time with the comparable quality to that of WaveGAN with respect to the inception scores and the human evaluation.

The paper presents an approach based on generative adversarial models for the unconditional  generation of audio. The authors take inspiration from WaveGAN, to which they add more sophisticated upsampling blocks (called the bandwidth extension module) instead of transposed convolutions. They also propose to add a sinc convolution layer to the discriminators to improve training. Finally, they propose a progressive training scheme similar in spirit to the progressive training of GANs in images. Experiments are performed on generating audio pronunciation of digits, and the authors compare their work in terms of inception score, human evaluation and computation cost to WaveGAN.
As stated by the authors, one of the main motivation oof their approach is the reduction of computation cost. The motivation, expressed in terms of the 10ms "interactive rate" follows a good story. The measurements performed at the end of the paper show a ~ x2 performance gains compared to WaveGAN on CPU and about same running times on GPU, which is significant but not compelling.
Overall, I liked the story of the paper, but the paper lacks clarity and details. An important aspect of the paper is the progressive training, which is detailed nowhere (e.g., what is the stopping criterion to get to the next stage), should there be a special initialization of the last block of a new stage, etc.). The "Bandwidth extension module", which from my understanding is one of thee main contribution of the paper, is detailed in the appendix and comes essentially without justification. One of the main motivations of the paper is to be able to generate 44kHz audio, but the only results available at this resolution are inception scores that are below those of the 16kHz generation, which leaves open the question of whether the goal is effectively achieved.
I found the different versions of PUGAN difficult to read. The picture uses 2 blocks of bandwidth extension to generate 16kHz, whereas the evaluations are done with PUGAN-1 (1 block), which if I understand correctly is based on the lightweight WaveGAN that generates at 8kHz (whereas Section 4.1 suggests that 16kHz is generated from 4kHz generations by WaveGAN and two blocks). Also, the fact that evaluations are carried out with PUGAN-1 suggests that the progressive training does not really works well past a single block.
In the text it is suggested that spectral normalization and sinc conv on the discriminator is an "improvement" of WaveGAN, and an independent contribution of the article. While Table 2 clearly shows an improvement in terms of inception score, the human evaluation is not that clear: the accuracy of human labelers drops to 0.52 from 0.63 and the quality seems totally within the variance (2.7 +/- 1.5 vs 2.6 +/- 1.3). While Table 3 also shows clear improvement over the basic WaveGAN with the changes made by the authors (in particular in terms of win ratio vs PUGAN-1), the loss of accuracy should be discussed.
The performance obtained by PUGAN-1 is nonetheless noticeable -- +1 quality score over WaveGAN, and much better pairwise win ratios. Nonetheless, the paper lacks clarity and motivation for the exact form of the bandwidth extraction module, and does not fulfill its promises (importance of progressive training, high quality generation at 44kHz), so I am leaning towards rejection.


Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.

This paper introduces a new method to make ensembles of decision trees differentiable, and trainable with (stochastic) gradient descent. The proposed technique relies on the concept of "oblivious decision trees", which are a kind of decision trees that use the same classifier (i.e. a feature and threshold) for all the nodes that have the same depth. This means that for an oblivious decision tree of depth d, only d classifiers are learned. Said otherwise, an oblivious decision tree is a classifier that split the data using d splitting features, giving a decision table of size 2^d. To make oblivious decision trees differentiable, the authors propose to learn linear classifiers using all the features, but add a sparsity inducing operator on the weights of the classifiers (the entmax transformation). Similarly, the step function used to split the data is replaced by a continuous version (here a binary entmax transformation). Finally, the decision function is obtained by taking the outer product of all the scores of the classifiers: [c_1(x), 1-c_1(x)] o [c_2(x), 1-c_2(x)] ... This "choice" operator transforms the d dimensional vectors of the classifier scores to a 2^d dimensional vector. Another interpretation of the proposed "differentiable oblivious decision trees" is a two layer neural network, with sparsity on the weights of the first layer,
and an activation function combining the entmax transformation and the outer product operator. The authors then propose to combine multiple differentiable decision trees in one layer, giving the neural decision oblivious ensemble (NODE). Finally, several NODE layers can be combined in a dense net fashion, to obtain a deep decision tree model. The proposed method is evaluated on 6 datasets (half classification, half regression), and compared to existing decision tree methods such as XGBoost or CatBoost, as well as feed forward neural networks.
The paper is clearly written, ideas are well presented, and it is easy to follow the derivation of the method. As a minor comment, I would suggest to the authors to give more details on the EntMax method, as it is quite important for the method, but not really introduced in the paper. The proposed algorithm is sound, and a nice way to make decision trees differentiable. One concern that I have though, is that it seems that NODE are close to fully connected neural networks, with sparsity on the weights. Indeed, I think that there are two ingredients in the paper to derive the method: adding sparsity to the weights and the outer product operator (as described in the previous paragraph). In particular, the improvement over vanilla feed forward neural networks seem small in the experimental section. I thus believe that it would be interesting to study if both two differences with feed forward networks are important, or if only is enough to get better results.
To conclude, I believe that this is a well written paper, proposing a differentiable version of decision trees which is interesting. However, the proposed method relies on existing techniques, such as EntMax, and I wonder if the (relatively small) improvement compared to feed forward network comes from these. I believe that it would thus be interesting to compare the method with feed forward network with sparsity on the weights. For now, I am putting a weak reject decision, but I am willing to reconsider my rating based on the author response.
Questions to the authors:
(1) do you use the same data preprocessing for all methods (quantile transform)?
(2) would it make sense to evaluate the effects of each the entmax and the outer product operator separately in the context of fully connected networks?


Fully convolutional deep correlation networks are integral components of state-ofthe-art approaches to single object visual tracking. It is commonly assumed that these networks perform tracking by detection by matching features of the object instance with features of the entire frame. Strong architectural priors and conditioning on the object representation is thought to encourage this tracking strategy. Despite these strong priors, we show that deep trackers often default to "tracking by saliency" detection -without relying on the object instance representation. Our analysis shows that despite being a useful prior, salience detection can prevent the emergence of more robust tracking strategies in deep networks. This leads us to introduce an auxiliary detection task that encourages more discriminative object representations that improve tracking performance.

This paper investigates representations learned by Siamese trackers. The paper argues that existing trackers rely on saliency detection despite being designed to track by template matching in feature space. An auxiliary detection task is proposed to induce stronger target representations in order to improve tracking performance. Experiments are performed on VOT2018 tracking dataset.
The paper investigates an interesting and active research problem of stronger object representations for deep visual object tracking. However, the proposed solution of just integrating an additional detection task branch within the Siamese tracking architecture is naive. The main idea of integrating instance driven detection as an auxiliary task is borrowed from [1].  [1] also utilizes a Siamese architecture that is similar to the ones generally used in visual object tracking to localize particular instances of objects. Therefore, the novelty of the proposed tracking approach is limited.
Some recent works, such as [2, 3] have also investigated a similar problem of richer object representations for deep visual tracking. These approaches are desired to be discussed and empirically compared in order to fully validate the strength of the proposed approach.
The paper shows some qualitative analysis. However, most of it is limited to just few frames of an image sequence. Tracking datasets, such as VOT and OTB, provide additional analysis tools (i.e., attribute analysis) to thoroughly evaluate visual trackers. Such analysis is missing in the paper. For instance, the main argument of this paper is that current approaches rely on center saliency and likely struggle in the presence of occlusion. How does the proposed approach fare, compared to SOTA, on the subset of VOT image sequences that are labeled with occlusion?
On page 3, it is stated that "Our model uses a lightweight backbone network (MobileNetV2), and is somewhat simpler than recent state-of-the-art models  .....................  Although the model doesn’t outperform state-of-the-art, it attains competitive performance." The reviewer does not fully agree with this statement. A comprehensive empirical evaluation is crucial to fully access the merits of the contributions. State-of-the-art visual object trackers [2, 3, 4] achieve competitive tracking performance while being computationally efficient and fast. Therefore, a proper state-of-the-art comparison is desired to compare the proposed tracker with SOTA methods. Further, currently experiments are only performed on the VOT2018 dataset. The reviewer recommends to perform additional experiments on other large-scale datasets, such as TrackingNet [5] and Lasot [6] and compare the performance with SOTA methods that are also investigating the problem of richer object representations for tracking.
[1] Phil Ammirato, Cheng-Yang Fu, Mykhailo Shvets, Jana Kosecka, Alexander C. Berg: Target Driven Instance Detection. CoRR abs/1803.04610 (2018).
[2] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg: ATOM: Accurate Tracking by Overlap Maximization. CVPR 2019.
[3] Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte: Learning Discriminative Model Prediction for Tracking. CoRR abs/1904.07220 (2019).
[4] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan: SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. CVPR 2019.
[5] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, Bernard Ghanem: TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild. ECCV  2018.
[6] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, Haibin Ling:
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking. CVPR 2019.


Attention mechanisms have advanced the state of the art in several machine learning tasks. Despite significant empirical gains, there is a lack of theoretical analyses on understanding their effectiveness. In this paper, we address this problem by studying the landscape of population and empirical loss functions of attentionbased neural networks. Our results show that, under mild assumptions, every local minimum of a two-layer attention model has low prediction error, and attention models require lower sample complexity than models not employing attention. Additionally for the popular self-attention, our theoretical results provide several guidelines for designing attention mechanisms. Our findings are validated with satisfactory experimental results on MNIST and IMDB reviews dataset.

Summary: This paper endeavors to develop some theoretical understanding of why attention mechanisms improve the performance of neural networks. It includes some theory around the sample complexity of models which have a multiplicative mask in them, as well as a few related experiments.
Review: Having some theoretical analysis of the benefits of attention is definitely needed and I commend the authors for working towards this goal. The main issue that I have with the paper is that the model initially considered (introduced in section 2) is unrelated to attention mechanisms because the attention weights "a" do not depend on the input in any way. If I understand correctly, the ground-truth function y_i = f*(a* . x_i) assumes there is a single globally-correct attention mask a* for all examples. This is not attention. Every attention mechanism that I am aware of depends on the input in some way (in other words, instead of a*, you'd have a function a*(x_i) which changes according to the input). Some additional discussion is needed to explain why it is interesting to study this dramatically simplified (to the point of not being relevant) model. For example, if it was explicitly described as a preliminary for the a*(x_i) case (section 3.2 onwards) it would make more sense. Similarly, the assumption ||a_0|| ≤ s_0 is not true for most attention masks in pratice. There are some hard attention mechanisms which try to induce binary masks/hard sparsity but they are extremely rarely used in practice. The combination of these two assumptions makes for what appears to me (if I'm understanding correctly) a somewhat trivial result that the "attention mechanisms constrain the parameters in a smaller space". Note that this criticism does not apply to section 3.2 where the attention mask is specified as a function of the input. Similarly section 4.2 relies on the "single attention mask" idea, which is the sole result considering generalization in the paper. Finally 5.1.1 consists of an experiment which assumes a model of the form G*(x . a*), where a* is a fixed random binary mask. Training a model to learn G* and a* has very little to do with attention - it is a learning problem where you have removed a subset of the features and are trying to determine the parameters of the function and which features were removed. The L_1 penalty on "a" is certainly a good way to learn to only include a sparse subset of your features, but again has very little relevance to most attention mechanisms (i.e. I know of almost no work which regularizes the attention mask in this way).
Overall I think most of the results in this paper are not relevant to attention models and as such it has limited impact. I would suggest reframing the paper around a focus on what the authors call the "self-attention" case (which is really just normal attention), possibly using the fixed a* case as a preliminary/motivation. I also would suggest removing the discussion of regularization the attention mask as I am not aware of this being used in practice. Finally I would suggest some more direct discussion of how the results apply to models which are used in practice.


Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the highdimensional latent code generated by the model, which needs to be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs), InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline while yielding comparable likelihood scores and reducing the NFEs on CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. Additional details at https://sites.google.com/view/infocnf-iclr/

This paper examines the problem of extending continuous normalizing flows (CNFs) to conditional modeling. The authors propose a model, InfoCNF, which models a latent code split into two partitions: one that is unique to a class, and another that is more general. InfoCNF relies on the accuracy of ODE solvers, so the paper also proposes a method that learns optimal error tolerances of these solvers. They perform experiments on CIFAR10, showing that InfoCNF outperforms a baseline in accuracy and negative log-likelihood. They also ablate through experiments that demonstrate the utility of learning the ODE error tolerances.
CNFs are an exciting tool, and it's an important problem to devise methodology that applies CNFs to conditional modeling tasks. The idea of splitting latent codes into two separate components -- one supervised, the other more general -- is interesting. And the approach to learning error tolerances is a good idea.
The main drawback of this paper is the lack of clarity. It is poorly written and the presented model is not clearly motivated. Even after reading through the paper multiple times, I find it difficult to understand various aspects of InfoCNF. Below are some examples of this lack of clarity:
- When motivating Conditional CNF (CCNF), the details for training the model are unclear. What is the loss function, and how does it balance between modeling x and learning the auxiliary distribution? Although these may seem like small details, since InfoCNF builds off on CCNF, it is crucial to solidify an understanding of the training procedure and how the auxiliary task relates to modeling x. Moreover, the cited reference (Kingma and Dhariwal, 2018) does not contain these details (and there is no mention of object classification in the paper, contrary to the claim on page 3). It would be helpful to cite other references when mentioning that this approach is widely used.
- The definition of InfoCNF is unclear. The variable y has been used to denote the image label, so why are there now L latent variables y_1, ..., y_L? The following terms of equation 4 are undefined: L_{NLL}, L_{Xent}, and y_hat. Although some readers would be able to understand these definitions from context (flow-based NLL, cross entropy loss, and the logits provided by the auxiliary distribution q), they are never explicitly defined and result in confusion and ambiguity. Most importantly, p(x|z,y) is never made explicit; although one can infer from context that  is transformed to x using CNFs, it is never made explicit in the definition (and that these flows only appear in L_{NLL}). Overall, the motivation for splitting the latent code into two pieces is not clearly explained, and the paper should spend more time arguing for this.
The paper compares InfoCNF to a single baseline (CCNF). I understand that the paper is proposing the first method that combines CNFs with conditional modeling, but there are plenty of non-CNF flow-based conditional approaches that could've been compared. The paper goes over these approaches in Section 5 and discusses their differences with InfoCNF, but these are never experimented with. It seems possible that these models could be extended in a straightforward manner to use CNFs instead of other normalizing flows. Even comparing with these models without CNFs would have been interesting. I think that using a single baseline, instead of applying a more complete set of comparisons, hurts the persuasiveness of their method.
In addition to comparing to a single baseline, the paper only compares for a single dataset (CIFAR10). A more convincing experiments section would compare against more models on more than a single dataset.
The paper proposes a promising idea to an important problem. Due to the lack of clarity throughout the paper and incomplete comparisons, I would argue to (weakly) reject.


We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB. This representation enables neural modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically large KBs. The sparse-matrix reified KB can be distributed across multiple GPUs, can scale to tens of millions of entities and facts, and is orders of magnitude faster than naive sparse-matrix implementations. The reified KB enables very simple end-to-end architectures to obtain competitive performance on several benchmarks representing two families of tasks: KB completion, and learning semantic parsers from denotations.

The paper provides a way to represent symbolic KBs called sparse matrix reified. Relations and entities' types are modelled using sparse matrices. This modelization allows distributing the computation on many GPUs. A neural model is used to manage these matrices. The trained model can be used to perform multi-hop queries. The proposed system has been compared with related systems. The results show that it achieves hits@1 that are comparable with the others.
The proposed approach seems promising, however, I feel that the paper is not ready for publication. The experiments lack a scalability test with related systems, the scalability test included in the paper only takes into account other definitions of the system. Also, comparison on the run time should be performed to see if the lower performance in terms of hits@1 (which are good anyway) is balanced by a better run time.
Therefore, it is difficult to see whether the proposed system is good or not.
As for the description of the system, it seems to me to be quite foggy. In my opinion, the neural model should be better described to show how the sparse matrices are mapped into the model.
Also, how are the training and testing sentences created? How would the output of a query look?
After reading the paper I have the feeling that it was written in a bit of a hurry, without working on the details. There are several typos, parentheses not correctly opened or closed, out of place commas that make some sentences seem unrelated to the rest of the paragraph. However, here last problems are easily fixable.
To sum up, I am conflicted about the choice of the final score. On the one hand, the approach is interesting, on the other hand the article seems to me not mature enough and strong enough from the point of view of organization, contents and experimental results shown.
Minor problems
On page 4, the size of the matrices XM_k uses the factor b that is introduced later.
In the introduction, next to footnote 1, I suggest specifying that A is the first query, the one about Tarantino's movies.
Also, his surname is misspelt throughout the paper. The correct one is Tarantino.


Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve. We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks, including single-speaker speech generation, multi-speaker speech generation, and music generation.

The authors introduce MelNet, an autoregressive model of Mel-frequency scaled spectrograms. They convert audio into high resolution spectrograms to reduce the audio artifacts introduced by inverting spectrograms (here they use gradient-based inversion over Griffin-Lim). To improve modeling of long term dependencies, they perform multi-scale splitting of the spectrograms and maximize the likelihood at each scale (avoiding dominance of noise at higher resolutions). They condition generation at finer scales from coarser scales, enabling sampling through an ancestral process. The authors also highlight the difference between temporal and frequency dimensions, creating different conditioning stacks for the past in time vs. the "past" in frequency (lower frequencies), and mixing conditioning between the two stacks through layers of the network. Multilayer RNNs are used throughout the network and external conditioning is incorporated at the input.
The challenge the authors are attempting to address is modeling of audio structure on both long and short timescales. As the authors demonstrate with strong baselines, WaveNet models, while superior on fine-scale fidelity, fail to capture dynamics more than a couple hundred milliseconds. The experiments demonstrate improvements on state-of-the-art for unconditional generation on text-to-speech datasets (generating coherent words and phrases) and the MAESTRO piano dataset (generating sections with consistent dynamics/timing/motifs). The continuations of primed examples in both domains are particularly impressive qualitatively, as they maintain much of the character of the priming sample. Ablation experiments qualitatively demonstrate the importance of multi-scale modeling for unconditional generation. Human listener studies support the claims made from qualitative evaluation of long term structure.
This paper should be accepted because it represents a non-trivial adaptation of autoregressive modeling to handle multi-scale structure in audio. The baselines comparisons and strong, and experiments validate the claims of the paper.
That said, several things could be done to improve the clarity and significance of the paper.
* While the network architecture is described in detail and some figures, the full network structure itself is non-trivial and still somewhat opaque from the plain text description. A schematic diagram of the full network architecture, even in the appendix, could help clarify how many layers are present connecting each component of the model, which would improve reproducibility.
* The paper is a bit thin on metrics. Human listening studies compare long-term structure, but not short-scale fidelity. For TTS, there are clear artifacts from the spectrogram inversion process. Mean opinion scores on conditional samples could help to quantify the importance of each element of the network for audio quality. For instance, how does MOS compare between Griffin-Lim MelNet, Gradient Inversion MelNet, and WaveNet? How does MelNet compare to Linear scaled spectrograms?
* Generating MelSpectrograms to model long-term structure is a fairly established technique, most notably employed by all of the Tacotron variants (https://google.github.io/tacotron/). These models are perhaps a more appropriate comparison for MelNet in many ways, and opt for spectrogram inversion by smaller WaveRNN models. One of the claims of the paper is that it is important to model the fine-scale structure of spectrograms, but it is not clear if that really is the case. A proper comparison to Tacotron models (where spectrograms are generated at the same resolution / the same inversion methods are used) would help clarify the importance of end-2-end training, vs. the learned inversion approach.


Humans understand novel sentences by composing meanings and roles of core language components. In contrast, neural network models for natural language modeling fail when such compositional generalization is required. The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance. Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models. Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

Summary
---
(motivation)
Consider SCAN, a synthetic task where setences like S1="jump twice and run left" are supposed to be translated into action sequences like A1=JUMP JUMP LTURN RUN. One might replace the word "jump" in S1 with "walk" then translate to get A2=WALK WALK LTURN RUN. If instead S1 is translated into A1 and then the action JUMP is replaced with the action WALK then we should still get the same A2. Such a translation model is equivariant to permutations of "jump" and "walk".
This paper aims to
1) define a general notion of compositionality as equivariance,
2) build a model which is compositional in this general sense, and
3) apply the model to SCAN.
(approach - theory)
This work considers this kind of compositionality as equivariance to group actions. Previous work (Kondor & Trivedi, 2018) viewed convolution as equivariance to actions by translation groups. This work views language compositionality as equivariance to actions by permutation groups applied to a set of similar words (e.g. verbs in SCAN).
(approach - model)
The paper proposes G-Embed, G-RNN, G-Attention, and G-Conv (not new) layers that are equivariant to word permuatations (e.g., switching "jump" and "walk"). It then composes these modules in a fairly standard fashion to build a new G-seq2seq model which is invariant to group actions.
(experiments)
Experiments apply a G-seq2seq model to the SCAN tasks, comparing to strong baselines. G-seq2seq requires slightly more knowledge (a set of related words like verbs) than all the baselines, but less knowledge than Lake 2019.
1. G-seq2seq outperforms all baselines except Lake 2019 (unfair comparison) on basic compositional tasks ("Add Jump" and "Around Right").
2. Like other models, G-seq2seq fails on the "Length" task, though it is still among the best performers.
Strengths
---
The theory of compositionality as invariance to actions by permutation groups is new, interesting, and could turn out to be significant.
The proposed models are also new, interesting, and could be significant.
Experiments on SCAN verify that the proposed models work about as expected, sometimes beating strong baselines in the process.
Weaknesses
---
It's hard to know what the impact of this paper will be because 1) it's unclear whether this model can generalize to more useful domains and 2) the presentation may turn some readers away. While neither of these issues can really be solved, I think paper could be substantially better in both aspects. Corresponding suggestions:
1) How expensive is this? It seems quite expensive because the representation size scales with the number of permutation of the set of words equivariance is with respect to. How will it scale to larger problems in terms of computation/memory costs (especially larger vocab sizes)? What knowledge is required for applying this method to new domains--i.e., how do I choose a set of permutation equivariant verbs in general? More discussion of these issues may help increase the impact of the paper.
2) See next section.
Presentation Weaknesses / Points of Confusion / Missing Details
---
To mimic a typical decoder RNN there should be another input which copies the word \tilde{w}_{t-1} from the previous iteration as input, somehow fused with the attention feature \tilde{a}_t. How does the G-RNN know what the last word it generated was?
The notation $g^{-1} w$ in the first equation of section 4.1: I think $\psi^i$ is supposed to take an integer as input but $g^{-1} w$ is a permutation applied to a function. I'm not sure how to apply permutations to functions like w and it doesn't seem like the output should be an integer in any case so I find this notation confusing.
Taking a step back, I find some of the notation (e.g., previous point) a bit confusing. This makes it hard for me to get the main point. I think the idea is that equivariant models can be achieved by tracking a representation (e.g., via rows of the G-Embed matrix) for (almost?) every member of the acting group.
It may help the presentation to more frequently demonstrate the general concepts with examples, though doing so may be in conflict with the general nature of the paper's theoretical contribution. I'm sure this is a familiar tradeoff, but from my perspective the paper would probably be more impactful if the presentation leaned more on examples.
Equation numbers would be a really great addition. I found it hard to reference some of the material in writing my review.
"and the use of algebraic computation"
* This seems specific to the chosen example whereas the rest of the sentence is trying to be general.
Suggestions
---
* This seems related to [1], which uses group theory to define a notion of disentangled representation similar to compositionality. That may inspire future work and would be useful to mention in the related work.
* Why didn't performance on SCAN get to 100%? It would be useful to spend some time addressing points of failure for the model other than compositionality.
* The G-RNN doesn't have a bias. It's not necessary, but it may be interesting to describe why this design choice was made.
[1]: Higgins, Irina et al. “Towards a Definition of Disentangled Representations.” ArXiv abs/1812.02230 (2018): n. pag.
Preliminary Evaluation
---
Quality: The theoretical contributions make sense and the experiments show they lead to useful models.
Clarity: The technical parts of the paper are somewhat unclear, but the rest of the paper is well written.
Significance: As discussed in the Weaknesses section this could turn out to be very significant or not significant at all, but that's true for a lot of good research.
Originality: The general notion of equivariant neural networks and good performance on SCAN are novel.
Overall, this is a very clear accept.
Post-Rebuttal Update
---
There was a lot of agreement between reviewers, though we came to slightly different conclusions about ratings. Though there is significant uncertainty about the impact of this work, I still think 8: Accept is the most appropriate rating. Overall, the other reviews and author responses only increased my confidence that this paper should be accepted.


Learning Mahalanobis metric spaces is an important problem that has found numerous applications. Several algorithms have been designed for this problem, including Information Theoretic Metric Learning (ITML) [Davis et al. 2007] and Large Margin Nearest Neighbor (LMNN) classification [Weinberger and Saul 2009]. We consider a formulation of Mahalanobis metric learning as an optimization problem, where the objective is to minimize the number of violated similarity/dissimilarity constraints. We show that for any fixed ambient dimension, there exists a fully polynomial-time approximation scheme (FPTAS) with nearly-linear running time. This result is obtained using tools from the theory of linear programming in low dimensions. We also discuss improvements of the algorithm in practice, and present experimental results on synthetic and real-world data sets. Our algorithm is fully parallelizable and performs favorably in the presence of adversarial noise.

The paper proposes a method to handle Mahalanobis metric learning thorough linear programming. The authors consider the specific setup where examples are labelled as similar or dissimilar and the task is to find a mapping such that the feature-space distance between examples is i) smaller than a certain value if the examples are labelled as 'similar' and ii) greater than a possibly different value if the examples are labelled as 'dissimilar'. Arguments from the theory of linear programming are leveraged to define exact and approximated algorithms.
I would tend to reject the paper because I do not fully understand where is where is the main novelty.  Transforming the problem into a linear programming does not look a very complicated step given the specific setup considered in the paper. Moreover, it is not clear enough if there are computational or theoretical gains in following the proposed approach instead of applying other existing methods. Especially because the provided experiments seem to show that there is no improvement  in the accuracy, the authors should have spent some more words  to motivate their strategy.
Questions:
- Are there any theoretical guarantees for the proposed approximation? Is the proposed approximation strategy completely new or similar approaches have been already applied to slightly different setups?
- What are the key differences between the proposed method and with other convex approximations for learning Mahalanobis metrics? As the experimental performance of the proposed approach and other existing methods, what are the net advantages to be associated with the geometric approximation?
- Why is the approximation needed?
- In the proof, why is it true that, for a given solution, adding a constraint implies this constraint is not satisfied?


This work presents the Poincaré Wasserstein Autoencoder, a reformulation of the recently proposed Wasserstein autoencoder framework on a non-Euclidean manifold, the Poincaré ball model of the hyperbolic space H n . By assuming the latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure on the learned latent space representations. We show that for datasets with latent hierarchies, we can recover the structure in a low-dimensional latent space. We also demonstrate the model in the visual domain to analyze some of its properties and show competitive results on a graph link prediction task.

*Paper summary*
The authors endow a variational auto-encoder with a hyperbolic space for the latent space. This enables them to model hierarchically ranked relationships in the data in a natural way. They use a Wasserstein formulation instead of the standard ELBO, which they claim reduces the variance of the gradients during training.
*Paper decision*
Thanks for an interesting paper, I enjoyed reading it. While there are parts of this paper I like very much and I think it is technically sound, I feel the experiments section is lacking somewhat (see comments below). I therefore am recommending a weak reject; although, upon a favourable rebuttal would happily upgrade this position.
*Supporting arguments*
- In the introduction is well written and the motivation for the introduction of a hyperbolic later space is well laid out.
- The section on Hyperbolic geometry is very well written indeed and presents this non-trivial material in a very simple to understand manner. Maybe what it lacks a bit is motivation for a machine learning audience, for why these mathematical tools from differential geometry would be useful or needed by the community to tackle presenting machine learning problems.
- The structure of the method is well laid out and every hyperbolic version of an auto-encoder operation is explained. I would prefer to see methods from cited works, which are deployed in this paper, written in the paper instead of just referred to, so that I don’t have to search through the cited works to find out how a key operation works.
- I think the method itself is technically sound.
- I am not sure about the novelty of the method, seeing that there is a very similar paper from earlier this year (Mathieu et al. 2019).
- The experiments are lacking a little. I do not feel that they show a clear reason why a hyperbolic latent space should be used and compared to standard methods, they do not perform any. better. It would have been nice to have a more structured discussion on the merits of the method.
*Questions/notes for the authors*
- Section 3.2, please explain the significance of \lambda_x^c
- Please explain in the related work and/or introduce in more detail than you already have the differences and similarities between this work and your nearest neighbour “Hierarchical representations with poincaré variational auto-encoders” by Mathieu et al. (2019). I would like to know the advantages/disadvantages of using a Wasserstein formulation instead of a sampled ELBO.
- In section 3.3, please point readers forward to the Appendix for a list of gyrovector operations.
- Please use equation numbers on all equations.
- In your section on “Dispersion representation” you state: “Since the maximum mean discrepancy can be estimated via samples, we do not require a closed form definition of the posterior density as is the case with training using the evidence lower bound. This allows the model to learn richer latent space representations.” Why exactly does not using a closed form posterior density lead to a richer latent space representation? Please back up this statement.
- In the presentation of the ELBO, there is a typo after the second equality where p(z) is listed twice.
- This may just be personal preference on my behalf, but I think a short treatment on VAEs and optimal transport would be useful in the related work and background sections.
- Experimentally it would have been nice to have a direct comparison between this method and Mathieu et al. (2019), that paper being very similar to this one. At least a shared experiment would have been useful.


We present an approach for efficiently training Gaussian Mixture Models (GMMs) with Stochastic Gradient Descent (SGD) on large amounts of high-dimensional data (e.g., images). In such a scenario, SGD is strongly superior in terms of execution time and memory usage, although it is conceptually more complex than the traditional Expectation-Maximization (EM) algorithm. For enabling SGD training, we propose three novel ideas: First, we show that minimizing an upper bound to the GMM log likelihood instead of the full one is feasible and numerically much more stable way in high-dimensional spaces. Secondly, we propose a new annealing procedure that prevents SGD from converging to pathological local minima. We also propose an SGD-compatible simplification to the full GMM model based on local principal directions, which avoids excessive memory use in highdimensional spaces due to quadratic growth of covariance matrices. Experiments on several standard image datasets show the validity of our approach, and we provide a publicly available TensorFlow implementation.

The paper proposes a new method based on Stochastic Gradient Descent to train Gaussian Mixture Models, and studies this method especially in the context of high dimension.
The method is based on optimizing max-lower bound to the log-likelihood, together with a regularization term, using stochastic gradient descent.
The method is applied to two image datasets, and seem to produce sensible results.
However, in my opinion, the results and presentation do not seem at the level suitable for publication.
There are no guarantees for convergence/other performance measures for the new method, in contrast to recent methods based on moment matching (e.g Ge, Huand and Kakade, 2015). Therefore, the new method and paper should provide excellent empirical results to be worthy of publication.
The authors write in the 'related work' section that GMM with regularization was proposed by [Verbeek et al. 2005], but it is an older idea - for example [Ormoneit&Tresp 1998]
In Section 3.1, the motivation for the maximization in eq. (2) is unclear.
Why is it easier to compute the gradient this way rather than keep the original likelihood?
Moreover, the max operation is non-smooth and can cause problems with the definition of the gradient at some points.
The authors point to a problem of underflow/overflow when evaluating the gradient of the
full log-likelihood because the densities p(x | k) can be very small - but it is standard practice in EM to keep all probabilities multiplied by say their maximum p_max and keep log(p_max) separately, to avoid underflow problems.
Section 3.2: I don't understand the requirement \Sigma_{i,j} >=0. Is it a requirement for each entry of the covariance matrix? (which covariance matrix? there are K such matrices). The requirement should be that each matrix is positive-definite, not the entries.
Section 3.3: The first sentence is wrong - EM can also suffer from the problem of local minima.
Also, the single-component solution doesn't seem like a local minimum - but rather the log-likelihood is unbounded here
since you can put another component on one of the data points with infinitely small variance.
The degenerate solution where all Gaussians have equal weights, mean and variance does not seem like a local minimum
of the log-likelihood. Say the data really comes from 2 distinct Gaussians - then separating the two Gaussians means a bit would increase the log-likelihood. is not a local minimum. I'm not even sure if the gradient is zero at this point - the authors should show this. Maybe the authors mean their modified loss L_MC - this should be stated clearly.
The change of regularization during the training process seems like a heuristic that worked well for the authors, but it is thus unclear what optimization problem is the optimization solving. The regularization is thus included for optimization reasons, and not in the usual sense of regularization.
The expression for \tau at the end of Section 3.3 seems wrong to me. I don't see how plugging it into eq. (7) gives a continuous \sigma(t) function.
Section 3.4: What is \mu^i? I didn't see a definition
I don't understand the local principal directions covariance structure. The authors write 'a diagonal covariance matrix of S < D entries). But what about the other D-S coordinates? are they all zero? or can have any values?
The parameters count lists (S+1)*D+1 for each Gaussian so I'm assuming S*D parameters are used for the covariance matrix, but it is unclear how. Eq. (9) has the parameters vectors d_{ks} for the principal directions, together with the \Sigma_ss scalar values - it would be good to relate them to the mean and variance of the Gaussians.
Section 4: The paragraph that describes the experimental details at the beginning is repeated twice.
The experimental results are not very convincing. The images in Figure 2,4 were picked by an unknown (manual?) criteria.
In the comparison to EM in Figure 3 there are missing details - which log-likelihood is used? L, L_MC? or different ones for different methods? is this test set log-likelihood? what fraction of the data was used?
There is also no comparison of running time between the two methods.


Graph Neural Networks (GNNs) have boosted the performance of many graph related tasks such as node classification and graph classification. Recent researches show that graph neural networks are vulnerable to adversarial attacks, which deliberately add carefully created unnoticeable perturbation to the graph structure. The perturbation is usually created by adding/deleting a few edges, which might be noticeable even when the number of edges modified is small. In this paper, we propose a graph rewiring operation which affects the graph in a less noticeable way compared to existing operators. We then use reinforcement learning to learn the attack strategy based on the proposed rewiring operation. Experiments on real world graphs demonstrate the effectiveness of the proposed framework. To understand the proposed framework, we further analyze how its generated perturbation to the graph structure affects the output of the target model and the advantages of the rewiring operation.

This paper proposes the ReWatt method to attack graph classification models by making unnoticeable perturbations on graph. Reinforcement learning was leveraged to find a rewiring operation a = (v1; v2; v3) at each step, which is a set of 3 nodes. In the first step, an existing edge (v1, v2) in the original graph is selected and removed. Then another node v3 that is 2-hop away from v1 and not 1-hop away is selected.  Finally (v3, v1) is connected as a new edge. Some analysis shows that the rewiring operation tends to make smaller changes to the eigenvalues of the graph's Laplacian matrix compared with simply adding and deleting edges, making it difficult to detect the attacks.
Pros
1. The rewiring operation is more unnoticeable. Small change is shown on the eigenvalues with one rewiring operation.
2. The proposed ReWatt method is effective in attacking the graph classification algorithm, facilitated by the policy network to pick the edges.
3. ReWatt outperforms the RL-S2V in terms of success rate, especially when the second step in the rewiring process is not limited by 2-hops away from v1.
4. The paper measured the relative difference between the graph embeddings in terms of L2 norm and measured the KL-divergence in probabilities.
Cons
1. It's quite surprising that ReWatt achieves higher success rate than RL-S2V (first two rows of Table 1).  RL-S2V considers a properly larger set of attacks and uses Q-learning (in contrast to actor critic in ReWatt).  So is it the conclusion that actor critic is better than Q-learning?  Perhaps it will be illustrative to experiment with replacing Q-learning in RL-S2V by actor critic.  This can be implemented in the framework of ReWatt: in Eq 5, replace $p_{fir} * p_{thi}$ by $p(add/remove | e_t)$.
2. The attack is specifically designed for graph classification, while the graph convolutional filter is widely used in other problems like node classification and link prediction. Can it be applied to such problems as well?
3. In addition to RL-S2V, it will be helpful to compare with Nettack (Z¨ugner et. al, 2018).  It employs an admissible set of perturbations, which can be adapted for the rewiring attack.
4. The paper shows the change of eigenvalues under one rewiring operation. How does it change after multiple operations?  In addition, the smaller change to the eigenvalues is compared with rewiring to more distant nodes or adding an edge between two distant nodes.  That is, it is under a *given* $v_{fir}$ and $v_{sec}$.  A different attack may select a different $v_{fir}$ and $v_{sec}$ in the first place.  So it is still not clear whether rewiring leads to less noticeable changes.
5. The experiment splits the dataset into three parts, training set, rewiring operation set, and test set. However, for those predicted incorrectly on the rewiring operation set, the success rate should not be counted.  Perhaps this is already done?