A key challenge of existing program synthesizers is ensuring that the synthesized program generalizes well. This can be difficult to achieve as the specification provided by the end user is often limited, containing as few as one or two inputoutput examples. In this paper we address this challenge via an iterative approach that finds ambiguities in the provided specification and learns to resolve these by generating additional input-output examples. The main insight is to reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct. As a result, to train our probabilistic models, we can take advantage of the large amounts of data in the form of program outputs, which are often much easier to obtain than the corresponding ground-truth programs.

[Summary]
This paper aims to improve the generalization of program synthesis, ensuring that the synthesized programs not only work on observed input/output (I/O) examples but also generalize well to assessment examples (i.e. model the real intent of the end-user). To this end, the paper proposes a framework that iteratively alternates between producing programs with an existing program synthesizer and augmenting examples to disambiguate possible programs with a neural oracle that learns to select correct outputs. Several architectures and the design of input of the neural oracle have been investigated. The experiments on Andriod layout program synthesis with an InferUI synthesizer show that the proposed framework can improve the generalization of synthesized programs. However, I find it is difficult to evaluate the effectiveness without sufficient qualitative results and the intermediate outputs (e.g. a distinguishing input and candidate outputs) of the proposed framework (see details below).
Significance: are the results significant? 4/5
Novelty: are the problems or approaches novel? 4/5
Evaluation: are claims well-supported by theoretical analysis or experimental results? 3/5
Clarity: is the paper well-organized and clearly written? 4/5
[Strengths]
*motivation*
- The motivation for improving the generalization of program synthesis by augmenting examples is convincing.
*novelty*
- The idea of utilizing a neural network to select correct outputs to augment examples for disambiguating the possible programs is intuitive and convincing. This paper presents an effective way to implement this idea.
*technical contribution*
- The paper investigates a set of network architectures and ways to specify the network input for learning the neural oracle. The RNN+CNN model that leverages both rendered views and features seems effective.
*clarity*
- The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 1 clearly shows the proposed framework.
*experimental results*
- The presentations of the results are clear. The results demonstrate that the proposed framework can improve generalization accuracy.
*reproducibility*
- Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible if the dataset is available.
[Weaknesses]
*related work*
The descriptions of the related work are not comprehensive. Some neural program synthesis works explore a variety of mechanisms to encode examples and fuse their features, which are not mentioned in the paper. [Devlin et al. in ICML 2017] investigates different attention mechanisms to sequentially encode a set of I/O examples and performs pooling to merge them. [Sun et al. in ICML 2018] proposes a doubly encoding method to capture more details of examples and merge the features using a relation network. I believe it would be interesting to see if these methods could further improve the performance of the neural oracle.
*experiment setup*
- The experiments are not sufficient. While the claims look promising, the proposed method is only evaluated in only one dataset, which is not sufficiently convincing. I suggest the authors to also experiment the FlashFillTest dataset where string transformation programs are synthesized.
- A more comprehensive description of the dataset is lacking.
*experiment results*
- I find it hard to judge the effectiveness of the proposed framework without seeing sufficient qualitative results. I suggest the authors randomly sample some synthesized programs (both success and failure) and present them in the paper.
- I believe it is important to present some examples of the given I/O pairs, initially synthesized programs (p_1), found distinguishing input (x*), candidate outputs (y), the prediction of the neural oracle (i.e. selected outputs), the augmented examples (I \cup {(x*, y*)}), and finally the next synthesized program. Without this, it is very difficult to understand the performance of the proposed framework and what could go wrong.
*ablation study: the neural oracle*
Only the final performance (i.e. the program synthesis performance) is shown in the paper. I believe it would be helpful if the performance of the neural oracle was also presented. As the whole framework depends on how accurate the neural oracle can select the correct output, it is important to evaluate this. One way to show this is to simply show the performance of all the neural oracles (with different architectures) trained on D_S (the positive samples and the incorrect samples) or even D_{S+}.
Devlin et al. "RobustFill: Neural Program Learning under Noisy I/O" in ICML 2017
Sun et al. "Neural Program Synthesis from Diverse Demonstration Videos" in ICML 2018


Visual relationship recognition models are limited in the ability to generalize from finite seen predicates to unseen ones. We propose a new problem setting named predicate zero-shot learning (PZSL): learning to recognize the predicates without training data. It is unlike the previous zero-shot learning problem on visual relationship recognition which learns to recognize the unseen relationship triplets (<subject, predicate, object>) but requires all components (subject, predicate, and object) to be seen in the training set. For the PZSL problem, however, the models are expected to recognize the diverse even unseen predicates, which is meaningful for many downstream high-level tasks, like visual question answering, to handle complex scenes and open questions. The PZSL is a very challenging task since the predicates are very abstract and follow an extreme long-tail distribution. To address the PZSL problem, we present a model that performs compatibility learning leveraging the linguistic priors from the corpus and knowledge base. An unbalanced sampled-softmax is further developed to tackle the extreme long-tail distribution of predicates. Finally, the experiments are conducted to analyze the problem and verify the effectiveness of our methods. The dataset and source code will be released for further study.

Title: Good work, requires some edits.
1. Summarize:
This paper proposes a new problem setting in visual relation detection which is called “Predicate Zero-shot Learning (PZSL)”. They provide a clear motivation and description of this setting. They propose a solution to this problem which leverages linguistic priors and knowledge bases. Furthermore they propose an unbalanced sampled-softmax to tackle the long tail distribution of predicates.
2. Clearly state your decision. One or two key reasons for this choice.
I will go for a weak accept for the paper at this stage. (+) I think the proposed problem setting is well-motivated and useful. Also, (+) the proposed initial solution to this problem is interesting. However, (-) they propose a “fast graph convolution network” which seems to be precisely equivalent to a PinSage.  Also, (-) the paper requires to be polished as it lacks clarity.
3. Main discussion
My first argument is: I’m not sure why the authors have changed the name of PinSage and just mentioned that “their” “Fast Graph Convolution Network” is “inspired” from PinSage. To me it looks exactly the same. If there are any differences, it should be stated clearly. In fact, I would not be against using PinSage as a part of their approach. However, trying to rename it without clear reasons is not a good idea.
My second argument is that the paper lacks clarity in writing (for detailed suggestions please refer to comments and feedbacks). Specially the evaluation section lacks details and clarity: a) In the beginning of this section (page 7), the authors talk about “generalized” and “traditional” settings without properly defining them. b) The descriptions for Table 1 and Table 2 fail to provide enough details to help understand the difference between the results in these two tables (one of them states “Accuracy of unseen predicate recognition” and the other one “Accuracy of recognition of triplets with unseen predicates”).
4. Comments and feedback.
Introduction:
Paragraph one in the:
1. The relationship recognition methods are mainly supervised “that” → “to”.
2. last line: …. and do not study “on generalizing” → “the generalization of”.
Paragraph two:
1. no manual annotations or “real samples” → “image samples”. (a real sample is ill-defined)
2. For example, no instance of chew → For example “given” no instance of chew.
Paragraph three:
1. … is difficult since predicates are often abstract not as specific →   is difficult since predicates are often abstract “and” not as specific.
2. Furthermore, unlike many object ZSL methods … → This line to the end is very complicated and hard to understand.
Related Works:
1. Visual Relationships: I would cite “Graph R-CNN for scene graph generation” since it is the most relevant work regarding the similarity of pipeline (using GCNs).
2. External Knowledge bases  (KB): I would cite “Improving Visual Relationship Detection using
Semantic Modeling of Scene Descriptions” since it is one of the most relevant works using knowledge graph modelings to improve visual relation detection.
Problem Setup:
Do you plan to provide the proposed dataset splits so others can work on this setting? I consider this very important given your paper’s contribution. Maybe it is better if it is also mentioned in the paper.
Pipeline:
1. Paragraph 2: … the output of which is fused with …: Given Figure 2, it does not seem like $V_p$ is being created by fusing $V_s$ and $V_o$. It looks more like it is extracted directly from the image (union of bounding boxes).
2. In Figure 2: In the representation of Pipeline (A), the graph is colored by dark blue for objects and light blue for predicates. The represented graphs show Object to Object and Predicate to Predicate connections which I’m not sure if it is correct. Shouldn’t we always have a light blue between every pair of dark blue connections?
Evaluation:
1. Please consider the mentioned points in the Main Discussions.
2. In Table 1, I suggest re-naming “embedding” to “initial embedding”.
3. In Table 1, Hit@k should be Hits@k.
4. Please define the metrics clearly (Hits@k).
Extra: I have a question regarding the ablation studies with GloVe, Normal and InferSent initialization. The question is whether this initialization is necessary? It seems like in the setting “W/O KG”, even though the embeddings are initialized with GloVe, there is no gain (all of the Hits@k values are 0.0). So GloVe embedding without KG bring no external semantic knowledge? Then why use them? Regarding that, I can see that a GCN, initialized with normally distributed embeddings (row 14 in Table 1) has given 0.0 accuracies, but I find this very counter intuitive, as graph convolution layers already have trainable weights capable of compensating for the lack of ‘proper’ initialized embedding and getting 0.0 does not make sense to me.
Conclusions and Future Work:
1. two lines before the last: please use “\citep”.


Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unrealistic. This problem is typically addressed by either pre-processing words into subword units or performing translation directly at the level of characters. The former is based on word segmentation algorithms optimized using corpus-level statistics with no regard to the translation task. The latter learns directly from translation data but requires rather deep architectures. In this paper, we propose to translate words by modeling word formation through a hierarchical latent variable model which mimics the process of morphological inflection. Our model generates words one character at a time by composing two latent representations: a continuous one, aimed at capturing the lexical semantics, and a set of (approximately) discrete features, aimed at capturing the morphosyntactic function, which are shared among different surface forms. Our model achieves better accuracy in translation into three morphologically-rich languages than conventional open-vocabulary NMT methods, while also demonstrating a better generalization capacity under low to mid-resource settings.

This paper addresses the problem of translating into morphologically-rich languages, which suffers from the problem of sparse vocabularies and high numbers of rare and unseen words. In particular, current approaches such as subwords lack explicit notions of morphology and are obtained independently of the translation objective, while operating at the character-level renders the learning of long-distance dependencies more difficult.
More concretely, this paper models the generation of target words in a stochastic, hierarchical process, where the morphological features are modelled as latent variables. At each target word, the model samples a vector representing its lemma, followed by a k-dimensional latent inflectional features (e.g. nominative or accusative). To induce sparsity in the inflectional features, the paper uses a "stretch-and-rectify" distribution (Louizos et al., 2018) using the Kumaraswamy distribution. The paper further applies a sparsity-inducing regulariser to encourage the inflectional features to take discrete values of "0" or "1". Parameter estimation is done by optimising a lower-bound on the marginal log-likelihood. Experiments on translations into three morphologically-rich languages, English-{Arabic, Czech, Turkish}, indicate fairly small but consistent and statistically significant gains in BLEU. Further analysis indicates improved perplexity upper bound on rare words compared to subword-based baselines, along with somewhat interpretable latent inflectional features.
Overall, this paper proposes an elegant solution to an important problem, and yields statistically significant BLEU improvements over the baselines. I have listed some pros and cons below, with a recommendation of "Weak Accept". I would be willing to raise my scores assuming my concerns are sufficiently addressed.
Pros:
1. The proposed approach is easy-to-follow and explained clearly, and the paper is overall well-written.
2. Experiments are done on translation into three morphologically-rich languages with both concatenative and non-concatenative morphology, and results in small but mostly consistent BLEU improvements over the best subword, hierarchical, and character baselines for each language.
3. The proposed approach is general and can potentially be applied on top of other model architectures (e.g. Transformers) and other language generation problems, such as language modelling, summarisation, etc, although the experiments in the paper focus on NMT with a GRU-based architecture.
Cons:
1. It is still unclear how much computational overhead is introduced by the approach (in terms of both training and prediction times), and how scalable the approach is when applied to language pairs that have much bigger training sets. In Tables 4 and 5, it seems that the largest dataset is English-Turkish multi-domain (434K sentence pairs), which is fairly small compared to other datasets.
2. Regarding the feature variations result in section 4.4.4, it is hard to draw any conclusions about the latent inflectional features just based on varying the inflectional features at one position. I would suggest running the same experiment for multiple words at different positions, and see whether the same set of inflection features always results in similar inflections. This would better convince the reader that the latent inflectional features really are capturing useful morphological information, and that the target word generation process is appropriately controlled by the latent variable.
3. Since some of the experiments did not use the same standard splits (e.g. the multi-domain Turkish experiments), it would be nice to report how well external models (e.g. OpenNMT) would do in the authors' dataset split, to make sure that the reported numbers here are at least comparable to that. This would help ensure the credibility of the findings.
4. The effect of data size experiments (Section 4.4.2) for the character model seems somewhat counter-intuitive. Why would making the training set bigger (by incorporating data from a more diverse domain) make the character model worse? The explanation that "[the character model]'s capacity cannot cope with the increased amount of sparsity" does not seem satisfactory.  Why does adding more data result in increased sparsity? Furthermore, one can simply use more hidden units or deeper layers to mitigate this problem in the character-based model. Assuming the authors are controlling for model capacity, then the character model's hidden state can be made bigger (since the vocabulary size is lower for character models, thus the embedding layer and the softmax layer are by definition smaller than the subword-based model).
Minor suggestions:
1. Section 3.5 is not very clear; adding some figures or more explanation there would help understand how prediction is done.
2. It would be interesting to explore other potential metric for measuring morphological generalisation. For instance, at evaluation time, does the model predict words that it has never seen before (e.g. predicting "plays", even though the model has only seen "play" at training time) at a higher frequency than character/subword/hierarchical baselines? If yes, this would provide more evidence that the latent inflectional features are properly capturing morphological information.


Deep artificial neural networks can achieve an extremely small difference between training and test accuracies on identically distributed training and test sets, which is a standard measure of generalization. However, the training and test sets may not be sufficiently representative of the empirical sample set, which consists of real-world input samples. When samples are drawn from an underrepresented or unrepresented subset during inference, the gap between the training and inference accuracies can be significant. To address this problem, we first reformulate a learning algorithm as a procedure for searching for a source code that maps input features to classes. We then derive a necessary and sufficient condition for generalization using a universal cognitive similarity metric, namely information distance, based on Kolmogorov complexity. Using this condition, we formulate an optimization problem to learn a more general classification function. To achieve this end, we extend the input features by concatenating encodings of them, and then train the classifier on the extended features. As an illustration of this idea, we focus on image classification, where we use channel codes on the input features as a systematic way to improve the degree to which the training and test sets are representative of the empirical sample set. To showcase our theoretical findings, considering that corrupted or perturbed input features belong to the empirical sample set, but typically not to the training and test sets, we demonstrate through extensive systematic experiments that, as a result of learning a more general classification function, a model trained on encoded input features is significantly more robust to common corruptions, e.g., Gaussian and shot noise, as well as adversarial perturbations, e.g., those found via projected gradient descent, than the model trained on uncoded input features.

This paper proposes a novel approach to training classifiers inspired by concepts from algorithmic information theory. Specifically, the inputs are augmented with additional features formed by encoding the image (using a source code). In addition, the paper argues to use the normalized compression distance as a loss criterion when training; this is a computationally tractable proxy for the normalized information distance, which the paper argues is what one really should use to generalize well, but which is not computationally tractable. (In particular, evaluating the Kolmogorov complexity, which is required to compute the normalized information distance, is not tractable.) The idea is demonstrated via experiments training common image classifiers (VGG-11 and ResNet-18) on clean and corrupted versions of the CIFAR-10 dataset, and comparing how they fare against adversarially trained methods on noisy and adversarial examples.
I believe that this paper contains interesting and novel ideas, but there are also some inconsistencies and some points that are vague. I would consider raising my score if the authors address or clarify these issues, which are detailed below.
The introduction discusses generalization very broadly, suggesting that the usual definition of generalization (e.g., accuracy on a hold-out set, drawn from the same distribution as the training set) is not the version of generalization that one should focus on. However, the precise definition of generalization is never provided. The intro also mentions briefly about adversarial examples, and this is the focus of the experiments. It would help to clarify whether the paper focuses on being robust to adversarial examples as a form of generalization, or more precisely, what how is the concept of generalization defined/measured in this paper. (E.g., the first sentence of Sec 2.1 mentions "our running definition of generalization," but no precise definition has yet been provided.
To make the paper self-contained, it would help to recall the formal definition of Kolmogorov complexity; this could be done in the appendix.
Sec 2 begins to make the connection between a classifier and a source code. However, these definitions are not precise. For example, in the usual PAC learning setting one has a distribution D over the training data (x,y). After specifying a particular loss function, the Bayes-optimal classifier is well defined. Does the source code C correspond this classifier? I would guess not, since the paper seems to be introducing a specific loss criterion, but it would be good to clarify this in the paper. Since you are assuming there is a "true output function" f, does this correspond to what is usually referred to as the "realizable" setting (i.e., where y = f(x) is a deterministic function of x)?
Notation is sometimes used without having been defined. For example, what is $\vec{x}_{S}$?
It isn't clear how one can train using the criterion (4), or equivalently, solve the problem set forth in Prop 2, in practice, since the true source code C is not known. Please elaborate on this.
In the experiments, how precisely is the VGG network modified to handle the encoded inputs? Are these just passed as additional input channels? Is the network architecture need to be modified in any other way to account for these additional inputs? Also, what loss criterion is used for training? Cross-entropy? Or something related to normalized information distance? This isn't clear from the discussion in Sec 3.
In Sec 3.2, is it really fair for the PGD attacks to only use the gradient of the loss wrt the uncoded input? Is this still really "white box"? How does the proposed approach perform if the attack also has access to the encoded input? It would be good to report both settings in the paper.
The introduction made connections to other forms of generalization, including domain generalization and domain adaptation. Based on this, I had expected to experiments illustrating the utility of the proposed for these problems.
The existing experiments do make clear that the encoded inputs are more robust to perturbations and aversarial attacks than uncoded inputs. Is the encoding assumed to be performed on the perturbed input, or on an unperturbed input (i.e., before it is perturbed)?
I don't find the results in Table 1 fully convincing that the proposed approach should be preferred over previous approaches. For instance, the approach of Madry et al. (2018) achieves slightly lower test accuracy, but substantially higher inference accuracy. In general, one expects to see a tradeoff here. If one were to plot these points, would the proposed method clearly fall above the Pareto frontier? Minor: Please clarify the definition of "inference accuracy" in the paper.


Spectral embedding is a popular technique for the representation of graph data. Several regularization techniques have been proposed to improve the quality of the embedding with respect to downstream tasks like clustering. In this paper, we explain on a simple block model the impact of the complete graph regularization, whereby a constant is added to all entries of the adjacency matrix. Specifically, we show that the regularization forces the spectral embedding to focus on the largest blocks, making the representation less sensitive to noise or outliers. We illustrate these results on both on both synthetic and real data, showing how regularization improves standard clustering scores.

This paper analyzes the effect of regularization on spectral embeddings in a deterministic block model and explicitly characterizes the spectra of the Laplacian of the regularized graph in terms of the regularization parameter and block sizes. To my knowledge, this has not been done before. Prior work either derives sufficient conditions for the recovery of all blocks in the asymptotic limit of an infinite number of nodes in the case of (Joseph & Yu, 2016), or lower bounds the number of small eigenvalues of the Laplacian of the unregularized graph on random graphs in expectation (therefore arguing in favor of regularization) in the case of (Zhang & Rohe, 2018). This paper, on the other hand, gives a precise characterization of the eigenvalues and eigenvectors (albeit in the case of a deterministic graph); the results are elegant and the analysis uses simple elementary techniques, which is very satisfying and seems to be easy to build on. The authors mention that they would like to extend this analysis to stochastic block models, which would indeed be interesting. The paper is also well written and the results are clearly presented. Overall, this is a nice contribution to spectral graph theory and so I recommend acceptance.


Evolutionary Strategies (ES) are a popular family of black-box zeroth-order optimization algorithms which rely on search distributions to efficiently optimize a large variety of objective functions. This paper investigates the potential benefits of using highly flexible search distributions in ES algorithms, in contrast to standard ones (typically Gaussians). We model such distributions with Generative Neural Networks (GNNs) and introduce a new ES algorithm that leverages their expressiveness to accelerate the stochastic search. Because it acts as a plug-in, our approach allows to augment virtually any standard ES algorithm with flexible search distributions. We demonstrate the empirical advantages of this method on a diversity of objective functions.

Summary:
As the title of the paper states, this paper tries to improve evolution strategies (ES) using a generative neural network. In the standard ES candidate solution is generated from a multivariate normal distribution, where the parameters of the distribution are adapted during the optimization process. The authors claim that the gaussian distribution, i.e., the ellipsoidal shape of the sampling distribution, is not adequate for the objective functions such as multimodal functions or functions with curved ridge levelsets such as the well-known Rosenbrock functions. The motivation is clearly stated. The technique is interesting and non-trivial. However, the experimental results are not very convincing to conclude that the proposed approach achieves the stated goal. Moreover, this paper may fit more to optimization conferences such as GECCO.
Because of the empirical results, I would rate this paper as the border line (around 5), but due to the slightly annoying rating system the rate appears as 6.
Comments:
P2: "Efficient Natural Evolutionary Strategies (xNES) (Sun et al., 2009) has been shown to reach state- of-the-art performances on a large ES benchmark."
This algorithm is "eNES" and  this algorithm is not competitive with the state-of-the-art ES such as CMA-ES. The authors might want to refer to exponential NES, which is xNES, proposed by Glasmachers et al 2010.
P5: "Indeed, other bijective GNN models like the Real-NVP (Dinh et al., 2016) introduce non-volume preserving transformations, which can easily overfit and lead to premature concentration and convergence." Has it been reported in a reference? If so provide the reference. If not, the authors should state that it has been observed the authors preliminary study. In any case, I think it depends how the model is used or trained, and this statement itself is not universally true.
P7: "By using fη instead of gη as the push-forward map of the NICE model, we ensure that the flexibility brought by the GNN only impacts the tails of the search distribution. As detailed in an ablation study presented in Appendix F, this additional tool turns out to be essential in order to use GNNs for ES."
I barely understood this point. Please make is clearer.
P8: Experimental results are not very convincing. The experiments are limited to dimension 2, 5, 10 and only a few functions are selected from the BBOB test function suite. How about on 20D? What happens if the target is 1e-8, which is the default setting in BBOB?
Figure 3 looks interesting, and this is what the authors are trying to achieve. Therefore, it looks like the authors reached the stated objective. However, this is only 2D. No results are provided to convince that the proposed strategy achieved the stated objective.
Figure 4  simply looks that the proposed algorithm failed to reach the "flexibility" stated in Section 2: "Another limitation of classical search distribution is their inability to follow multiple hypothesis, that is to explore at the same time different local minima. Even if mixture models can show such flexibility, hyper-parameters like the number of mixtures have optimal values that are impossible to guess a priori."
From these results, I am not convinced that the proposed strategy really achieved more flexible distribution than the classical methods, and whether the flexibility contributes to improve the performance.
Another critical point to be discussed is its usefulness. Since this algorithm is proposed to "improve evolution strategy" as a black-box optimizer (not for specific tasks), I expect to improve the state-of-the-art performance. Are the reported results outperform the CMA-ES? Based on Glasmachers et al (2010), xNES tends to require more objective function evaluations than CMA-ES, especially for higher dimensional cases. I am curious to know if the proposed approach outperforms the CMA-ES on Rosenbrock functions.


How to represent the sentence "That's the last straw for her"? The answer of the self-attention is a weighted sum of each individual words, i.e.
. But the weighted sum of "That's", "the", "last", "straw" can hardly represent the semantics of the phrase. We argue that the phrases play an important role in attention. If we combine some words into phrases, a more reasonable representation with compositions is semantics = α 1 Emb(That's)+Emb 2 (the last straw)+α 3 Emb(for)+α 4 Emb(her)
. While recent studies prefer to use the attention mechanism to represent the natural language, few noticed the word compositions. In this paper, we study the problem of representing such compositional attentions in phrases. In this paper, we proposed a new attention architecture called PhraseTransformer. Besides representing the words of the sentence, we introduce hypernodes to represent the candidate phrases in attention. PhraseTransformer has two phases. The first phase is used to attend over all word/phrase pairs, which is similar to the standard Transformer. The second phase is used to represent the inductive bias within each phrase. Specially, we incorporate the non-linear attention in the second phase. The non-linearity represents the the semantic mutations in phrases. The experimental performance has been greatly improved. In WMT16 English-German translation task, the BLEU increases from 20.90 (by Transformer) to 34.61 (by PhraseTransformer).

This paper addresses an issue of compositionality in self-attention models such Transformer. A simple idea of composing multiple words into a phrase as a hypernode and representing it using a non-linear function to capture the semantic mutation is proposed. In the machine translation and PoS tagging tasks, the proposed PhraseTransformer achieves impressive gain, especially +13.7 BLEU score compared to the Transformer.
The motivation of the paper is very clear, and I love this kind of paper; with a simple idea, making a huge impact on the field. I appreciate the real example to compare how word-level self-attention is different from the phrase-level self-attention in Abstract and Figure 1. The problem itself; tackling the semantic compositionality of self-attention, is a very important problem, and I like the part that authors described it as an inductive bias as a model perspective.
However, this work seems to be problematic in terms of presentation, clarity, and meaningful comparisons. Please see my detailed comments below.
First, what exactly is “semantic mutation”? The term has been used here and there to describe the inductive bias in semantic compositionality and to show how the nonlinearity can effectively capture it. But, I couldn’t find any definition from the paper, couldn’t find any formal definition from any ACL papers, and couldn’t guess myself based on the context. I am guessing it is probably some sort of combination of meaning in phrase-level words. If so, more importantly, how does the simple non-linear function (i.e., sigmoid) can capture such semantic combinations of words? How could it make such a huge gain (PhraseTransformer vs Linear PhraseTransformer) in Table 1 on MT task? This seems to be the most important contribution of this paper, of which I don’t understand yet.
What is the “concept of hypernodes in hypergraph”? I think it is not a common word that I can understand it clearly without any references or any background. It would be better to add some references for the concept. Again, I am guessing it is a sort of graph theory that decomposes a large node into small pieces but keeps their connectivity. But, then how is that exactly linked to phrases of words? If you make phrases only on consecutive words, it is basically just chunking. I don’t find any relevance of phrases in a sentence with the (hyper)graph something.
In Figure 2, how is the bidirectional path made between the word representations and phrase representations? In my understanding of your algorithm based on Equation 2-6, I only see the attention of phrases is computed by word attention, but not the other way.  Please clarify this. If so, how does the gradient back-propagate to each other?
The biggest concern of this work is the scores reported in Table 1. I have checked the recent papers which used the Mutli30K(de-en) and other results from WMT[16-18] reports, but the BLEU score reported in Table 1 (20.90) seems way lower than the scores reported by any systems trained by either non-Transformer or Transformer systems. For that reason, it would be fair to include some results from the state-of-the-art systems on the same dataset.
A minor point but in the complexity analysis, your m is basically n because you take consecutive words from n length of sentence. You better distinguish which variables are dependent on each other first.
There are MANY typos, missing captions, grammatical errors in the paper. Here are only some of them:
There is no caption for Figure 1.
When you cite a reference with its model name, you better make the citation under parenthesis.
“equation equation 5” -? “equation 5”
“Wee” -> “We”


Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the loglikelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.

This paper presents a method for evaluating latent-variable generative models in terms of the rate-distortion curve that compares the number of bits needed to encode the representation with how well you can reconstruct an input under some distortion measure. To estimate this curve, the author’s use AIS and show how intermediate distributions in AIS can be used to bound and estimate rate and distortion. They apply their evaluation to GANs, VAEs, and AAEs trained on MNIST and CIFAR-10.
I found this paper well written, with a number of interesting technical contributions, particularly how to leverage AIS to compute R-D curves for an individual model. However, the utility and interpretation of these R-D curves for single models remains confusing to me, and there is insufficient discussion and comparison to other joint diversity/sample quality metrics proposed in the GAN literature. The compute time required for evaluation may also limit the applicability: 4-7 hours for 50 images on MNIST, and 7 days for CIFAR-10.

Major comments:
* How should we interpret rate-prior distortion for an individual model vs. rate-distortion where models are optimized for each rate? Past work in learned compression and generative models (Theis et al. 2016, Balle et al. 2016, Alemi et al., 2018) show that models must adjust their decoder (and prior) as a function of rate to be optimal in terms of distortion. For a fixed decoder, optimizing the prior may still be required to achieve low rate. Given that many of the models you compare are trained to do well at one point on the R-D curve, why does it make sense to evaluate them at other points? Additionally, you only evaluate models with deterministic decoders and many of the experimental conclusions are highly specific to this setting but not noted.
* As you focus on general distortion metrics instead of NLL alone, it'd be interesting to compare curves under different distortion measures, e.g. MS-SSIM for images or L1 vs. L2. Right now there's not much experimental novelty vs. prior work that looked at rate-distortion curves with NLL distortion and Gaussian observation models.
* It’d be useful to include experiments comparing Rate-Prior distortion curves and Rate-distortion curves where you a) optimize over the prior, b) optimize the decoder, fixing the prior, and c) optimize both the prior and decoder.
* There’s no comparison of other approaches to generate the rate-prior distortion curve. For example, you could just use an amortized inference network like in VAE w/ a flexible variational family and anneal beta over time.
* There are several related papers which should be discussed and contrasted, in particular https://arxiv.org/abs/1901.07821 which looks at rate-distortion-perception tradeoffs, and https://arxiv.org/abs/1806.00035 which presents precision-recall curves for diversity/quality metrics applied to implicit models. How do the insights gained from the rate-distortion curves relate to precision/recall and why should one be preferred over the other? https://arxiv.org/abs/1611.02163 also looked at distortion as a metric for GANs (equivalent to beta -> infinity in your framework).
Minor comments:
* Missing discussion of several related works: that presents a complexity measure for the latent space of GANs: https://arxiv.org/abs/1802.04874
* “Wasserstein distance remains difficult to approximate…” - see https://openreview.net/forum?id=HkxKH2AcFm that advocates for evaluation with Wasserstein
* Tightness of bound on simulated data (what BDMC provides) may not correspond to tightness of bound on real data (what you care about in practice).
* The treatment of VAEs as implicit models only makes sense with location scale family p(x|z), thus the entire framework proposed here doesn’t make sense with e.g. autoregressive p(x|z), as used in PixelVAE and others.
* Why focus on fixed prior p(z)? An alternative would be to optimize p(z), q(z|x) and fix p(x|z). How would this change the resulting rate-prior distortion curves?
* “We can compute the R-D curve by sweeping over \beta rather than by sweeping over D” - this is not the case when the R-D curve has linear segments, see e.g. Rezende & Viola 2018
* Many of the properties and discussion around rate-prior distortion functions (especially w/NLL distortion) are also in Alemi et al. 2018 as their definition of “rate” is identical to your definition of “rate-prior”. Also many of these properties are specific to continuous latents which isn’t noted.
* Should clarify that q_k(z|x) correspond to points along R_p(D)
* The results in Eqn 14-17 showing you can tractably estimate distortion and get an upper bound on rate using the AIS-derived distributions are very cool!
* “AIS variance is proportional to 1/MK” - this is for variance in the partition function? How does this translate to variance in estimates of rate/distortion?
* “In the case of probabilistic decoders, …” -> need the caveat this is with NLL distortion
* Validation on the linear VAE is great!It looks like some of the points for AIS at low distortion are below the analytic rate, but the proofs indicate the estimated rate should be greater than the analytic rate. Is this just noise?
* Fig 4 and 5: hard to see difference between the dashed lines
* VAE results would change drastically if you targeted them to different regimes (e.g. beta-VAE or constrained optimization like GECO)
* Statements like “VAE is trained with the ELBO objective, which encourages good reconstructions” only make sense when the decoder is location-scale. VAEs w/rich autoregressive decoders typically do a horrible job reconstructing.
* How robust are model differences across random initialization? It’d be great to add error bars to all these plots, especially given that GAN training can stochastically succeed.
* Eqn 18/Fig6a: depending on the dataset, you could easily notice the difference of 4.6 nats and log-likelihood could still tell these two models apart. It’d be useful to add a line at beta=1 to show that the likelihood would be the same but the R-D curves are different.
========================
Update after rebuttal
Thank you to the authors for addressing a number of my concerns, adding additional text and experiments. However, my main concern remains: if rate-distortion in an individual model is a useful method for evaluating generative models, you should compare it empirically with other metrics that have been proposed for this purpose (e.g. precision-recall). Additionally, while the theoretical novelty of getting the full R-D curve from a single AIS run is very cool, I'm skeptical of the practical utility as a metric for generative models due to the computational costs of AIS (4-7 hours for 50 images on MNIST). The simple baseline I suggested of training an amortized inference network with beta annealed over time would not require training a separate encoder for each point in R-D, you could just start beta large, and anneal beta in steps to 0 over time, tracing out an R-D curve. Given the current experiments, it's not obvious if the win of AIS in terms of accurate posterior inference is worth the increased computational cost over a simple VI baseline.


We study the use of hypermodels to represent epistemic uncertainty and guide exploration. This generalizes and extends the use of ensembles to approximate Thompson sampling. The computational cost of training an ensemble grows with its size, and as such, prior work has typically been limited to ensembles with tens of elements. We show that alternative hypermodels can enjoy dramatic efficiency gains, enabling behavior that would otherwise require hundreds or thousands of elements, and even succeed in situations where ensemble methods fail to learn regardless of size. This allows more accurate approximation of Thompson sampling as well as use of more sophisticated exploration schemes. In particular, we consider an approximate form of information-directed sampling and demonstrate performance gains relative to Thompson sampling. As alternatives to ensembles, we consider linear and neural network hypermodels, also known as hypernetworks. We prove that, with neural network base models, a linear hypermodel can represent essentially any distribution over functions, and as such, hypernetworks are no more expressive.

The authors demonstrate advantages of a linear hypermodel over an ensemble method in exploration guided by epistemic uncertainty. They perform an empirical study in the bandit setting and claim that their approach both outperforms the ensemble method and offers a significant increase in computational efficiency. The theoretical contribution is that they prove universality in the sense that an arbitrary distribution over functions can be represented by a linear hypermodel. The experiments support their claims. Some of the explanations, however, are confusing, and relations to prior work should be clarified.
Figure 3 shows a surprisingly large performance gap between the hypermodel and the ensemble method as the number of actions increases. But how about comparing linear hypermodels with different index sizes? Do we also expect asymptotic improvement as we increase the index size?
Imprecise or confusing explanations in the paper:
1) Page 2, first Q: In theory, the effectiveness of the ensemble method should converge to that of the hypermodel as the ensemble size increases. They only tried ensemble size [10, 30, 100, 300] and then concluded that linear hypermodel can be effective regardless of the size of ensembles. Why?
3) Page 3, Section 2.1, second paragraph, first sentence: Please clarify a bit more what do you mean by perturbing data? Random shuffling of the dataset in each training epoch? What does ‘response variables’ mean?
4) Page 3, Section 2.1, second paragraph, last 2 sentences about $A_t$: we guess it should be $A_t ~ N(0, I)$ if $p_z$ is unit Gaussian according to the description in this paragraph. The current text claims it is the other way around, perhaps a typo?
5) Page 3, Section 2.1, first equation: Why take the inner product between $a$ and $z$ ? How does this reflect the randomized computation (the motivation for augmented random vector $A_t$)? The objective is to maximize the log-likelihood of the prediction under the Gaussian assumption. Please clarify the assumptions about random variables $Y_t$ at the beginning of this paragraph.
6) Same place as in 5): Why regularize hypermodel parameters such that they are not too far from the initial vector? Is $\nu_0$ actually the additive prior model described in Section 2.5?
7) Page 3, Section 2.1, second equation: why multiply $|D|$ in the first term within the parentheses? Why not just $1/|D_tilde|$ to average the prediction error over the mini-batch?
8) Page 3, Section 2.1, second equation: is the cardinality of the index set $|Z_tilde|$ independent of mini-batch size? I.e. for each training data point there could be multiple models realized by multiple indices $z$
9) Page 4, Section 2.5: Why use this decomposition for training the hypermodel? If the intuition is to keep the initial weight small, what if we just simply initialize small values for $f_\theta(x)$ without decomposition?
10) Page 5, last second sentence: The notation of partition (the set notation after ‘Here,….’) is supposed to be $\hat{\mathcal{Z}}_{x^*} = \{ z\in \hat{\mathcal{Z}} | x^* in \argmax_{x} f_{g_{\nu}(z)}(x) \}$
Minor typos:
- Page 2, third paragraph: ‘…we compare their [efficacy] when used...’ ->  [efficiency] ?
- Page 2, the last paragraph before Section 2, first sentence: ‘Approaches to approximating TS and [informatino]-directed sampling...’ -> [information]
Relations to prior work:
1. Page 2: Hypernetworks (where one neural net learns to generate the weights of another net) are much older than this recent reference of 2016. One should relate this work to the original references since 1991 [FAST0-3a][FAST5][FASTMETA1-3][CO2] in section 8 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html
2. Intro 2nd par: dropout was first published much earlier in 1990 as the stochastic delta rule:
Hanson, S. J.(1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272. See also arXiv:1808.03578, 2018.
We might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.
Edit after rebuttal: The authors replied: "Thanks for pointing out typos and citations that we will add." But apparently in the revised PDF this did not happen.


Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data.

The main contribution of this paper is that it proposed an estimator of Jacobian regularization term for neural networks to reduce the computational cost reduced by orders of magnitude, and the estimator is mathematically proved unbiased. In details, the time consumed for the application of Jacobian regularizer and the unbiasedness of the proposed estimator are proved mathematically. Then the author experimentally demonstrated that the proposed regularization term retains all the practical benefits of the exact method but with a low computation cost. Quantitative experiments are provided to illustrate that the proposed Jacobian regularizer does not adversely affect the model, can be used simultaneously with other regularizers and effectively improve the model's robustness against random and adversarial input perturbations.
In general, this paper was well organized and it is also great that an efficient approximation of the Jacobian regularizer can be derived. However, the paper was written in a quite misleading or over-claimed way. The major comments are as follows:
(1)	It is not new to use Jacobian regularizer for improving the robustness learning. Such idea has been elaborated in [1] (though it was cited in the paper). The main contribution is, the paper proposed an efficient approximated way to the exact Jacobian term. All the benefits of robust learning are rooted in the Jacobian regularization.
(2)	In light of point (1), the title of this paper was first quite misleading. Robust learning with Jacobian was not proposed by the paper, so the title makes no sense.  Instead, efficient approximation should be emphasized.
(3)	Most of the advantages shown in the quantitative experiments are the benefits of Jacobian regularization! The authors should focus on the approximating algorithm rather than the merits of Jacobian regularization which has been discussed in [1]. In another word, it is better to add more comparison based on running time so as to illustrate the significant performance between the proposed regularizer and the exact one.
(4)	In section 3, it may be interesting to see more comparison results if two regularizers were combined, e.g., (L2+Dropout, L2+Jacobianor Dropout+Jacobian).
[1] Varga, Dániel, Adrián Csiszárik, and Zsolt Zombori. "Gradient regularization improves accuracy of discriminative models." arXiv preprint arXiv:1712.09936 (2017).
===============
I have read the authors' response. I would have to say that, though I enjoy reading the paper and the other parts of this paper are very good,  the paper's contribution/novelty may be limited because Jacobian regularization has been earlier thoroughly discussed in [1] and even another recent  paper published in ECCV 2018.
I am afraid that I  would firmly keep my original rating.


In this paper, we investigate the problem of training neural machine translation (NMT) systems with more than 40 billion bilingual sentence pairs, which is orders of magnitude larger than the largest dataset to date. Several challenges distinguish this problem from previous NMT works such as the extreme noisiness of the data and prohibitively long training time. We propose practical solutions to handle these issues and demonstrate that large-scale pretraining will significantly improve NMT performances. By merely using vanilla transformer structures, we are able to push the BLEU score of WMT17 Chinese-English dataset to 32.3, with a significant performance boost of +3.2 over existing SOTA results, along with achieving SOTA results on WMT18 (+2.6) and WMT19 (+2.9).

The paper proposes an approach to train NMT models on extremely large parallel corpora. Because of the dataset size, training several epochs on the full dataset with a single model is too expensive. As a result, the dataset is split into several chunks, on which different models are trained independently. The different models are combined to form an ensemble model. Different strategies are proposed to split the dataset effectively. The resulting model achieves a performance of 32.3, outperforming the previous SOTA by 3.2 BLEU.
In Section 4.3, I'm curious about why only 6 layers are used in the encoder and decoder? Large scale pretraining like in BERT usually benefits from very large datasets, but also very large models. 6 layers seems very small given the size of the training set. Table 2 reports results with "Small" and "Large" models. What does this correspond to? Only Section 4.3 discusses the size of the model, but it does not mention different architectural choices.
In Section 5.1 the paper mentions "the single-model achieves a BLEU score of 29.7, already outperforming the current best system", but in Tables 1 and 2 it seems that the best score with single model is 28.7, not 29.7.
I feel that the results are a bit disappointing given the scale of the experiments. Table 2 suggests that a single model, even trained with 40B sentence pairs, does not outperform a single model trained with 20M sentence pairs as in "He et al., 2019", while being significantly more expensive to train. Also, the comparisons in Table 1 are done between single and ensemble models, which is not a fair comparison. The model with a BLEU score of 32.0 uses an ensemble of 10 models. What would be the performance of an ensemble of 10 models trained with the regular 20M parallel sentence pairs?
Also, did you try the approach of "Hassan et al (2018)" suggested in Section 4.1, where only in-domain sentences are selected? It is true that  "every time we encounter a new domain, we have to retrain the model", but I think this is still a more viable approach than pretraining on the full 40B sentences. Why not trying to train on the top-100M sentence pairs that are the most in-domain?
In the back-translation (BT) experiments, did you select 100M monolingual sentences randomly? If that is the case, this is expected to see a drop in performance, BT is usually a very effective, but not so much when the monolingual data is noisy or out of domain. Although it is critical to work with cleaned data in NLP (especially in the context of generation), dataset cleaning is not really addressed in the paper.
Overall, the experimental setup is impressive, but the improvements in terms of BLEU are relatively small, and the technical contributions seem quite thin to me for a ML conference. Moreover, the dataset used in the paper is not available to the research community, which prevents reproducibility. Also, as mentioned above, I think several important experiments are missing.


To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). Our contributions are two-fold. First, we prove that when the gradient is small at every training sample, student node specializes to teacher nodes in the lowest layer under mild conditions. Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training. As a result, it could take a long time to converge into these small-gradient critical points. Our analysis shows that over-parameterization is a necessary condition for specialization to happen at the critical points, and helps student nodes cover more teacher nodes with fewer iterations. Both improve generalization. Different from Neural Tangent Kernel (Jacot et al., 2018) and statistical mechanics approach (Goldt et al., 2019), our approach operates on finite width, mild overparameterization (as long as there are more student nodes than teacher) and finite input dimension. Experiments justify our finding.

The paper studies the role of over-parametrization in the student-teacher multilayer ReLU networks. It presents a theoretical part about properties of SGD critical points for the teacher-student setting. And a heuristic and empirical part on dynamics of the SDG algorithm as a function of properties of the teacher networks. Overall, given previous literature, I do not find the presented results novel nor fundamentally very interesting and some parts are hard to understand due to missing details. I tend to vote for rejection at this point. More detailed questions, comments follow.
In related works:
** Paragraph on "Teacher-student/realizable setting": The recent line of works is interesting, but the authors should be clearer about this being a very classical setting dating back several decades. The first paper I know where the teacher student setting appeared is by Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf). In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a very detailed account of many results on the setting from 80s and 90s.
** "A line of works (Saad & Solla, 1996; 1995; Goldt et al., 2019; Freeman & Saad, 1997; Mace & Coolen, 1998) studied the dynamics from a statistical mechanics point of view, focusing on local analysis near to some critical points." and "(Goldt et al., 2019) assumes Gaussian input and symmetric parameterization to analyze local structure around critical points," The statements that these works focus on local analysis is not correct. While some formal analysis in these works required an infinitesimally informed start toward the teacher the experiments (in particular all those in Goldt et al., 2019) are run from random initialization and these works show empirically that randomly initialized training converges exactly to the fixed points described in the analysis.
** "Local minima is Global" paragraph: This paragraph seems to neglect the empirically observed fact (e.g. https://arxiv.org/pdf/1906.02613.pdf) that there can be global minima that generalize bad. Hence being global does not ensure good generalization.
Body of the paper:
** The authors cite: "Previous works (Ge et al., 2017; Livni et al., 2014) show that empirically SGD does not recover the parameters of a teacher network up to permutation." but they fail to mention that separate line of work, e.g.  (Saad & Solla, 1996; 1995; Goldt et al., 2019) observed empirically the opposite.The different exiting works have to be reconciles and understood and that may be beyond the scope of the present work. But presenting only one side of the results is not helping.
** The part on the dynamics with strong and weak directions reminds me on the results on so called "INCREMENTAL LEARNING" e.g. in the work:
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
also later: https://arxiv.org/pdf/1809.10374.pdf and others.
It would be useful to understand what is the relation in more detail and comment on it.
** The experimental part of the paper has numerous flaws that make it hard to be understood. For instance the authors do not specify the distribution of the input data. Some experiments are run with CIFAR and others with "random" data, but random in which sense? While generalization is the main focus of the paper the experimental results focus on the alignments of the teacher and students without really being clear how specifically the speed or the generalization error improves when neural networks are overparametrised. I found this information only in Fig. 8 for the test error. In Fig. 11 I do not know what are the different panels. What is the parameter p? So I do not know what to conclude from this figure .... in the first pannel the non-overparametrized loss (blue) decreases fastest. In the last pannel all curves are comparable. But this would suggest that over-parametrizatoin is not really helping which seems to go agains the rest of the conclusion in the paper.
** A side remark: I note that the paper is on 10 pages and hence according to the paper call higher standards should be applied in the review process.


Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers. In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima. In a network of d−1 hidden layers with n k neurons in layers k = 1, . . . , d, we construct continuous paths between equivalent global minima that lead through a 'permutation point' where the input and output weight vectors of two neurons in the same hidden layer k collide and interchange. We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape. We also find that a permutation point for the exchange of neurons i and j transits into a flat high-dimensional plateau that enables all n k ! permutations of neurons in a given layer k at the same loss value. Moreover, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of K-th order permutation points is much larger than the (already huge) number of equivalent global minima -at least by a polynomial factor of order K. In two tasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometric approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations.

This paper studies the permutation symmetry of deep neural networks. It was known that by reordering neurons and their connections in each layer, the input -> output map the neural network represents can be preserved. This corresponded to a set of unconnected equivalent points in the weight space. The authors study the weight-space connectivity of these points through midpoints they call “permutation points”. They demonstrate that such points are members of high-dimensional manifolds of equivalent points. After that, they look at empirical experiments and explicitly construct a path between two equivalent weight-space points on a toy task and MNIST.
I generally like the paper and its geometrical lens on the problem. I find the figures very helpful in understanding what is going on. However, there are a few points that I wasn’t entirely clear on. I will detail them below.
-- Point 1 --
Connecting equivalent minima vs connecting SGD-found minima.
If I understood the paper correctly, the derivation connects two weight space points A and B whose weights and biases, once loaded to the neural network, would have the exact same answers on all inputs X i.e. f_A(X) == f_B(X), i.e. they are a pair of equivalent points. I understand that those are the ones we obtain by using the permutation symmetries.
However, some of the papers cited look at the low-loss paths between pairs of optima found by training with SGD from independent initializations, which in turn represent different functions. I.e. for two such optima C and D, the predictions on the val/test set are (sometimes) different, showing that the functions are not the same. I found the initial evidence in:
Large Scale Structure of Neural Network Loss Landscapes. Stanislav Fort and Stanislaw Jastrzebski. NeurIPS 2019. (https://arxiv.org/abs/1906.04724)
And also in much more detail in another OpenReview submission:
Deep Ensembles: A Loss Landscape Perspective. (https://openreview.net/forum?id=r1xZAkrFPr)
I found your results very compelling, however, the two problems seem to be quite different -- on one hand you are connecting a pair of minima that are in fact *identical* by construction. On the other hand the empirical work in literature (especially in the two papers I provided above) deals with pairs of minima that in fact do differ on the test set (at least).
Would you mind commenting on how the two approaches relate to each other?
-- Point 2 --
Higher order connectivity
In Large Scale Structure of Neural Network Loss Landscapes. Stanislav Fort and Stanislaw Jastrzebski. NeurIPS 2019. (https://arxiv.org/abs/1906.04724), the authors look at higher-order connectivity between SGD-found optima (e.g. connecting 3 optima on a 2-manifold, 4 optima on a 3-manifold etc.). They also have a particularly simple path-finding algorithm. This seems relevant to the approach you are presenting, although the points in Point 1 still stand.
-- Point 3 --
Previous work on connecting two optima using layer-wise weight merging
Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets. Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge. (https://arxiv.org/abs/1906.06247)
They prove that a low-loss path between 2 optima exists provided you can apply a p_keep = 0.5 dropout on each of the optima without incurring a significant loss punishment for it. This paper seems very related to your approach. Would you mind commenting on the differences?
-- Conclusion --
I like the paper and the idea in general. I appreciate the geometrical lens the authors took. My main point of confusion relates to the connection between this work and the low-loss connectivity between inequivalent optima found in literature, which (at least to me) seems to be the more interesting of the two connectivities.


We propose a black-box algorithm called Adversarial Variational Inference and Learning (AdVIL) to perform inference and learning in a general Markov random field (MRF). AdVIL employs two variational distributions to approximately infer the latent variables and estimate the partition function of an MRF, respectively. The two variational distributions provide an estimate of the negative log-likelihood of the MRF as a minimax optimization problem, which is solved by stochastic gradient descent. AdVIL is proven convergent under certain conditions. On one hand, compared to the contrastive divergence, AdVIL requires minimal assumptions about the model structure and can deal with a broader family of MRFs. On the other hand, compared to existing black-box methods, AdVIL provides a tighter estimate of the log partition function and achieves much better empirical results.

This paper presents a black-box style learning algorithm for Markov Random Fields (MRF). The approach doubles down on the variational approach with variational approximations for both the positive phase and negative phase of the log likelihood objective function. For the negative phase, the authors use two separate variational approximations, one of which involves the modeling of the latent variable prior under the approximating distribution,
The approach is novel, as far as I know, though not particularly so, and I view this as one of the weak point of the paper. That said, it does seems like a fairly creative combination of existing approaches. As others have found in the past, a variational approximation to the partition function contribution to the loss function (i.e. the negative phase) results in the loss of the variational lower bound on log likelihood and the connection between the resulting approximation and the log likelihood becomes unclear. To deal with this issue, the authors argue (in Lemma 1) that the gradient of their approximate objective is at least in the same direction as the ELBO (lower bound) objective. The result is fairly obvious, but the conditions for validity have interesting consequences for the training algorithm, as it relates the approximation error to the norm of the gradient of the ELBO loss.
I have a minor issue with the discussion (in the last paragraph of sec. 3.2) stating that the theoretical statement of the proposed objective relies on a much weaker assumption than the nonparametric assumption made in the theoretical justification of GANs. While I agree with the statement as such, the GAN development makes a stronger statement about the nature of the learning trajectory. Specifically, it states that the generator is minimizing a Jenson-Shannon divergence which has a fixed point at the true data density. In the current development, Theorem 1 only states that the optimization process will converge to the stationary points of the approximate ELBO objective (L1 in the paper's notation).
Clarity: I found the paper to be very well written with a clear exposition of the material and sound development of the technical details.
Relevance and Significance: This paper is highly relevant to the ICLR community and -- to the extent that one believes that training and inference in MRFs is important -- also significant. One this last point, it seems ironic to me that the proposed strategy for training the MRF is through the use of three separate directed graphical models (an encoder q(h | x),  a decoder and a VAE to model the approximate prior over the latents h). In most modeling situations, one would simply impose the directed graphical model directly and skip the formalization in terms of an MRF. I would appreciate a more forceful motivation of the relevance of MRFs rather than just stating it as a important model with applications. What is unique
about the MRF formalism that -- for practical applications -- could not be effectively captured in a directed graphical model?
I note that I am aware of the theoretical representation differences between directed and undirected models, I am wondering how these differences actually matter in practical applications at scale.
Experiments: The authors show the empirical advantages offered by the proposed method over the existing literature. I was surprised not to see how this model performs on the binarized MNIST dataset, and would like to see that result as well as CIFAR likelihood.  MNIST, in particular, is a well studied dataset that many readers will be able to easily interpret. Its absence seems like a serious omission.
What is meant by "RBM loss" in Fig. 2(d), I do not see this defined?
I am somewhat alarmed at the use of 100 updates of the joint model q(v,h) (K1 = 100) for every update of the other parameters. For larger scale domains, I fear this could become an important obstacle to effective model training. The comparison to PCD-1 in Fig. 3 seems a bit unfair in that the learning curve ends at 8000 iterations, while PCD-1 continues to
improve NLL. I would like to see this curve extended until we start to see signs of overfitting. Perhaps PCD-1 results in performance that is far better than AdVIL. I would also like to see a comparison to CD-k, which often outperforms PCD-k. While I understand the stance taken by the authors that these methods leverage the tractability of the conditional
distributions, these strategies are sufficiently general to be considered widely applicable and a true competitor for AdVIL.
With respect to Deep Boltzmann Machine (DBM), I would prefer to see quantitative comparisons against published results. Here again, MNIST would be a useful dataset.  It seems as though, in the application of AdVIL to the DBM, the authors are exploiting the structure of the model in how they define their sampling procedure. Is that the case? More detail for this application of AdVIL would be nice. Also, I would like to see the test estimated NLL (via AIS) learning curves for VCD and AdVIL. Given the comparison to PCD in the RBM setting, I am somewhat surprised that AdVIL is so competitive with VCD in
the case of the DBM.


Standard compression algorithms work by mapping an image to discrete code using an encoder from which the original image can be reconstructed through a decoder. This process, due to the quantization step, is inherently non-differentiable so these algorithms must rely on approximate methods to train the encoder and decoder end-to-end. In this paper, we present an innovative framework for lossy image compression which is able to circumvent the quantization step by relying on a non-deterministic compression codec. The decoder maps the input image to a distribution in continuous space from which a sample can be encoded with expected code length being the relative entropy to the encoding distribution, i.e. it is bitsback efficient. The result is a principled, end-to-end differentiable compression framework that can be straight-forwardly trained using standard gradient-based optimizers. To showcase the efficiency of our method, we apply it to lossy image compression by training Probabilistic Ladder Networks (PLNs) on the CLIC 2018 dataset and show that their rate-distortion curves on the Kodak dataset are competitive with the state-of-the-art on low bitrates.

Summary:
This paper aims to circumvent the quantization step and associated gradient approximations for compression algorithms that make use of entropy coding for compression. Entropy coding requires a probability mass function over discrete symbols. As an alternative approach the authors adapt the MIRACLE algorithm by Havasi et al. (2018), which was originally used to compress Bayesian neural networks, to work for lossy source compression with probability density functions over continuous latent variables.
The algorithm is based on taking several importance samples from the prior p(z) with the encoding distribution q(z|x) as the target distribution. The number of samples is equal to the exponent of the KL between the encoding and prior distribution, which can quickly grow to an uncontrollably large number. The index of the sample with the maximum importance weight is used as the code for the original image. If both sender and receiver have access to the same random sampler, the receiver can reproduce the sample by drawing a number of samples equal to this index, and decoding the last sample. Similar in spirit to Havasi et al, the authors take several steps to ensure that the KL divergence (and therefore the number of samples) does not become prohibitively large. The compression performance is evaluated by training the proposed model and the competing neural network-based method [1] on the Clic dataset, and evaluating it on a subset of the images of the Kodak dataset. JPEG is also used as a baseline. The authors show results achievable if coding using the relative entropy was perfect (denoted with ‘theoretical’), and the practically achieved compression performance (‘actual’).
Decision:
Weak reject: although the idea of circumventing the quantization step required by the use of entropy coders is certainly valid and interesting, the results in the paper show that the resulting compression performance is worse than the competing method that does quantize. Moreover, the encoding time is significantly longer than this same baseline.
Supporting arguments for decision:
Although the motivation for circumventing the quantization step seems plausible, the authors show no evidence that the competing method [1], which does perform a post-training quantization step, actually suffers from it. The authors even state on page 8 that “ Most notably though, they only used the continuous relaxation during the training of the model, thereafter switching back to quantization and entropy coding, which, they show does not impact the predicted performance, and hence confirming that their relaxation during training is reasonable.“ If post-training quantization is reasonable, then this overthrows the entire motivation. More importantly, the “theoretically” achievable results of the proposed method in seem only competitive in the low-bit rate regime and worse in the higher bit rate regimes. Even more, the “actual” practically achieved compression results are worse than [1] and also considerably worse than the “theoretically” achievable compression results. The authors do provide a reason for why the “theoretical” and “actual” results are so far apart, but are unfortunately not able to overcome this issue.
In the conclusion the authors honestly admit that the runtime of their method is much slower than the competitors (1-5 min for proposed method vs ~0.5 s  for [1] for encoding times). I appreciate that the authors mention this. The authors then state “improving the rate factor and the run-time does not seem too difficult a task, but since the focus of our work was to demonstrate the efficiency of relative entropy coding, it is left for future work.” I do not think this paper demonstrates the efficiency of relative entropy coding, the results simply don’t support this claim, and I therefore think that stating that the issues seem not too difficult to overcome is insufficiently convincing.
The quality of the empirical study can be improved. Another neural network-based compression baseline would make the empirical evaluation of the proposed method more insightful. Now we only see that the result is worse than [1], but it would be good to know how it compares to other baselines such as [2]. Furthermore, the paper does not show compression results aggregated over the entire Kodak dataset, but rather picks 2 images for the main part of the paper, and shows 3 in the appendix. Showing aggregate results gives a more robust estimate of the performance. Individual image results can just be put in the appendix.

Additional feedback to improve paper (not part of decision assessment):
- In section 5 on the dependency structure of latents in the ladder VAE: is it really necessary to indicate the dependency structure with “topological structure”? Seems unnecessary to me as dependency structure is a clear enough description already without making it sound overly complicated.
- Page 9: “Further, our architecture also only uses convolutions and deconvolutions as non-linearities.” Convolutions and deconvolutions are not non-linearities.
- Fig 2: I’m not sure if this figure is relevant enough for such a prominent placement in the paper. It doesn’t discuss anything relevant to the contributions claimed in this paper.
- I can’t find a definition of O in line 3 of “procedure” in algorithm 2 and in the return statement.
[1] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. ICLR 2018.
[2] Lucas Theis et al.  Lossy Image Compression with Compressive Autoencoders. ICLR 2017


Given two views of data, we consider the problem of finding the features of one view which can be most faithfully inferred from the other. We find that these are also the most correlated variables in the sense of deep canonical correlation analysis (DCCA). Moreover, we show that these variables can be used to construct a non-parametric representation of the implied joint probability distribution. This representation can be used to compute the expectations of functions over one view of data conditioned on the other, such as Bayesian estimators and their standard deviations. We test the approach using inference on occluded MNIST images, and show that our representation contains multiple modes. Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective. We also explore using this approach to discover salient independent variables of a single dataset.

The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided.
My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren't very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here.
***
Decision-related suggestions/questions:
* Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting.
* The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful.
* There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples:
	* In the third paragraph under theory, "...linear spaces spanned by the probability distributions..." should probably be "...linear spaces spanned by all probability distributions..." (?)
	* The following sentence is a run-on.
	* In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n).
	* The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1.
	* Introducing a "Bayesian estimator for an l^2 distance" w/o explanation. What does this mean?
* How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it?
* Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features?
* Is there an interpretation for why both spaces require the same feature dimension, k_0?
***
Comments not related to decision:
* It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: "...produce a useful model of correlations... for the task of data generation..."
* Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option.
* The last statement of the paragraph under (3) needs a reference.
* It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of "faster convergence" should be demonstrated in wall-clock time.


Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years-mostly on tasks involving node-level predictions. The setup of prediction tasks over entire graphs (such as property prediction for a molecule, or side-effect prediction for a drug), however, proves to be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction. Most prior work attempts to predict these graph-level properties while considering only one graph at a time-not allowing the learner to directly leverage structural similarities and motifs across graphs. Here we propose a setup in which a graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations to easily exchange structural information across them. We first show that such a setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand to more generic graph regression and graph classification setups: enhancing predictions over QM9, D&D and PROTEINS, standard molecular prediction benchmarks. Our setup is flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.

In this paper, the authors proposed a method to extend graph-based learning with a co-attentional layer. Feeding graphs pairwisely into the model allows nodes to easily exchange information with nodes in other graphs as well as within the graph. This method outperforms other previous ones on a pairwise graph classification task (drug-drug interaction prediction).
This model is generalized from Neural Message Passing for Quantum Chemistry (Justin Gilmer et al.) and Graph Attention Networks (Petar Velickovic et al.), but most ideas are directly from the two previous papers. Combining the two methods do provide insights into understanding the interactions between graphs and get really good results on DDI prediction, but the novelty is limited.
Questions:
1 Are atoms encoded as only atom numbers, charges and connected hydrogen atoms? Because some atoms might have much larger atom numbers than others, e.g. carbon (6) and sulfur (16), will there be some scale problems? Will one-hot encoding of atom type help (like in Neural Message Passing for Quantum Chemistry)?
2 According to the paper, bond types will be encoded as e_{ij}. But in molecules, bond type is way more complex than only single/double/triple bonds, especially for drug molecules which are enriched for aromatic systems. For example, bonds in benzene or pyridine rings are between single and double (also not necessarily 3/2). Are there other possible methods to encode graph edges?
3 In result table 2 of Section 4 (quantum chemistry), I didn’t see a principle of choosing K value and choosing neighbors because different properties reaches the lowest MAE at different K values. This might cause some confusion in real application. Moreover, the authors should compare the performance with previous methods.


Gaining more comprehensive knowledge about drug-drug interactions (DDIs) is one of the most important tasks in drug development and medical practice. Recently graph neural networks have achieved great success in this task by modeling drugs as nodes and drug-drug interactions as links and casting DDI predictions as link prediction problems. However, correlations between link labels (e.g., DDI types) were rarely considered in existing works. We propose the graph energy neural network (GENN) to explicitly model link type correlations. We formulate the DDI prediction task as a structure prediction problem, and introduce a new energy-based model where the energy function is defined by graph neural networks. Experiments on two real world DDI datasets demonstrated that GENN is superior to many baselines without consideration of link type correlations and achieved 13.77% and 5.01% PR-AUC improvement on the two datasets, respectively. We also present a case study in which GENN can better capture meaningful DDI correlations compared with baseline models.

This paper proposes a framework to learn correlated drug-drug interaction based on structured prediction energy networks (SPEN). The core idea is to model the dependency structure of the labels (multi-label) by minimizing a designed energy function. The graph energy is designed as MLP over the mean of all nodes embeddings, where the nodes embeddings are obtained through a graph convolutional network. The edge information is included in the node embedding when aggregating neighborhood information. The proposed method also introduces an additional test inference network to jointly train with the cost-augmented training network under the semi-supervised setting. The authors tested on two DDI datasets and the result shows improvement compared to several baseline methods.
Strengths
- motivation: the authors consider the correlations between DDI labels (multi-label) which could potentially improve prediction of DDI. The proposed method uses structure prediction energy networks to model such dependency.
- method: the proposed method introduce an additional test inference network to fit the energy minimization framework into a semi-supervised setting.
Weakness
- The core part of the proposed method which differs from previous works needs to be elaborated more.
- The performance improvement seems to be marginal, especially on the second dataset.
Detailed comments
- The work is based on structured prediction energy networks (SPEN, Belanger, et al. 2016) and its follow-up works (eg., Belanger et al. 2016, Lifu Tu et al. 2018), the overall framework is very similar with previous work, where the feature extraction part is replaced by a graph convolutional network module. The core difference lies in the additional testing inference network which is jointly trained for adapting supervised SPEN into a semi-supervised setting. This part actually differs from all previous works and needs to be elaborated more. Why does such design make sense and what benefit can be brought through such design? The formulation for semi-supervised SPEN could be defined more clearly and worth elaboration.
- The experiments are run on 3 different random splits, based on the mean(std) of the evaluation metric, the performance of the proposed method does not vary much compared to baselines, especially on the second dataset, eg., GNN 0.25 +/- 0.02 compared with GENN 0.26. Also, GLENN < GNN seems to imply including energy is not the most important part for helping the task, but rather the semi-supervised joint training truly improves the performance.


Vanilla RNN with ReLU activation have a simple structure that is amenable to systematic dynamical systems analysis and interpretation, but they suffer from the exploding vs. vanishing gradients problem. Recent attempts to retain this simplicity while alleviating the gradient problem are based on proper initialization schemes or orthogonality/unitary constraints on the RNN's recurrence matrix, which, however, comes with limitations to its expressive power with regards to dynamical systems phenomena like chaos or multi-stability. Here, we instead suggest a regularization scheme that pushes part of the RNN's latent subspace toward a line attractor configuration that enables long short-term memory and arbitrarily slow time scales. We show that our approach excels on a number of benchmarks like the sequential MNIST or multiplication problems, and enables reconstruction of dynamical systems which harbor widely different time scales.

Overview
This paper proposes a type of regularization for recurrent networks, with the goal of encouraging particular dynamical structures (in this case, line attractors) in the dynamics of the networks. The proposed regularization penalty is only applied to a subset of the recurrent units; the motivation for this is to allow neurons not contained in the subset to learn different structures. The paper applies this regularization method on three example machine learning sequence tasks: an addition task, a multiplication task, and sequential MNIST classification, as well as on learning a 2-D dynamical system model of a bursting neuron with two different timescales.
Major comments
I have a number of serious concerns about the paper's motivation, logic, and experiments:
- First, the paper motivates the proposed regularization as a way to encourage the network to have line attractor dynamics. In particular, the paper dismisses gated architectures as not being interpretable, stating that LSTMs and GRUs "are complicated and tedious to analyze from a DS perspective." (pg 2). However, there is recent work both theoretical (https://arxiv.org/abs/1906.01005) and empirical (https://arxiv.org/abs/1907.08549) that analyzes these gated architectures from a DS perspective. In particular, these papers demonstrate that LSTMs and GRUs are perfectly capable of learning line attractors. Given that it is possible to analyze gated architectures as dynamical systems, the overall motivation of the paper is much weaker.
- The paper proposes a squared penalty on subsets of the weights in the recurrent network as a way to encourage line attractor dynamics. However, it is not clear to me that this is sufficient. In particular, unless the subset of the network that implements the line/plane attractor is completely disconnected from the rest of the network, then the overall dynamics may not contain a line attractor (the units will interact with the unregularized units). Also, the proposed regularization penalty only penalizes the diagonal elements of the A matrix to be close to 1--but shouldn't the off diagonal elements also be penalized to be close to zero?
- Moreover, the paper makes no mention of the Jacobian of these recurrent networks. The eigenvalues of the Jacobian of the recurrent networks determine the behavior of the linearized system around fixed points--specifically, eigenvalues with real part close to 1 will exhibit slow dynamics (approximate line attractors along those dimensions). It seems to me that a much more natural way of encouraging line attractor dynamics is to place a regularization penalty on the Jacobian itself (which is analytically tricky, but numerically more plausible with modern autodifferentiation software). Regardless, the authors should compare the eigenvalues of the recurrent networks' Jacobian when using their regularization method vs without it. Does the proposed regularization encourage the Jacobian of the resulting networks to have eigenvalues close to 1?
- The paper compares the proposed method with a number of vanilla RNNs with different initializations, and an LSTM. However, a critical missing baseline is simply an RNN with l2 regularization on the weights (standard regularization in the literature). This baseline is important to determine if the proposed regularization simply helps because it is an l2 penalty on the weights (note that none of the other baselines have regularization).
- The paper motivates the method as trying to study line attractor dynamics, but then does not apply them to tasks where line attractors are required. For example, the addition and multiplication tasks require discrete memories, not line attractors. The bursting neuron approximation (2D dynamical system) also does not involve a line attractor. However, there definitely exist tasks both in neuroscience (e.g. sensory integration in decision making, path integration in navigation, etc.) and in machine learning (c.f. https://arxiv.org/abs/1906.10720) that use or require line attractors. The motivation of the paper would be much better tested on these tasks.
Minor comments
- The authors comment at the beginning of page 4 that by setting A=I, W=0, and h=0, the network contains a line attractor, but the more precise language would be to state that the network contains an N-dimensional plane attractor, where N is the number of units. Typically, 'line attractor' refers to a 1-dimensional manifold of fixed points along which the system can integrate inputs, but perturbations off of the line attractor are not remembered (decay back to the line attractor).