We design a new policy, called a nearest neighbor policy, that does not require any optimization for simple, low-dimensional continuous control tasks. As this policy does not require any optimization, it allows us to investigate the underlying difficulty of a task without being distracted by optimization difficulty of a learning algorithm. We propose two variants, one that retrieves an entire trajectory based on a pair of initial and goal states, and the other retrieving a partial trajectory based on a pair of current and goal states. We test the proposed policies on five widely-used benchmark continuous control tasks with a sparse reward: Reacher, Half Cheetah, Double Pendulum, Cart Pole and Mountain Car. We observe that the majority (the first four) of these tasks, which have been considered difficult, are easily solved by the proposed policies with high success rates, indicating that reported difficulties of them may have likely been due to the optimization difficulty. Our work suggests that it is necessary to evaluate any sophisticated policy learning algorithm on more challenging problems in order to truly assess the advances from them.

This work shows that a simple non-parametric approach of storing state embeddings with the associated Monte Carlo returns is sufficient to solve several benchmark continuous control problems with sparse rewards (reacher, half-cheetah, double pendulum, cart pole) (due to the need to threshold a return the algorithms work less well with dense rewards, but with the introduction of a hyper-parameter is capable of solving several tasks there). The authors argue that the success of these simple approaches on these tasks suggest that more changing problems need to be used to assess new RL algorithms.
This paper is clearly written and it is important to compare simple approaches on benchmark problems. There are a number of interesting and intriguing side-notes and pieces of future work mentioned.
However, the originality and significance of this work is a significant drawback. The use non-parametric approaches to the action-value function go back to at least [1] (and probably much further). So the algorithms themselves are not particularly novel, and are limited to nearly-deterministic domains with either single sparse rewards (success or failure rewards) or introducing extra hyper-parameters per task.
The significance of this work would still be quite strong if, as the author's suggest, these benchmarks were being widely used to assess more sophisticated algorithms and yet these tasks were mastered by such simple algorithms with no learnable parameters. Yet, the results do not support the claim. Even if we ignore that for most tasks only the sparse reward (which favors this algorithm) version was examined, these author's only demonstrate success on 4, relatively simple tasks.
While these simple tasks are useful for diagnostics, it is well-known that these tasks are simple and, as the author's suggest "more challenging tasks  .... are necessary to properly assess advances made by sophisticated, optimization-based policy algorithms." Lillicrap et al. (2015) benchmarked against 27 tasks, Houtfout et al. (2016) compared in the paper also used Walker2D and Swimmer (not used in this paper) as did [2], OpenAI Gym contains many more control environments than the 4 solved here and significant research is pursing complex manipulation and grasping tasks (e.g. [3]). This suggests the author's claim has already been widely heeded and this work will be of limited interest.
[1] Juan, C., Sutton, R. S., & Ram, A. Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces.
[2] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017). Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560.
[3] Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2017). Overcoming exploration in reinforcement learning with demonstrations. arXiv preprint arXiv:1709.10089.


We present a Neural Program Search, an algorithm to generate programs from natural language description and a small number of input / output examples. The algorithm combines methods from Deep Learning and Program Synthesis fields by designing rich domain-specific language (DSL) and defining efficient search algorithm guided by a Seq2Tree model on it. To evaluate the quality of the approach we also present a semi-synthetic dataset of descriptions with test examples and corresponding programs. We show that our algorithm significantly outperforms sequence-to-sequence model with attention baseline.

This paper presents a seq2Tree model to translate a problem statement in natural
language to the corresponding functional program in a DSL. The model uses
an RNN encoder to encode the problem statement and uses an attention-based
doubly recurrent network for generating tree-structured output. The learnt model is
then used to perform Tree-beam search using a search algorithm that searches
for different completion of trees based on node types. The evaluation is performed
on a synthetic dataset and shows improvements over seq2seq baseline approach.
Overall, this paper tackles an important problem of learning programs from
natural language and input-output example specifications. Unlike previous
neural program synthesis approaches that consider only one of the specification
mechanisms (examples or natural language), this paper considers both of them
simultaneously. However, there are several issues both in the approach and the
current preliminary evaluation, which unfortunately leads me to a reject score,
but the general idea of combining different specifications is quite promising.
First, the paper does not compare against a very similar approach of Parisotto et al.
Neuro-symbolic Program Synthesis (ICLR 2017) that uses a similar R3NN network
for generating the program tree incrementally by decoding one node at a time.
Can the authors comment on the similarity/differences between the approaches?
Would it be possible to empirically evaluate how the R3NN performs on this dataset?
Second, it seems that the current model does not use the input-output examples at
all for training the model. The examples are only used during the search algorithm.
Several previous neural program synthesis approaches (DeepCoder (ICLR 2017),
RobustFill (ICML 2017)) have shown that encoding the examples can help guide
the decoder to perform efficient search. It would be good to possibly add another
encoder network to see if encoding the examples as well help improve the accuracy.
Similar to the previous point, it would also be good to evaluate the usefulness of
encoding the problem statement by comparing the final model against a model in which
the encoder only encodes the input-output examples.
Finally, there is also an issue with the synthetic evaluation dataset. Since the
problem descriptions are generated syntactically using a template based approach,
the improvements in accuracy might come directly from learning the training templates
instead of learning the desired semantics. The paper mentions that it is prohibitively
expensive to obtain human-annotated set, but can it be possible to at least obtain a
handful of real tasks to evaluate the learnt model? There are also some recent
datasets such as WikiSQL (https://github.com/salesforce/WikiSQL) that the authors
might consider in future.
Questions for the authors:
Why was MAX_VISITED only limited to 100? What happens when it is set to 10^4 or 10^6?
The Search algorithm only shows an accuracy of 0.6% with MAX_VISITED=100. What would
the performance be for a simple brute-force algorithm with a timeout of say 10 mins?
Table 3 reports an accuracy of 85.8% whereas the text mentions that the best result
is 90.1% (page 8)?
What all function names are allowed in the DSL (Figure 1)?
Can you clarify the contributions of the paper in comparison to the R3NN?
Minor typos:
page 2: allows to add constrains --> allows to add constraints
page 5: over MAX_VISITED programs has been --> over MAX_VISITED programs have been


Machine reading has recently shown remarkable progress thanks to differentiable reasoning models. In this context, End-to-End trainable Memory Networks (MemN2N) have demonstrated promising performance on simple natural language based reasoning tasks such as factual reasoning and basic deduction. However, the task of machine comprehension is currently bounded to a supervised setting and available question answering dataset. In this paper we explore the paradigm of adversarial learning and self-play for the task of machine reading comprehension. Inspired by the successful propositions in the domain of game learning, we present a novel approach of training for this task that is based on the definition of a coupled attention-based memory model. On one hand, a reader network is in charge of finding answers regarding a passage of text and a question. On the other hand, a narrator network is in charge of obfuscating spans of text in order to minimize the probability of success of the reader. We experimented the model on several question-answering corpora. The proposed learning paradigm and associated models present encouraging results.

Summary:
This paper proposes an adversarial learning framework for machine comprehension task. Specifically, authors consider a reader network which learns to answer the question by reading the passage and a narrator network which learns to obfuscate the passage so that the reader can fail in its task. Authors report results in 3 different reading comprehension datasets and the proposed learning framework results in improving the performance of GMemN2N.
My Comments:
This paper is a direct application of adversarial learning to the task of reading comprehension. It is a reasonable idea and authors indeed show that it works.
1. The paper needs a lot of editing. Please check the minor comments.
2. Why is the adversary called narrator network? It is bit confusing because the job of that network is to obfuscate the passage.
3. Why do you motivate the learning method using self-play? This is just using the idea of adversarial learning (like GAN) and it is not related to self-play.
4. In section 2, first paragraph, authors mention that the narrator prevents catastrophic forgetting. How is this happening? Can you elaborate more?
5. The learning framework is not explained in a precise way. What do you mean by re-initializing and retraining the narrator? Isn’t it costly to reinitialize the network and retrain it for every turn? How many such epochs are done? You say that test set also contains obfuscated documents. Is it only for the validation set? Can you please explain if you use obfuscation when you report the final test performance too? It would be more clear if you can provide a complete pseudo-code of the learning procedure.
6. How does the narrator choose which word to obfuscate? Do you run the narrator model with all possible obfuscations and pick the best choice?
7. Why don’t you treat number of hops as a hyper-parameter and choose it based on validation set? I would like to see the results in Table 1 where you choose number of hops for each of the three models based on validation set.
8. In figure 2, how are rounds constructed? Does the model sees the same document again and again for 100 times or each time it sees a random document and you sample documents with replacement? This will be clear if you provide the pseudo-code for learning.
9. I do not understand author's’ justification for figure-3. Is it the case that the model learns to attend to last sentences for all the questions? Or where it attends varies across examples?
10. Are you willing to release the code for reproducing the results?
Minor comments:
Page 1, “exploit his own decision” should be “exploit its own decision”
In page 2, section 2.1, sentence starting with “Indeed, a too low percentage …” needs to be fixed.
Page 3, “forgetting is compensate” should be “forgetting is compensated”.
Page 4, “for one sentences” needs to be fixed.
Page 4, “unknow” should be “unknown”.
Page 4, “??” needs to be fixed.
Page 5, “for the two first datasets” needs to be fixed.
Table 1, “GMenN2N” should be “GMemN2N”. In caption, is it mean accuracy or maximum accuracy?
Page 6, “dataset was achieves” needs to be fixed.
Page 7, “document by obfuscated this word” needs to be fixed.
Page 7, “overall aspect of the two first readers” needs to be fixed.
Page 8, last para, references needs to be fixed.
Page 9, first sentence, please check grammar.
Section 6.2, last sentence is irrelevant.


Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter. Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains. We introduce XGAN, a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions. We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space. We report promising qualitative results for the task of face-to-cartoon translation. The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

This paper proposes a new GAN-based model for unpaired image-to-image translation. The model is very similar to DTN [Taigman et al. 2016] except with trained encoders and a domain confusion loss to encourage the encoders to map source and target domains to a shared embedding. Additionally, an optional teacher network is introduced, but this feels rather tangential and problem-specific. The paper is clearly presented and I enjoyed the aesthetics of the figures. The method appears technically sound, albeit a bit complicated. The new cartoon dataset is also a nice contribution.
My main criticism of this paper is the experiments. At the end of reading, I don’t know clearly which aspects of the method are important, why they are important, and how the proposed system compares against past work. First, the baselines are insufficient. Only DTNs are compared against, yet there are many other recent methods for unpaired image-to-image translation, notably, cycle-consistency-based methods and UNIT. These methods should also be compared against, as there is little evidence that DTNs are actually SOTA on cartoons (rather, the cartoon dataset was not public so other papers did not compare on that dataset). Second, although I appreciated the ablation experiments, they are not comprehensive, as discussed more below. Third, there is no quantitative evaluation. The paper states that quantifying performance on style transfer is an unsolved problem, but this is no excuse for not at least trying. Indeed, there are many proposed metrics in the literature for quantifying style transfer / image generation, including the Inception score [Salimans et al. 2016], conditional variants like the FCN-score [Isola et al. 2017], and human judgments. These metrics could all be adapted to the present task (with appropriate modifications, e.g., switching from Inception to a face attribute classifier). Additionally, as the paper mentions at the end, the method could be applied to domain adaptation, where plenty of standard metrics and benchmarks exist.
Ultimately, the qualitative results in the paper are not convincing to me. It’s hard to see the advantages/disadvantages in each comparison. For example in Figure 8, it’s hard to even see any overall change in the outputs due to ablating the semantic consistency loss and the teacher loss (especially since I’m comparing these to Figure 6, which is referred to “Selected results” and therefore might not be a fair comparison). Perhaps the effect of the ablations would be clearer if the figures showed a single input followed by a series of outputs for that same input, each with a different term ablated. A careful reader might be able to examine the images for a long time and find some insights, but it would be much better if the paper distilled these insights into a more concise and convincing form. I feel sort of like I’m looking at raw data, and it still needs to be analyzed.
I also think the ablations are not sufficiently comprehensive. In particular, there is no ablation of the domain adversarial loss. This seems like an important one to test since it’s one of the main differences from DTNs. I was a bit confused by the “finetuned DTN” in Section 7.2. Is this an ablation experiment where the domain adversarial loss and teacher loss are removed? If so, referring to it as so may be clearer than calling it a finetuned DTN. Interestingly, the results of this method look pretty decent, suggesting that the domain adversarial loss might not be having a big effect, in which case XGAN looks very close indeed to DTNs. It would be great here to actually quantify the mentioned sensitivity to hyperparameters.
In terms of presentation, at several points, the paper argues that previous, pixel-domain methods are more limited than the proposed feature-space method, but little evidence is given to support these claims. For example, “we argue that such a pixel-level constraint is not sufficient in our case” in the intro, and “our proposed semantic consistency loss acts at the feature level, allowing for more flexible transformations” in related work. I would like to see more motivation for these assertions, and ultimately, the limitations should be concretely demonstrated in experiments. In models like CycleGAN the pixel-level constraint is between inputs and reconstructed inputs, and I don’t see why this necessarily is overly restrictive on the kinds of transformations in the outputs. The phrasing in the current paper seems to suggest that the pixel-level constraints are between input and output, which, I agree, would be directly restrictive. The reasoning here should be clarified. Better yet would be to provide empirical evidence that pixel-domain methods are not successful (e.g., by comparing against CycleGAN).
The usage of the term “semantic” is also somewhat confusing. In what sense is the latent space semantic? The paper should clarify exactly what this term refers to, perhaps simply defining it to mean a “low-dimensional shared embedding.”
I think the role of the GAN objective is somewhat underplayed. It is quite interesting that the current model achieves decent results even without the GAN. However, there is no experiment keeping the GAN but ablating other parts of the method. Other papers have shown that a GAN objective plus, e.g., cycle-consistency, can do quite well on this kind of problem. It could be that different terms in the current objective are somewhat redundant, so that you can choose any two or three, let’s say, and get good results. To check this, it would be great to see more comprehensive ablation experiments.
Minor comments:
1. Page 1: I wouldn’t call colorization one-to-one. Even though there is a single ground truth, I would say colorization is one-to-many in the sense that many outputs may be equally probable according to a Bayes optimal observer.
2. Fig 1: It should be clarified that the left example is not a result of the method. At a glance this looks like an exciting new result and I think that could mislead casual readers.
3. Fig 1 caption: “an other” —> “another”
4. Page 2: “Recent work … fail for more general transformations” — DiscoGAN (Kim et al. 2017) showed some success beyond pixel-aligned transformations
5. Page 5: “particular,the” —> “particular, the”; quotes around “short beard” are backwards
6. Page 6: “founnd” —> “found”
7. Page 11: what is \mathcal{L}_r? I don’t see it defined above.


Predictive models that generalize well under distributional shift are often desirable and sometimes crucial to machine learning applications. One example is the estimation of treatment effects from observational data, where a subtask is to predict the effect of a treatment on subjects that are systematically different from those who received the treatment in the data. A related kind of distributional shift appears in unsupervised domain adaptation, where we are tasked with generalizing to a distribution of inputs that is different from the one in which we observe labels. We pose both of these problems as prediction under a shift in design. Popular methods for overcoming distributional shift are often heuristic or rely on assumptions that are rarely true in practice, such as having a well-specified model or knowing the policy that gave rise to the observed data. Other methods are hindered by their need for a pre-specified metric for comparing observations, or by poor asymptotic properties. In this work, we devise a bound on the generalization error under design shift, based on integral probability metrics and sample re-weighting. We combine this idea with representation learning, generalizing and tightening existing results in this space. Finally, we propose an algorithmic framework inspired by our bound and verify is effectiveness in causal effect estimation.

The paper proposes a novel way of causal inference in situations where in causal SEM notation the outcome Y = f(T,X) is a function of a treatment T and covariates X. The goal is to infer the treatment effect E(Y|T=1,X=x) - E(Y|T=0,X=x) for binary treatments at every location x. If the treatment effect can be learned, then forecasts of Y under new policies that assign treatment conditional on X will still "work" and the distribution of X can also change without affecting the accuracy of the predictions.
What is proposed seems to be twofold:
- instead of using a standard inverse probability weighting, the authors construct a bound for the prediction performance under new distributions of X and new policies and learn the weights by optimizing this bound. The goal is to avoid issues that arise if the ratio between source and target densities become very large or small and the weights in a standard approach would become very sparse, thus leading to a small effective sample size.
- as an additional ingredient the authors also propose "representation learning" by mapping x to some representation Phi(x).
The goal is to learn the mapping Phi (and its inverse) and the weighting function simultaneously by optimizing the derived bound on the prediction performance.
Pros:
- The problem is relevant and also appears in similar form in domain adaptation and transfer learning.
- The derived bounds and procedures are interesting and nontrivial, even if there is some overlap with earlier work of Shalit et al.
Cons:
- I am not sure if ICLR is the optimal venue for this manuscript but will leave this decision to others.
- The manuscript is written in a very compact style and I wish some passages would have been explained in more depth and detail. Especially the second half of page 5 is at times very hard to understand as it is so dense.
- The implications of the assumptions in Theorem 1 are not easy to understand, especially relating to the quantities B_\Phi, C^\mathcal{F}_{n,\delta} and D^{\Phi,\mathcal{H}}_\delta. Why would we expect these quantities to be small or bounded? How does that compare to the assumptions needed for standard inverse probability weighting?
- I appreciate that it is difficult to find good test datasets for evaluating causal estimator.  The experiment on the semi-synthetic IHDP dataset is ok, even though there is very little information about its structure in the manuscript (even basic information like number of instances or dimensions seems missing?). The example does not provide much insight into the main ideas and when we would expect the procedure to work more generally.


FLIPOUT: EFFICIENT PSEUDO-INDEPENDENT WEIGHT PERTURBATIONS ON MINI-BATCHES
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services.

In this article, the authors offer a way to decrease the variance of the gradient estimation in the training of neural networks.
They start in the Introduction and Section 2 by explaining the multiple uses of random connection weights in deep learning and how the computational cost often restricts their use to a single randomly sampled set of weights per minibatch, which results to higher-variance gradient estimatos than could be achieved otherwise. In Section 3 the authors offer to get the benefits of multiple weights without most of the cost, when the distribution of the weights is symmetric and fully factorized, by multiplying sampled-once random perturbations of the weights by a rank-1 random sign matrix. This efficient mechanism is only twice as costly as a single random perturbation, and the authors show how to efficiently parallelize it on GPUs, thereby also allowing GPU-ization of evolution strategies (something so far difficult toachieve). Of note, they provide a theoretical analysis in Section 3.2, proving the actual variance reduction of their efficient pseudo-sampling scheme. In Section 4 they provide quite varied empirical analysis: they confirm their theoretical results on four architectures; they show its use it to regularise on language models; they apply it on large minibatch settings where high variance is a main problem; and on evolution strategies.
While it is a rather simple idea which could be summarised much earlier in the  single equation (3), I really like the thoroughness and the clarity of the exposure of the idea. Too many papers in our community skimp on details and on formalism, and it is a delight to see things exposed so clearly -- even accompanied with a proof.
However, the painful part: while I am convinced by the idea and love its detailed exposure, and the gradient variance reduction is made very clear, the experimental impact in terms of accuracy (or perplexity) is, sadly,  not very convincing. Nowhere in the text did I find a clear rationale of why it is beneficial to reduce the variance of the gradient. The numerical results in Table 1 and Table 2 also do not show a clear improvement: Flipout does not provide the best accuracy. The gain in wall clock could be a factor, but would need to be measured on the figures more clearly. And the validation errors in Figure 2 for Evolution strategies seem to be worse than backprop.The main text itself also only claims performance “comparable to the other methods”.  The only visible gain is on the lower part Figure 2.a on a ConvNet.
This makes me wonder if the authors could do a better job of putting forward the actual advantages of their methods on the end-results: could wall clock measure be put more forward, to justify the extra work? This would, in my mind, strongly improve the case for publication of this article.
A few improvement suggestions:
* Could put earlier more emphasis of superiority to Local Reparameterization Trick in terms of architecture, not wait until Section 2.2 and section 4.1
*Should also put more emphasis on limitations, not wait until 3.1.
* Proposition 1 is quite straightforward, not sure it deserves a proposition, but it’s elegant to put it forward.
* Footnote 1 on re-using the matrices is indeed practical, but also somewhat surprising in terms of bias risks. Could it be explained in more depth, maybe by the random permutations of the minibatches making the bias non systematic and cancelling out?
* Theorem 1: For readability could merge the expectations on the joint distribution as E_{x, \hat \delta W} , rather than separate expectations with the conditional distributions.
* Theorem 1: could the authors provide a clearer intuitive explanation of the \beta term alone, not only as part of \alpha + \beta, especially as it plays such a key role, being the only one that does not disappear? And how do they explain their empirical observation that \beta is close to 0? Any intuition on that?
* Experiments: I salute the authors for providing all the details in exhaustive manner in the Appendix. Very commendable.
* Experiments: I like the empirical verification of the theory. Very neat to see.
Minor typo:
* page 2 last paragraph, “evolution strategies” is plural but the verbs are used in singular (“is black box”, “It doesn’t”, “generates”)


Published as a conference paper at ICLR 2018 VARIATIONAL NETWORK QUANTIZATION
In this paper, the preparation of a neural network for pruning and few-bit quantization is formulated as a variational inference problem. To this end, a quantizing prior that leads to a multi-modal, sparse posterior distribution over weights, is introduced and a differentiable Kullback-Leibler divergence approximation for this prior is derived. After training with Variational Network Quantization, weights can be replaced by deterministic quantization values with small to negligible loss of task accuracy (including pruning by setting weights to 0). The method does not require fine-tuning after quantization. Results are shown for ternary quantization on LeNet-5 (MNIST) and DenseNet (CIFAR-10).

This paper presents Variational Network Quantization; a variational Bayesian approach for quantising neural network weights to ternary values post-training in a principled way. This is achieved by a straightforward extension of the scale mixture of Gaussians perspective of the log-uniform prior proposed at [1]. The authors posit a mixture of delta peaks hyperprior over the locations of the Gaussian distribution, where each peak can be seen as the specific target value for quantisation (including zero to induce sparsity). They then further propose an approximation for the KL-divergence, necessary for the variational objective, from this multimodal prior to a factorized Gaussian posterior by appropriately combining the approximation given at [2] for each of the modes. At test-time, the variational posterior for each weight is replaced by the target quantisation value that is closest, w.r.t. the squared distance, to the mean of the Gaussian variational posterior. Encouraging experimental results are shown with performance comparable to the state-of-the-art for ternary weight neural networks.
This paper presented a straightforward extension of the work done at [1, 2] for ternary networks through a multimodal quantising prior. It is generally well-written, with extensive preliminaries and clear equations. The visualizations also serve as a nice way to convey the behaviour of the proposed approach. The idea is interesting and well executed so I propose for acceptance. I only have a couple of minor questions:
- For the KL-divergence approximation you report a maximum difference of 1 nat per weight that seems a bit high; did you experiment with the `naive` Monte Carlo approximation of the bound (e.g. as done at Bayes By Backprop) during optimization? If yes, was there a big difference in performance?
- Was pre-training necessary to obtain the current results for MNIST? As far as I know, [1] and [2] did not need pre-training for the MNIST results (but did employ pre-training for CIFAR 10).
- How necessary was each one of the constraints during optimization (and what did they prevent)?
- Did you ever observe posterior means that do not settle at one of the prior modes but rather stay in between? Or did you ever had issues of the variance growing large enough, so that q(w) captures multiple modes of the prior (maybe the constraints prevent this)? How sensitive is the quantisation scheme?
Other minor comments / typos:
(1) 7th line of section 2.1 page 2, ‘a unstructured data’ -> ‘unstructured data’
(2) 5th line on page 3, remove ‘compare Eq. (1)’ (or rephrase it appropriately).
(3) Section 2.2, ’Kullback-Leibler divergence between the true and the approximate posterior’; between implies symmetry (and the KL isn’t symmetric) so I suggest to change it to e.g. ‘from the true to the approximate posterior’ to avoid confusion. Same for the first line of Section 3.3.
(4) Footnote 2, the distribution of the noise depends on the random variable so I would suggest to change it to a general \epsilon \sim p(\epsilon).
(5) Equation 4 is confusing.
[1] Louizos, Ullrich & Welling, Bayesian Compression for Deep Learning.
[2] Molchanov, Ashukha & Vetrov, Variational Dropout Sparsifies Deep Neural Networks.


Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away. This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds. Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations. Prior work shows that, if the neural dynamics are in the 'null space' of the representation -so that changes to neural activity do not affect the downstream read-out of stimulus information -then information can be retained for periods much longer than the time-scale of individual-neuronal activities. The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network. To identify mechanisms through which biological networks can self-organize to learn memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing. Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.

A neural network model consisting of recurrently connected neurons and one or more readouts is introduced which aims to retain some output over time. A plasticity rule for this goal is derived. Experiments show the robustness of the network with respect to noisy weight updates, number of non-plastic connections, and sparse connectivity. Multiple consecutive runs increase the performance; furthermore, remembering multiple stimuli is possible. Finally, ideas for the biological implementation of the rule are suggested.
While the presentation is generally comprehensible a number of errors and deficits exist (see below). In general, this paper addresses a question that seems only relevant from a neuroscience perspective. Therefore, I wonder whether it is relevant in terms of the scope of this conference. I also think that the model is rather speculative. The authors argue that the resulting learning rule is biologically plausible. But even if this is the case, it does not imply that it is implemented in neuronal circuits in the brain. As far as I can see, there exists no experimental evidence for this rule.
The paper shows the superiority of the proposed model over the approach of Druckmann & Chkolvskii (2012), however, it lacks in-depth analysis of the network behavior. Specifically, it is not clear how the information is stored. Do neurons show time-varying responses as in Druckmann & Chkolvskii (2012) or do all neuron stabilize within the first 50 ms (as in Fig. 2A, it is not detailed how the neurons shown there have been selected)? Do the weights change continuously within the delay period or do they also converge rapidly? This question is particularily important when considering multiple consecutive trials (cf. Fig. 5) as it seem that a specific but constant network architecture can retain the desired stimulus without further plasticity. Weight histograms should be presented for the different cases and network states. Also, since autapses are allowed, an analysis of their role should be performed. This information is vital to compare the model to findings from neuroscience and judge the biologic realism.
The target used is \hat{s}(t) / \hat{s}(t = 0), this is dubbed "fraction of stimulus retained". In most plots, the values for this measure are <= 1, but in Fig. 3A, the value (for the FEVER network) is > 1. Thus, the name is arguably not well-chosen: how can a fraction of remembrance be greater than one? Also, in a realistic environment, it is not clear that the neuronal activities decay to zero (resulting in \hat{s}(t) also approaching zero). A squared distances measure should therefore be considered.
It is not clear from the paper when and how often weight updates are performed. Therefore, the biologic plausability cannot be assessed, since the learning rule might lead to much more rapid changes of weights than the known learning rules in biological neural networks. Since the goal seems to be biologic realism, generally, spiking neurons should be used for the model. This is important as spiking neural networks are much more fragile than artificial ones in terms of stability.
Further remarks:
- In Sec. 3.2.1, noise is added to weight updates. The absolute values of alpha are hard to interpret since it is not clear in what range the weights, activities, and weight updates typically lie.
- In Sec. 3.2.2 it is shown that 10% plastic synapses is enough for reasonable performance. In this case, it should be investigated whether the full network is essential for the memory task at all (especially since later, it is argued that 100 neurons can store up to 100 stimuli).
- For biologic realism, just assuming that the readout value at t = 0 is the target seems a bit too simple. How does this output arise in the first place? At least, an argument for this choice should be presented.
Remarks on writing:
- Fig. 1A is too small to read.
- The caption of Fig. 4C is missing.
- In Fig. 7AB, q_i and q_j are swapped. Also, it is unclear in the figure to which connection the ds and qs belong.
- In 3.6.1, Fig. 7 is referenced, but in the figure the terminology of Eq. 5 is used, which is only introduced in Sec. 3.6.2. This is confusing.
- The beginning of Sec. 3.6 claims that all information is local except d\hat{s}_k / dt, but this is not the case as d_i is not local (which is explained later).
- The details of the "stimulus presentation" (i.e. it is not performed explicitly) should be emphasised in 2.1. Also, the description of the target \hat{s} is much clearer in 3.4 than in 2.1 (where it should primarily be explained).
- The title of the citation Cowan (2010) is missing.
- In Fig. 2A, the formulas are too small too read in a printed version.
- In Sec. 3.6.1 some sums are given over k, but k is also the index of a neuron in Fig. 7A (which is referenced there), this can be ambiguous and could be changed.


Training generative adversarial networks is unstable in high-dimensions as the true data distribution tends to be concentrated in a small fraction of the ambient space. The discriminator is then quickly able to classify nearly all generated samples as fake, leaving the generator without meaningful gradients and causing it to deteriorate after a point in training. In this work, we propose training a single generator simultaneously against an array of discriminators, each of which looks at a different random low-dimensional projection of the data. Individual discriminators, now provided with restricted views of the input, are unable to reject generated samples perfectly and continue to provide meaningful gradients to the generator throughout training. Meanwhile, the generator learns to produce samples consistent with the full data distribution to satisfy all discriminators simultaneously. We demonstrate the practical utility of this approach experimentally, and show that it is able to produce image samples with higher quality than traditional training with a single discriminator.

The paper proposes to stabilize GAN training by using an ensemble of discriminators, each workin on a random projection of the input data, to provide the training signal for the generator model.
Q1: “In relation to “Theorem 3.1. … will produce samples from a distribution whose marginals along each of the projections W_k match those of the true distribution”.. I presume an infinite number of generator distributions could give rise to the correct marginals however not necessarily be converged to the data distribution. In Theorem A.2 the authors upperbound this residual as a function of the smoothness and support of the distributions as well as the projections presented to the discriminators. Can the authors comment on how tight this bound is e.g. as a function the number of used discriminators or the choosen projection methods ?
Q2: Related to the above. Did the authors do or considered any frequency analysis of the ensemble of random projection? I guess you could easily do a numeric simulation of the expected frequency spectrum of the combined set discriminators?
Q3: My primary concern with the work is the above mentioned computational complexity of running K discriminators in parallel. This is especially in relation to the experimental results showing significant high-frequency artefacts when running with K=12 classifiers (K=12 celebA results and “Random Imagenet-Canine Images: Proposed Method” in suplementary results). I think this is as expected as the authors are effectively fitting each classifier to the distributions of smoothed  (with 8x8 random kernel)  subsampled version of the input image. I would expect that each discriminator sees none or only a very limited amount  the high frequency component in the images.  Do the authors have any comments on how the sampling of the projection kernels affects the image results especially if the number of needed classifiers can be reduced somehow? I would expect that a combination of smoothing and high frequency filters would be needed to remove the high frequency artefacts?
Q4: Whats the explanation of the oscilating patterns in figure 2?
Q5: In the conclusion the authors mention that their framework is currently limited by the computational of running K discriminators and proposes:
“In our current framework, the number of discriminators is limited by computational cost. In future work, we plan to investigate training with a much larger set of discriminators, employing only a small subset of them at each iteration, or every set of iterations”
In the extreme case of only using a single randomly discriminator the approach is quite similar to the quite widely used input dropout to the discriminator?
Overall I like the simplicity of the proposed idea. However i’m not completely convinced that the “marginal” convergence proof holds for the relative low number of discriminators possible to use in practice. At least i would like the authors to touch on this key aspect of the method both theoretically and with experiments/simulations. Also several other methods have recently been proposed to improve stability of GANs, however no experimental comparisons is made with these methods (WGAN, EGAN, LSGAN etc.)


In this paper, we propose a mix-generator generative adversarial networks (PGAN) model that works in parallel by mixing multiple disjoint generators to approximate a complex real distribution. In our model, we propose an adjustment component that collects all the generated data points from the generators, learns the boundary between each pair of generators, and provides error to separate the support of each of the generated distributions. To overcome the instability in a multiplayer game, a shrinkage adjustment component method is introduced to gradually reduce the boundary between generators during the training procedure. To address the linearly growing training time problem in a multiple generators model, we propose a method to train the generators in parallel. This means that our work can be scaled up to large parallel computation frameworks. We present an efficient loss function for the discriminator, an effective adjustment component, and a suitable generator. We also show how to introduce the decay factor to stabilize the training procedure. We have performed extensive experiments on synthetic datasets, MNIST, and CIFAR-10. These experiments reveal that the error provided by the adjustment component could successfully separate the generated distributions and each of the generators can stably learn a part of the real distribution even if only a few modes are contained in the real distribution.

Overall, the writing is very confusing at points and needs some attention to make the paper clearer. I’m not entirely sure the authors understand the material particularly well, as I found some of the arguments and narrative confusing or just incorrect. I don’t really see any significant contribution here except “we had this idea for this model, and it works”. There’s no interesting questions being asked about missing modes (and no answers through good experimentation), no insight that might contribute to our understanding of the problem, and no comparison to other models. My guess is this submission was rushed (and perhaps they were just looking for feedback). I like the idea, don’t get me wrong: a model that is trainable across multiple GPUs and that distributes generative work is pretty cool, and I want to see this work succeed (after a *lot* more work). But the paper really lacks what I’d consider good science, and I don’t see it publishable without significant improvement.
Personally I think you should change the angle from missing modes to parallel training. I don’t see any strong guarantees that the model will do what you say it will, especially as beta goes to zero.
Detailed comments
P1
“, that explicitly approximate data distribution, the approximation of GAN is implicit”
The wording of this is pretty strange: by “implicit”, we mean that we only have *samples* from the distribution(s) of interest, but what does it mean for an approximation to be “implicit”?
From the intro, it doesn’t sound like the approach is meant for the “mode collapse” problem, but for dealing with missing modes. These are different types of failures for GANs, and while there are many theories for why these happen, to my knowledge there’s no such consensus that these issues are the same.
For instance, what is keeping each of the generators from collapsing onto a single value? We often see the model collapse on several different values: why couldn’t each of your generators do this?
P2: No, it is incorrect that the KL is what is causing mode collapse, and I think actually you mean “missing modes”. Arjovsky et al addresses the mode collapse problem, which is just another word for a type of instability in GANs. But this isn’t because of “vanishing gradients”, as the “proxy loss” (which you call “heuristic loss”, this isn’t a common term, fyi), which is what GANs are trained on in practice don’t vanish, but show some other sorts of instabilities (Arjovsky 2016). That said, other GAN variants without regularization also show collapse *and* missing modes, such as LSGAN and all the f-GAN variants (even the auto encoder variants).
You should also probably cite Che et al 2016 as another model that addressed missing modes. Also, what about ALI, BiGAN, and ALiCE? These also address missing modes (at least they claim to).
I don’t understand why you’re comparing f-GAN and WGAN convergences: they are addressing different things with GANs: one shows insight into what exactly traditional GANs are doing (solving a dual problem of minimizing an f-divergence) versus addressing stability through using an IPM (though also a dual formulation of the wasserstein). f-GANs ensure neither stability nor non-vanishing gradients.
P3: I like the breakdown of how the memory is organized.
This is for multi-GPU, correct? This needs to be explicitly stated.
P6:
There’s a sign error in proof 1 (both in the definition of the reverse KL and when the loss is written out).
Also, the gradient w.r.t. theta magically appears in the second half.
This is a pretty round-about way to arrive at that you’re minimizing the reverse KL: I’m pretty sure this can be shown by formulating the second term in f-gan (the one where you sample from the generator), that is f*(T), where f* is the convex conjugate of f = -log
Mixture of Gaussians: common *missing modes* experiment.
So my general comments about the experiments
You need to compare to other models that address missing modes. Overall, many people have shown success with experiments similar to your simple mixture of Gaussians experiments, so in order to show something significant here, you will need to have a more challenging experiments and show a comparison to other models.
The real-world experiments are fairly unconvincing, as you only show MNIST and CIFAR-10 (and MNIST doesn’t look very good). Overall, the good inception scores aren’t too surprising given the model has several generators for each mode, but I think we need to see a demonstration on better datasets.


Published as a conference paper at ICLR 2018 COULOMB GANS: PROVABLY OPTIMAL NASH EQUI- LIBRIA VIA POTENTIAL FIELDS
Generative adversarial networks (GANs) evolved into one of the most successful unsupervised techniques for generating realistic images. Even though it has recently been shown that GAN training converges, GAN models often end up in local Nash equilibria that are associated with mode collapse or otherwise fail to model the target distribution. We introduce Coulomb GANs, which pose the GAN learning problem as a potential field, where generated samples are attracted to training set samples but repel each other. The discriminator learns a potential field while the generator decreases the energy by moving its samples along the vector (force) field determined by the gradient of the potential field. Through decreasing the energy, the GAN model learns to generate samples according to the whole target distribution and does not only cover some of its modes. We prove that Coulomb GANs possess only one Nash equilibrium which is optimal in the sense that the model distribution equals the target distribution. We show the efficacy of Coulomb GANs on LSUN bedrooms, CelebA faces, CIFAR-10 and the Google Billion Word text generation.

In this paper, the authors interpret the training of GAN by potential field and inspired from which to provide new training procedure for GAN. They claim that under the condition that global optima are achieved for discriminator and generator in each iteration, the Coulomb GAN converges to the global solution.
I think there are several points need to be addressed.
1, I agree that the "model collapsing" is due to converging to a local Nash Equilibrium. However, there are more reasons besides the drawback of the loss function, which is emphasized in the paper. Leave the stochastic gradient descent optimization algorithm apart (since most of the neural networks are trained in this way), the parametrization and the richness of discriminator family play a vital role in the model collapsing issue. In fact, even with KL-divergence in which log operation is involved, if one can select reasonable parametrization, e.g., directly handling in function space, the saddle point optimization is convex-concave, which means under the same assumption made in the paper, there is only one global Nash Equilibrium. On the other hand, the richness of the discriminator also important in the training of GAN. I did not get the point about the drawback of III. If indeed as the paper considered in the ideal case, the discriminator is rich enough, III cannot happen.
The model collapsing is not just because loss function in training GAN. It is caused by the twist of these three issues listed above. Modifying the loss can avoid partially model collapsing, however, it is not appropriate to claim that the proposed algorithm is 'provable'. The assumption in this paper is too restricted, and the discussion is unfair to the existing variants of GAN, e.g., GMMN or Wasserstein GAN, which under some assumptions, there is also only one global Nash Equilibrium.
2, In the training procedure, the discriminator family is important as we discussed. The paper claims that the reason to introduce the extra discriminator is reducing variance. However, such parametrization will introduce bias too. The bias and variance tradeoff should be explicitly discussed here. Ideally, it should contain all the functions formed with Plummer kernel, but not too large (otherwise, it will increase the sample complexity.). Which function family used in the paper is not clear.
3, As the authors already realized, the GMMN is one closely related model. It will be more convincing to add the comparison with GMMN.
In sum, this paper provides an interesting perspective modeling GAN from the potential field, however, there are several issues need to be addressed. I expect to see the reply of the authors regarding the mentioned issues.


When machine learning models are used for high-stakes decisions, they should predict accurately, fairly, and responsibly. To fulfill these three requirements, a model must be able to output a reject option (i.e. say "I Don't Know") when it is not qualified to make a prediction. In this work, we propose learning to defer, a method by which a model can defer judgment to a downstream decision-maker such as a human user. We show that learning to defer generalizes the rejection learning framework in two ways: by considering the effect of other agents in the decision-making process, and by allowing for optimization of complex objectives. We propose a learning algorithm which accounts for potential biases held by decision-makers later in a pipeline. Experiments on real-world datasets demonstrate that learning to defer can make a model not only more accurate but also less biased. Even when operated by highly biased users, we show that deferring models can still greatly improve the fairness of the entire pipeline.

Strengths:
1. This paper proposes a novel framework for ensuring fairness in the classification pipeline. To this end, this work explores models that learn to defer.
2. The work is conceptually very interesting. The idea of learning to defer (as proposed in the paper) as a means to fairness is not only novel but also quite apt.
3. Experimental results demonstrate that the proposed learning strategy can not only increase predictive accuracy but also reduce bias in decisions.
Weaknesses:
1. While this work is conceptually quite novel and interesting, the technical novelty and contributions seem fairly minimal.
2. The proposed formulations are essentailly regularized variants of fairly standard classification models and the optimization also relies upon standard search procedures.
3. Experimental analysis on deferring to a biased decision maker (Section 7.3) is rather limited.
Summary: This paper proposes a novel framework for ensuring fairness in the classification pipeline. More specifically, the paper outlines a strategy called learn to defer which enables the design of predictive models which not only classify accurately and fairly but also defer if necessary. Deferring a decision is used as a mechanism to ensure both fairness and accuracy. Furthermore, the authors consider two variants depending on if the model has some information about the decision maker or not. Experimental results on real world datasets demonstrate the effectiveness of the proposed approach in building an end to end pipeline that ensures accuracy and fairness.
Novelty: The main novelty of this work stems from the idea of introducing learning to defer mechanisms in the context of fairness. While the ideas of learning to defer have already been studied in the context of classification models, this is the first contribution which leverages learning to defer strategy as a means to achieve fairness. However, beyond this conceptual novelty, the work does not demonstrate a lot of technical novelty or depth. The objective functions proposed are simple extensions of work done by Zafar et. al. (WWW, AISTATS 2017). The optimization procedures being used are also fairly standard. Furthermore, the authors do not carry out any rigorous theoretical analysis either.
Other detailed comments:
1. I would strongly encourage the authors to carry out a more in-depth theoretical analysis of the proposed framework (Refer to "Provably Fair Representations" McNamara et. al. 2017)
2. Experimental evaluation can also be strengthened. More specifically, analysis in Section 7.3 can be made more thorough. Instead of just sticking to one scenario where the decision maker is extremely biased (how are you quantifying this?), how about plotting a graph where x axis denotes the extent of bias in decision-maker's judgments and y-axis captures the model performance?
3. Overall, the paper is quite well written and is well motivated. There are however some typos and incorrect figure refernces (e.g., Section 7.2 first line, Figure 7.2, there is no such figure).


AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS
In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and the context in which it appears, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform stateof-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

==Update==
I appreciate the response, and continue to recommend acceptance. The evaluation metric used in this paper (SentEval) represents an important open problem in NLP—learning reusable sentence representations—and one of the problems in NLP best suited to presentation at IC*LR*. Because of this, I'm willing to excuse the fact that the paper is only moderately novel, in light of the impressive reported results.
While I would appreciate a direct (same codebase, same data) comparison with some outside baselines, this paper meets or exceeds the standards for rigor that were established by previous published work in the area, and the existing results are sufficient to support some substantial conclusions.
==========
This paper proposes an alternative formulation of Kiros's SkipThought objective for training general-purpose sentence encoder RNNs on unlabeled data. This formulation replaces the decoder in that model with a second encoder, and yields substantial improvements to both speed and model performance (as measured on downstream transfer tasks). The resulting model is, for the first time, reasonably competitive even with models that are trained end-to-end on labeled data for the downstream tasks (despite the requirement, imposed by the evaluation procedure, that only the top layer classifier be trained for the downstream tasks here), and is also competitive with models trained on large labeled datasets like SNLI. The idea is reasonable, the topic is important, and the results are quite strong. I recommend acceptance, with some caveats that I hope can be addressed.
Concerns:
A nearly identical idea to the core idea of this paper was proposed in an arXiv paper this spring, as a commenter below pointed out. That work has been out for long enough that I'd urge you to cite it, but it was not published and it reports results that are far less impressive than yours, so that omission isn't a major problem.
I'd like to see more discussion of how you performed your evaluation on the downstream tasks. Did you use the SentEval tool from Conneau et al., as several related recent papers have? If not, does your evaluation procedure differ from theirs or Kiros's in any meaningful way?
I'm also a bit uncomfortable that the paper doesn't directly compare with any baselines that use the exact same codebase, word representations, hyperparameter tuning procedure, etc.. I would be more comfortable with the results if, for example, the authors compared a low-dimensional version of their model with a low-dimensional version of SkipThought, trained in the *exact* same way, or if they implemented the core of their model within the SkipThought codebase and showed strong results there.
Minor points:
The headers in Table 1 don't make it all that clear which additions (vectors, UMBC) are cumulative with what other additions. This should be an easy fix.
The use of the check-mark as an output in Figure 1 doesn't make much sense, since the task is not binary classification.
"Instead of training a model to reconstruct the surface form of the input sentence or its neighbors, our formulation attempts to focus on the semantic aspects of sentences. The meaning of a sentence is the property that creates bonds between a sequence of sentences and makes it logically flow." – It's hard to pin down exactly what this means, but it sounds like you're making an empirical claim here: semantic information is more important than non-semantic sources of variation (syntactic/lexical/morphological factors) in predicting the flow of a text. Provide some evidence for this, or cut it.
You make a similar claim later in the same section: "In figure 1(a) however, the reconstruction loss forces the model to predict local structural information about target sentences that may be irrelevant to its meaning (e.g., is governed by grammar rules)." This is a testable prediction: Are purely grammatical (non-semantic) variations in sentence form helpful for your task? I'd suspect that they are, at least in some cases, as they might give you clues as to style, dialect, or framing choices that the author made when writing that specific passage.
"Our best BookCorpus model (MC-QT) trains in just under 11hrs, compared to skip-thought model’s training time of 2 weeks." –  If you say this, you need to offer evidence that your model is faster. If you don't use the same hardware and low-level software (i.e., CuDNN), this comparison tells us nearly nothing. The small-scale replication of SkipThought described above should address this issue, if performed.


This paper proposes a powerful regularization method named ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake regularization that decreases error rates by disturbing learning. While Shake-Shake can be applied to only ResNeXt which has multiple branches, ShakeDrop can be applied to not only ResNeXt but also ResNet, Wide ResNet and PyramidNet in a memory efficient way. Important and interesting feature of ShakeDrop is that it strongly disturbs learning by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. The effectiveness of ShakeDrop is confirmed by experiments on CIFAR-10/100 and Tiny ImageNet datasets.  (Huang et al., 2016)  and Shake-Shake (Gastaldi, 2017) are known to be effective regularization methods for ResNet and its improvements. Among them, Shake-Shake applied to ResNeXt is the one achieving the lowest error rates on CIFAR-10/100 datasets (Gastaldi, 2017).
Shake-Shake, however, has following two drawbacks. (1) Shake-Shake can be applied to only multibranch architectures (i.e., ResNeXt). (2) Shake-Shake is not memory efficient. Both drawbacks come from the same root. That is, Shake-Shake requires two branches of residual blocks to apply. If it is true, it is not difficult to conceive its solution: a similar disturbance to Shake-Shake on a single residual block. It is, however, not trivial to realize it.
The current paper addresses the problem of realizing a similar disturbance to Shake-Shake on a single residual block, and proposes a powerful regularization method, named ShakeDrop regularization. While the proposed ShakeDrop is inspired by Shake-Shake, the mechanism of disturbing learning is completely different. ShakeDrop disturbs learning more strongly by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. In addition, a different factor from the forward pass is multiplied in the backward training pass. As a byproduct, however, learning process gets unstable. Our solution to this problem is to stabilize the learning process by employing ResDrop in a different usage from the usual. Based on experiments using various base network architectures, we reveal the condition that the proposed ShakeDrop successfully works.

The paper proposes ShakeDrop regularization, which is essentially a combination of the PyramidDrop and Shake-Shake regularization. The procedure consists of essentially weighting the residual branch with a random weight, in the style of Shake-Shake, where the weight is sampled from a mixture of uniform distribution in [-1, 1] and delta at 1, such that the mixture of those two distributions varies linearly with layer depth, in the style of PyramidDrop. In the style of Shake-Shake, a different random weight (in [0, 1]) is used for the backward pass. The most surprising part is that that the forward weight can be negative thus inverting the output of a convolution. Apparently the goal is to "disturb" the training, and the procedure yields state-of-the-art results on CIFAR-10/100.
Positives:
- Results: state-of-the-art on CIFAR-10/100
Negatives:
1. No real motivation on why should this work. I guess the motivation is the mixture of PyramidDrop and Shake-Shake motivations, but the main surprising part (forward weight can be negative) is not motivated at all. There is a tiny bit of discussion at the very end, section 4.4, where authors examine the training loss (showing it's non-zero so less overfitting) and mean/variance of gradients (increased). However, this doesn't really satisfy me - it is clear that more disturbance will cause this behaviour, but that doesn't mean any disturbance is good, e.g. if I always apply the negative weight and make my model weights go in the wrong direction, I'm pretty sure training loss and gradients will be even larger, but it's a bad idea to do.
2. I'm concerned with the "weird trick that happens to work on CIFAR" line of work (not saying that this paper is the only offender) - are these methods actually useful and generalizable to other problems, or are we overfitting on CIFAR and creating MNIST v2.0 ? It would be nice to demonstrate that this regularization works in at least one more problem, maybe ImageNet, though maybe regularization is not needed there but just find one more dataset that needs regularization and test this on that.
3. The paper doesn't explain well what is the problem with Shake-Shake and memory. I see that the author of Shake-Shake has made a comment on this and that makes a lot of sense, i.e. there is no memory issue, just because there are 2x branches doesn't mean shake-shake needs 2x memory as it can use less capacity=memory to achieve the same performance. So it seems the main premise of the paper - "let's apply Shake-Shake to deeper models but we need to come up with a modified method because Shake-Shake cannot be applied due to memory problems" - seems wrong.
4. The writing quality is quite bad, it is very hard to understand what authors mean in parts of the text. E.g. at two places "it has almost the same residual block as Eqn. (1)" - how is it "almost"? Below equation 5, it is never specified that alpha and beta are sampled uniformly(?) from those ranges, one could think that alpha and beta are fixed constants that take a specific value that is in that range. There are also various grammatical errors such as "is expected to be powerful but slight memory overhead" or "which is introduced essence", etc.
Smaller comments:
- Isn't it surprising that alpha in [-1, 1] and beta in [0, 1] works well, but alpha in [0, 1] and beta in [-1, 1] works much worse? The two important cases, (alpha negative, beta positive) and (alpha positive, beta negative), seem to me like they are conceptually very similar.
- End of section 4.1, should it be b_l as p_L is a constant and b_l is what is sampled?
- I don't like that exactly the same text is repeated 3 times (abstract, end of intro, end of 1.1) and in very short distance from each other - repeating the same words 3 times doesn't make the reader understand it better, slight rephrasing is much more beneficial.
Overall:
Good to know that this method sets the new state of the art on CIFAR-10/100, so as such it should be of interest to the community to be available online (arXiv). But with fairly little novelty (is a combination of 2 methods), very little insights of why this should work at all (especially the negative scaling coefficient which is the only extra thing that one learns from this paper, since the rest is a combination of PyramidDrop and Shake-Shake), no idea on whether the method would work outside of the CIFAR-world, and bad quality of the text - I don't think the manuscript is sufficiently good for ICLR.


Published as a conference paper at ICLR 2018 TOWARDS DEEP LEARNING MODELS RESISTANT TO ADVERSARIAL ATTACKS
Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network. To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization. This approach provides us with a broad and unifying view on much prior work on this topic. Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal. In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries. These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks. They also suggest robustness against a first-order adversary as a natural security guarantee. We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

This paper proposes to look at making neural networks resistant to adversarial loss through the framework of saddle-point problems. They show that, on MNIST, a PGD adversary fits this framework and allows the authors to train very robust models. They also show encouraging results for robust CIFAR-10 models, but with still much room for improvement. Finally, they suggest that PGD is an optimal first order adversary, and leads to optimal robustness against any first order attack.
This paper is well written, brings new ideas and perfoms interesting experiments, but its claims are somewhat bothering me, considering that e.g. your CIFAR-10 results are somewhat underwhelming. All you've really proven is that PGD on MNIST seems to be the ultimate adversary. You contrast this to the fact that the optimization is non-convex, but we know for a fact that MNIST is fairly simple in that regime; iirc a linear classifier gets something like 91% accuracy on MNIST. So my guess is that the optimization problem on MNIST is in fact pretty convex and mostly respects the assumptions of Danskin's theorem, but not so much for CIFAR-10 (maybe even less so for e.g. ImageNet, which is what Kurakin et al. seem to find).
Considering your CIFAR-10 results, I don't think anyone should "suggest that secure neural networks are within reach", because 1) there is still room for improvement 2) it's a safe bet that someone will always just come up with a better attack than whatever defense we have now. It has been this way in many disciplines (crypto, security) for centuries, I don't see why deep learning should be exempt. Simply saying "we believe that our robust models are significant progress on the defense side" was enough, because afaik you did improve on CIFAR-10's SOTA; don't overclaim.
You make these kinds of claims in a few other places in this paper, please be careful with that.
The contributions in your appendix are interesting.
Appendix A somewhat confirms one of the postulates in Goodfellow et al. (2014): "The direction of perturbation, rather than the specific point in space, matters most. Space is not full of pockets of adversarial examples that finely tile the reals like the rational numbers".
Appendix B and C are not extremely novel in my mind, but definitely add more evidence.
Appendix E is quite nice since it gives an insight into what actually makes the model resistant to adversarial examples.
Remarks:
- The update for PGD should be using \nabla_{x_t} L(\theta,x_t,y), (rather than only \nabla_x)?
- In table 2, attacking a with 20-step PGD is doing better than 7-step.  When you say "other hyperparameter choices didn’t offer a significant decrease in accuracy", does that include the number of steps? If not why stop there? What happens for more steps? (or is it too computationally intensive?)
- You only seem to consider adversarial examples created from your dataset + adv. noise. What about rubbish class examples? (e.g. rgb noise)


CAUSALGAN: LEARNING CAUSAL IMPLICIT GENER- ATIVE MODELS WITH ADVERSARIAL TRAINING
We introduce causal implicit generative models (CiGMs): models that allow sampling from not only the true observational but also the true interventional distributions. We show that adversarial training can be used to learn a CiGM, if the generator architecture is structured based on a given causal graph. We consider the application of conditional and interventional sampling of face images with binary feature labels, such as mustache, young. We preserve the dependency structure between the labels with a given causal graph. We devise a two-stage procedure for learning a CiGM over the labels and the image. First we train a CiGM over the binary labels using a Wasserstein GAN where the generator neural network is consistent with the causal graph between the labels. Later, we combine this with a conditional GAN to generate images conditioned on the binary labels. We propose two new conditional GAN architectures: CausalGAN and CausalBEGAN. We show that the optimal generator of the CausalGAN, given the labels, samples from the image distributions conditioned on these labels. The conditional GAN combined with a trained CiGM for the labels is then a CiGM over the labels and the generated image. We show that the proposed architectures can be used to sample from observational and interventional image distributions, even for interventions which do not naturally occur in the dataset.

In their paper "CausalGAN: Learning Causal implicit Generative Models with adv. training" the authors address the following issue: Given a causal structure between "labels" of an image (e.g. gender, mustache, smiling, etc.), one tries to learn a causal model between these variables and the image itself from observational data. Here, the image is considered to be an effect of all the labels. Such a causal model allows us to not only sample from conditional observational distributions, but also from intervention distributions. These tasks are clearly different, as nicely shown by the authors' example of "do(mustache = 1)" versus "given mustache = 1" (a sample from the latter distribution contains only men). The paper does not aim at learning causal structure from data (as clearly stated by the authors). The example images look convincing to me.
I like the idea of this paper. IMO, it is a very nice, clean, and useful approach of combining causality and the expressive power of neural networks. The paper has the potential of conveying the message of causality into the ICLR community and thereby trigger other ideas in that area. For me, it is not easy to judge the novelty of the approach, but the authors list related works, none of which seems to solve the same task. The presentation of the paper, however, should be improved significantly before publication. (In fact, because of the presentation of the paper, I was hesitating whether I should suggest acceptance.) Below, I give some examples (and suggest improvements), but there are many others. There is a risk that in its current state the paper will not generate much impact, and that would be a pity. I would therefore like to ask the authors to put a lot of effort into improving the presentation of the paper.
- I believe that I understand the authors' intention of the caption of Fig. 1, but "samples outside the dataset" is a misleading formulation. Any reasonable model does more than just reproducing the data points. I find the argumentation the authors give in Figure 6 much sharper. Even better: add the expression "P(male = 1 | mustache = 1) = 1". Then, the difference is crystal clear.
- The difference between Figures 1, 4, and 6 could be clarified.
- The list of "prior work on learning causal graphs" seems a bit random. I would add Spirtes et al 2000, Heckermann et al 1999, Peters et al 2016, and Chickering et al 2002.
- Male -> Bald does not make much sense causally (it should be Gender -> Baldness)... Aha, now I understand: The authors seem to switch between "Gender" and "Male" being random variables. Make this consistent, please.
- There are many typos and comma mistakes.
- I would introduce the do-notation much earlier. The paragraph on p. 2 is now written without do-notation ("intervening Mustache = 1 would not change the distribution"). But this way, the statements are at least very confusing (which one is "the distribution"?).
- I would get rid of the concept of CiGM. To me, it seems that this is a causal model with a neural network (NN) modeling the functions that appear in the SCM. This means, it's "just" using NNs as a model class. Instead, one could just say that one wants to learn a causal model and the proposed procedure is called CausalGAN? (This would also clarify the paper's contribution.)
- many realizations = one sample (not samples), I think.
- Fig 1: which model is used to generate the conditional sample?
- The notation changes between E and N and Z for the noises. I believe that N is supposed to be the noise in the SCM, but then maybe it should not be called E at the beginning.
- I believe Prop 1 (as it is stated) is wrong. For a reference, see Peters, Janzing, Scholkopf: Elements of Causal Inference: Foundations and Learning Algorithms (available as pdf), Definition 6.32. One requires the strict positivity of the densities (to properly define conditionals). Also, I believe the Z should be a vector, not a set.
- Below eq. (1), I am not sure what the V in P_V refers to.
- The concept of data probability density function seems weird to me. Either it is referring to the fitted model, then it's a bad name, or it's an empirical distribution, then there is no pdf, but a pmf.
- Many subscripts are used without explanation. r -> real? g -> generating? G -> generating? Sometimes, no subscripts are used (e.g., Fig 4 or figures in Sec. 8.13)
- I would get rid of Theorem 1 and explain it in words for the following reasons. (1) What is an "informal" theorem? (2) It refers to equations appearing much later. (3) It is stated again later as Theorem 2.
- Also: the name P_g does not appear anywhere else in the theorem, I think.
- Furthermore, I would reformulate the theorem. The main point is that the intervention distributions are correct (this fact seems to be there, but is "hidden" in the CIGN notation in the corollary).
- Re. the formulation in Thm 2: is it clear that there is a unique global optimum (my intuition would say there could be several), thus: better write "_a_ global minimum"?
- Fig. 3 was not very clear to me. I suggest to put more information into its caption.
- In particular, why is the dataset not used for the causal controller? I thought, that it should model the joint (empirical) distribution over the labels, and this is part of the dataset. Am I missing sth?
- IMO, the structure of the paper can be improved. Currently, Section 3 is called "Background" which does not say much. Section 4 contains CIGMs, Section 5 Causal GANs, 5.1. Causal Controller, 5.2. CausalGAN, 5.2.1. Architecture (which the causal controller is part of) etc. An alternative could be:
Sec 1: Introduction
Sec 1.1: Related Work
Sec 2: Causal Models
Sec 2.1: Causal Models using Generative Models (old: CIGM)
Sec 3: Causal GANs
Sec 3.1: Architecture (including controller)
Sec 3.2: loss functions
...
Sec 4: Empricial Results (old: Sec. 6: Results)
- "Causal Graph 1" is not a proper reference (it's Fig 23 I guess). Also, it is quite important for the paper, I think it should be in the main part.
- There are different references to the "Appendix", "Suppl. Material", or "Sec. 8" -- please be consistent (and try to avoid ambiguity by being more specific -- the appendix contains ~20 pages). Have I missed the reference to the proof of Thm 2?
- 8.1. contains copy-paste from the main text.
- "proposition from Goodfellow" -> please be more precise
- What is Fig 8 used for? Is it not sufficient to have and discuss Fig 23?
- IMO, Section 5.3. should be rewritten (also, maybe include another reference for BEGAN).
- There is a reference to Lemma 15. However, I have not found that lemma.
- I think it's quite interesting that the framework seems to also allow answering counterfactual questions for realizations that have been sampled from the model, see Fig 16. This is the case since for the generated realizations, the noise values are known. The authors may think about including a comment on that issue.
- Since this paper's main proposal is a methodological one, I would make the publication conditional on the fact that code is released.


Auto-Encoders are unsupervised models that aim to learn patterns from observed data by minimizing a reconstruction cost. The useful representations learned are often found to be sparse and distributed. On the other hand, compressed sensing and sparse coding assume a data generating process, where the observed data is generated from some true latent signal source, and try to recover the corresponding signal from measurements. Looking at auto-encoders from this signal recovery perspective enables us to have a more coherent view of these techniques. In this paper, in particular, we show that the true hidden representation can be approximately recovered if the weight matrices are highly incoherent with unit 2 row length and the bias vectors takes the value (approximately) equal to the negative of the data mean. The recovery also becomes more and more accurate as the sparsity in hidden signals increases. Additionally, we empirically also demonstrate that auto-encoders are capable of recovering the data generating dictionary when only data samples are given.

This papers proposes to analyze auto-encoders under sparsity constraints of an underlying signal to be recovered.
Based on concentration inequality, the reconstruction provided for a simple class of functions is guaranteed to be accurate in l1 norm with high probability.
The proof techniques are classical, but the results seem novel as far as I know.
As an open question, could the results be given for other lp norms, in particular for infinity-norm? Indeed, this is a privileged norm for support recovery.
Presentation issues:
- section should be Section when stating for instance "section 1". Idem for eq, equation, assumption...
- bold fonts for vectors are randomly used: some care should be given to harmonizing symbols fonts.
- equations should be cited with brackets
References issues:
- harmonize citations: if you add first name for some authors add it for all references: why writing Roland Makhzani and J. Wright?
- Candes -> Cand\`es
- Consider citing "Sparse approximate solutions to linear systems", Natarajan 1995 when mentioning Amaldi and Kann 1998.
Specific comments:
page 1:
- hasn't -> has not.
page 2:
- "activation function": at this stage s_e and s_d are just functions. What is the "activation" refers to? Also a clarification on the space they act on should be stated. Idem for b_e and b_d.
- "the identity of h in eq. 1 is only well defined in the presence of l1 regularization due to the over-completeness of the dictionary" : this is implicitly stating the uniqueness of the Lasso. Not that it is well known that there are cases where the Lasso is non-unique. Please, clarify your statement accordingly.
- for simplicity b_d could be removed here.
- in (4) it would be more natural to write f_j(h_j) instead of f(h_j)
- "has is that to be bounded"-> is boundedness?
- what is l_max_j here? moreover the bold letters seem to represent vectors but this should be state explicitly somewhere.
page 3:
- what is the precise meaning of "distributed" when referring to representation
- In remark 1: the font has changed weirdly for W and h.
- "two class"->two classes
- Definition 1: again what is a precise definition of activation function?
- "if we set": bold issue.
- b should b_e in Theorem 1, right? Also, please recall the definition of the sigmoid function here. Moreover l_max and mu_h seem useless in this theorem... why referring to them?
- "if the rows of the weight matrix is"-> if the rows of the weight matrix are
page 4:
- Proposition 1 could be stated as a theorem and Th.1 as a corollary (with e=0). The same is true for proposition 2 I suspect.
- Again the influence of l_max and mu_h are none here...
- Please, provide the definition of the ReLu function here. Is this just x->x_+ ?
page 6:
- R^+m -> font issue again.
- "are maximally incoherent": what is the precise meaning of this statement?
- what the motivation for Theorem 3? This should be discussed.
- De-noising -> de-noising
- the discussion after (15) should be made more precise.
page 7:
- Figure 1 and 2 should be postponed to page 8.
- in Eq. (16) one needs to known E_h(x) and E_h_i(h_i), but I suspect this quantity are usually unknown to the practitioner. Can the author comment on that?
page 8:
- "the recovery is denoised through thresholding": where is this step analyzed?
page 9:
- figure 3: sparseness-> sparsity; also what is the activation function used here?
- "are then generate"->are then generated
- "by greedily select"->by greedily selecting
- "the the"
- "and thus pre-process"-> and thus pre-processing
Supplementary:
page 1:
- please define \sigma, and its simple properties used along the proof.
page 2:
- g should be g_j (in eq 27 - > 31)
- overall this proof relies on ingredients such as the one used for Hoeffding's inequality.
Most ingredients could be taken from standard tools on concentration (see for instance Boucheron, Lugosi, Massart: "Concentration Inequalities: A Nonasymptotic Theory of Independence", 2013).
Moreover, some elements should be factorized as they are shared among the next proofs. This should reduce the size of the supplementary dramatically.
page 7:
- Eq. (99): it should be reminded that W_ii=1 here.
- the upper bound used on \mu to get equation 105 seems to be in the wrong order.


We propose a new, multi-component energy function for energy-based Generative Adversarial Networks (GANs) based on methods from the image quality assessment literature. Our approach expands on the Boundary Equilibrium Generative Adversarial Network (BEGAN) by outlining some of the short-comings of the original energy and loss functions. We address these short-comings by incorporating an l 1 score, the Gradient Magnitude Similarity score, and a chrominance score into the new energy function. We then provide a set of systematic experiments that explore its hyper-parameters. We show that each of the energy function's components is able to represent a slightly different set of features, which require their own evaluation criteria to assess whether they have been adequately learned. We show that models using the new energy function are able to produce better image representations than the BEGAN model in predicted ways.

This paper proposed some new energy function in the BEGAN (boundary equilibrium GAN framework), including l_1 score, Gradient magnitude similarity score, and chrominance score, which are motivated and borrowed from the image quality assessment techniques. These energy component in the objective function allows learning of different set of features and determination on whether the features are adequately represented. experiments on the using different hyper-parameters of the energy function, as well as visual inspections on the quality of the learned images, are presented.
It appears to me that the novelty of the paper is limited, in that the main approach is built on the existing BEGAN framework with certain modifications. For example, the new energy function in equation (4) larges achieves similar goal as the original energy (1) proposed by Zhao et. al (2016), except that the margin loss in (1) is changed to a re-weighted linear loss, where the dynamic weighting scheme of k_t is borrowed  from the work of Berthelot et. al (2017). It is not very clear why making such changes in the energy would supposedly make the results better, and no further discussions are provided.  On the other hand, the several energy component introduced are simply choices of the similarity measures as motivated from the image quality assessment, and there are probably a lot more in the literature whose application can not be deemed as a significant contribution to either theories or algorithm designs in GAN.
Many results from the experimental section rely on visual evaluations, such as in Figure~4 or 5; from these figures, it is difficult to clearly pick out the winning images. In Figure~5, for a fair  evaluation on the performance of model interploations, the same human model should be used for competing methods, instead of applying different human models and different interpolation tasks in different methods.


Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.

This is an interesting paper focusing on building discrete reprentations of sequence by autoencoder.
However, the experiments are too weak to demonstrate the effectiveness of using discrete representations.
The design of the experiments on language model is problematic.
There are a few interesting points about discretizing the represenations by saturating sigmoid and gumbel-softmax, but the lack of comparisons to benchmarks is a critical defect of this paper.
Generally, continuous vector representations are more powerful than discrete ones, but discreteness corresponds to some inductive biases that might help the learning of deep neural networks, which is the appealing part of discrete representations, especially the stochastic discrete representations.
However, I didn't see the intuitions behind the model that would result in its superiority to the continuous counterpart.
The proposal of DSAE might help evaluate the usage of the 'autoencoding function' c(s), but it is certainly not enough to convince people.
How is the performance if c(s) is replaced with the representations achieved from autoencoder, variational autoencoder or simply the sentence vectors produced by language model?
The qualitative evaluation on 'Deciperhing the Latent Code' is not enough either.
In addition, the language model part doesn't sound correct, because the model cheated on seeing the further before predicting the words autoregressively.
One suggestion is to change the framework to variational auto-encoder, otherwise anything related to perplexity is not correct in this case.
Overall, this paper is more suitable for the workshop track. It also needs a lot of more studies on related work.


Graph-structured data such as social networks, functional brain networks, gene regulatory networks, communications networks have brought the interest in generalizing deep learning techniques to graph domains. In this paper, we are interested to design neural networks for graphs with variable length in order to solve learning problems such as vertex classification, graph classification, graph regression, and graph generative tasks. Most existing works have focused on recurrent neural networks (RNNs) to learn meaningful representations of graphs, and more recently new convolutional neural networks (ConvNets) have been introduced. In this work, we want to compare rigorously these two fundamental families of architectures to solve graph learning tasks. We review existing graph RNN and ConvNet architectures, and propose natural extension of LSTM and ConvNet to graphs with arbitrary size. Then, we design a set of analytically controlled experiments on two basic graph problems, i.e. subgraph matching and graph clustering, to test the different architectures. Numerical results show that the proposed graph ConvNets are 3-17% more accurate and 1.5-4x faster than graph RNNs. Graph ConvNets are also 36% more accurate than variational (non-learning) techniques. Finally, the most effective graph ConvNet architecture uses gated edges and residuality. Residuality plays an essential role to learn multi-layer architectures as they provide a 10% gain of performance.

The paper proposes an adaptation of existing Graph ConvNets and evaluates this formulation on a several existing benchmarks of the graph neural network community. In particular, a tree structured LSTM is taken and modified. The authors describe this as adapting it to general graphs, stacking, followed by adding edge gates and residuality.
My biggest concern is novelty, as the modifications are minor. In particular, the formulation can be seen in a different way. As I see it, instead of adapting Tree LSTMs to arbitary graphs, it can be seen as taking the original formulation by Scarselli and replacing the RNN by a gated version, i.e. adding the known LSTM gates (input, output, forget gate). This is a minor modification. Adding stacking and residuality are now standard operations in deep learning, and edge-gates have also already been introduced in the literature, as described in the paper.
A second concern is the presentation of the paper, which can be confusing at some points. A major example is the mathematical description of the methods. When reading the description as given, one should actually infer that Graph ConvNets and Graph RNNs are the same thing, which can be seen by the fact that equations (1) and (6) are equivalent.
Another example, after (2), the important point to raise is the difference to classical (sequential) RNNs, namely the fact that the dependence graph of the model is not a DAG anymore, which introduces cyclic dependencies.
Generally, a clear introduction of the problem is also missing. What are the inputs, what are the outputs, what kind of problems should be solved? The update equations for the hidden states are given for all models, but how is the output calculated given the hidden states from variable numbers of nodes of an irregular graph?
The model has been evaluated on standard datasets with a performance, which seems to be on par, or a slight edge, which could probably be due to the newly introduced residuality.
A couple of details :
- the length of a graph is not defined. The size of the set of nodes might be meant.
- at the beginning of section 2.1 I do not understand the reference to word prediction and natural language processing. RNNs are not restricted to NLP and I think there is no need to introduce an application at this point.
- It is unclear what does the following sentence means: "ConvNets are more pruned to deep networks than RNNs"?
- What are "heterogeneous graph domains"?