Every second, innumerable text data, including all kinds news, reports, messages, reviews, comments, and twits have been generated on the Internet, which is written not only in English but also in other languages such as Chinese, Japanese, French and so on. Not only SNS sites but also worldwide news agency such as Thomson Reuters News 1 provide news reported in more than 20 languages, reflecting the significance of the multilingual information. In this research, by taking advantage of multi-lingual text resources provided by the Thomson Reuters News, we developed a bidirectional LSTM based method to calculate cross-lingual semantic text similarity for long text and short text respectively. Thus, users could understand the situation comprehensively, by investigating similar and related cross-lingual articles, when there an important news comes in.

* PAPER SUMMARY *
This paper proposes a siamese net architecture to compare text in different languages. The proposed architecture builds upon siamese RNN by Mueller and Thyagarajan. The proposed approach is evaluated on cross lingual bitext retrieval.
* REVIEW SUMMARY *
This paper is hard to read and need proof-reading by a person proficient in English. The experiments are extremely limited, on a toy task. No other baseline than (Mueller and Thyagarajan, 2016) is considered. The related work section lacks important references. It is hard to find positive points that would advocate for a presentation at ICLR.
* DETAILED REVIEW *
On related work, the authors need to consider related work on cross lingual retrieval, multilingual document representation:
Bai, Bing, et al. "Learning to rank with (a lot of) word features." Information retrieval 13.3 (2010): 291-314. (Section 4).
Schwenk, H., Tran, K., Firat, O., & Douze, M. Learning Joint Multilingual Sentence Representations with Neural Machine Translation, ACL Workshop on Representation Learning for NLP, 2017
Karl Moritz Hermann and Phil Blunsom.  Multilingual models for compositional distributed semantics. In ACL 2014. pages 58–68.
Hieu Pham, Minh-Thang Luong, and Christopher D. Manning. Learning distributed representations for multilingual text sequences. In Workshop
on Vector Space Modeling for NLP. 2015
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. Cross-lingual sentiment classification with bilingual document representation learning. In ACL 2016
...
On evaluation, the authors need to learn about standard retrieval evaluation metrics such as precision at top 10, etc and use them. For instance, this book will be a good read.
Baeza-Yates, Ricardo, and Berthier Ribeiro-Neto. Modern information retrieval. Vol. 463. New York: ACM press, 1999.
On learning objective, the authors might want to read about learn-to-rank objectives for information retrieval, for instance,
Liu, Tie-Yan. "Learning to rank for information retrieval." Foundations and Trends in Information Retrieval 3.3 (2009): 225-331.
Burges, Christopher JC. "From ranknet to lambdarank to lambdamart: An overview." Learning 11, no. 23-581 (2010): 81.
Chapelle, Olivier, and Yi Chang. "Yahoo! learning to rank challenge overview." Proceedings of the Learning to Rank Challenge. 2011.
Herbrich, Ralf, Thore Graepel, and Klaus Obermayer. "Large margin rank boundaries for ordinal regression." (2000).
On experimental setup, the authors want to consider a setup with more than 8k training documents. More importantly, ranking a document set of 1k documents is extremely small, toyish. For instance, (Schwenk et al 2017) search through 1.5 million sentences. (Bai, Bing, et al 2009) search through 140k documents. Since you mainly introduces 2 modifications with respect to (Mueller and Thyagarajan, 2016), i.e  (i) not sharing the parameters on both branch of the siamese and (ii) the fully connected net on top, I would suggest to measure the effect of each of them both on multilingual data and on the SICK dataset used in (Mueller and Thyagarajan, 2016).


Published as a conference paper at ICLR 2018 DEMYSTIFYING MMD GANS
We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

The quality and clarity of this work are very good. The introduction of the kernel inception metric is well-motivated and novel, to my knowledge. With the mention of a bit more related work (although this is already quite good), I believe that this could be a significant resource for understanding MMD GANs and how they fit into the larger model zoo.
Pros
 - best description of MMD GANs that I have encountered
 - good contextualization of related work and descriptions of relationships, at least among the works surveyed
 - reasonable proposed metric (KID) and comparison with other scores
 - proof of unbiased gradient estimates is a solid contribution
Cons
 - although the review of related work is very good, it does focus on ~3 recent papers. As a review, it would be nice to see mention (even just in a list with citations) of how other models in the zoo fit in
 - connection between IPMs and MMD gets a bit lost; a figure (e.g.  flow chart) would help
 - wavers a bit between proposing/proving novel things vs. reviewing and lacks some overall structure/storyline
 - Figure 1 is a bit confusing; why is KID tested without replacement, and FID with? Why 100 vs 10 samples? The comparison is good to have, but it's hard to draw any insight with these differences in the subfigures. The figure caption should also explain what we are supposed to get out of looking at this figure.
Specific comments:
 - I suggest bolding terms where they are defined; this makes it easy for people to scan/find (e.g. Jensen-Shannon divergence, Integral Probability Metrics, witness functions, Wasserstein distance, etc.)
 - Although they are common knowledge in the field, because this is a review it could be helpful to provide references or brief explanations of e.g. JSD, KL, Wasserstein distance, RKHS, etc.
 - a flow chart (of GANs, IPMs, MMD, etc., mentioning a few more models than are discussed in depth here, would be *very* helpful.
 - page 2, middle paragraph, you mention "...constraints to ensure the kernel distribution embeddings remained injective"; it would be helpful to add a sentence here to explain why that's a good thing.


We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data. Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior. In order to obtain a valid generalization bound, we show that an ε-differentially private prior yields a valid PAC-Bayes bound, a straightforward consequence of results connecting generalization with differential privacy. Using stochastic gradient Langevin dynamics (SGLD) to approximate the wellknown exponential release mechanism, we observe that generalization error on MNIST (measured on held out data) falls within the (empirically nonvacuous) bounds computed under the assumption that SGLD produces perfect samples. In particular, Entropy-SGLD can be configured to yield relatively tight generalization bounds and still fit real labels, although these same settings do not obtain state-of-the-art performance.

Brief summary:
    Assume any neural net model with weights w. Assume a prior P on the weights. PAC-Bayes risk bound show that for ALL other distributions Q on the weights, the the sample risk (w.r.t to the samples in the data set) and expected risk (w.r.t distribution generating samples) of the random classifier chosen according to Q, averaged over Q, are close by a fudge factor that is KL divergence of P and Q scaled by m^{-1} + some constant.
Now, the authors first show that optimizing the objective of the Entropy SGD algorithm is equivalent to optimizing the empiricial risk term + fudge term over all data dependent priors P and the best Q for that prior. However, PAC-Bayes bound  holds only when P is NOT dependent on the data. So the authors invoke results from differential privacy to show that as long as the prior choosing mechanism in the optimization algorithm is differentially private with respect to data, differentially private priors can be substituted for valid PAC-Bayes bounds rectifying the issue. They show that when entrop SGD is implemented with pure gibbs sampling steps (as in Algorithm 3), the bounds hold.
Weakness that remains is that the gibbs sampling step in Entropy SGD (as in algo 3 in the appendix) is actually approximated by samples from SGLD that converges to this gibbs distribution when run for infinite hops. The authors leave this hole unsolved. But under the very strong sampling assumption, the bound holds. The authors do some experiments with MNIST to demonstrate that their bounds are not trivial.
Strengths:
  Simple connections between PAC-Bayes bound and entropy SGD objective is the first novelty. Invoking results from differential privacy for fixing the issue of validity of PAC-Bayes bound is the second novelty. Although technically the paper is not very deep, leveraging existing results (with strong assumptions) to show generalization properties of entropy-SGD is good.
Weakness:
  a) Obvious issue : that analysis assumes the strong gibbs sampling step.
  b) Experimental results are ok. I see that the bounds computed are non-vacuous. - but can the authors clarify what exactly they seek to justify ?
 c) Typos:
   Page 4 footnote "the local entropy should not be <with>.." - with is missing.
   Eq 14 typo - r(h) instead of e(h)
   Definition A.2 in appendix - must have S and S' in the inequality -both seem S.
d) Most important clarification: The way Thm 5.1, 5.2 and the exact gibbs sampling step connect with each other to produce Thm 6.1 is in Thm B.1. How do multiple calls on the same data sample do not degrade the loss ? Explanation is needed. Because the whole process of optimization in TRAIN with may steps is the final 'data dependent prior choosing mechanism' that has to be shown to be differentially private. Can the authors argue why the number of iterations of this does not matter at all ?? If I get run this long enough, and if I get several w's in the process (like step 8 repeated many times in algorithm 3) I should have more leakage about the data sample S intuitively right ?
e) The paper is unclear in many places. Intro could be better written to highlight the connection at the expression level of PAC-Bayes bound and entropy SGD objective and the subsequent fix using differentially private prior choosing mechanism to make the connection provably correct. Why are all the algorithms in the appendix on which the theorems are claimed in the paper ??
Final decision: I waver between 6 and 7 actually. However I am willing to upgrade to 7 if the authors can provide sound arguments to my above concerns.


Published as a conference paper at ICLR 2018 DEEP LEARNING WITH LOGGED BANDIT FEEDBACK
We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training. Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data. To this effect, we propose a counterfactual risk minimization approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the resulting objective can be decomposed in a way that allows stochastic gradient descent training. We empirically demonstrate the effectiveness of the method by showing how deep networks -ResNets in particular -can be trained for object recognition without conventionally labeled images.

This paper proposes a new output layer in neural networks, which allows them to use logged contextual bandit feedback for training. The paper is well written and well structured.
General feedback:
I would say the problem addressed concerns stochastic learning in general, not just SGD for training neural nets. And it's not a "new output layer", but just a softmax output layer (Eq. 1) with an IPS+baseline training objective (Eq. 16).
Others:
- The baseline in REINFORCE (Williams'92), which is equivalent to introduced Lagrange multiplier, is well known and well defined as control variate in Monte Carlo simulation, certainly not an "ad-hoc heuristic" as claimed in the paper [see Greensmith et al. (2004). Variance Reduction for Gradient Estimates in Reinforcement Learning, JMLR 5.]
- Bandit to supervised conversion: please add a supervised baseline system trained just on instances with top feedbacks -- this should be a much more interesting and relevant strong baseline. There are multiple indications that this bandit-to-supervised baseline is hard to outperform in a number of important applications.
- The final objective IPS^lambda is identical to IPS with a translated loss and thus re-introduces problems of IPS in exactly the same form that the article claims to address, namely:
    * the estimate is not bounded by the range of delta
    * the importance sampling ratios can be large; samples with high such ratios lead to larger gradients thus dominating the updates. The control variate of the SNIPS objective can be seen as defining a probability distribution over the log, thus ensuring that for each sample that sample’s delta is multiplied by a value in [0,1] and not by a large importance sampling ratio.
    * IPS^lambda introduces a grid search which takes more time and the best value for lambda might not even be tested. How do you deal with it?
- As author note, IPS^lambda is very similar to an RL-baseline, so results of using IPS with it should be reported as well:
    In more detail, Note:
    1. IPS for losses<0 and risk minimization: raise the probability of every sample in the log irrespective of the loss itself
    2. IPS for losses>0 and risk minimization: lower the same probability
    3. IPS^lambda: by the translation of the loss, it divides the log into 2 groups: a group whose probabilities will be lowered and a group whose probabilities will be raised (and a third group for delta=lambda but the objective will be agnostic to these)
    4. IPS with a baseline would do something similar but changes over time, which means the above groups are not fixed and might work better. Furthermore, there is no hyperparameter/grid search required for the simple RL-baseline
    -> results of using IPS with the RL-baseline should be reported for the BanditNet rows in Table 1 and in CIFAR-10 experiments.
- What is the feedback in the CIFAR-10 experiments? Assuming it's from [0..1], and given the tested range of lambdas, you should run into the same problems with IPS and its degenerate solutions for lambdas >=1.0. In general, how are your methods behaving for lambda* (corresponding to S*) such that makes all difference (delta_i - lambda*) positive or negative?
- The claim of Theorem 2 in appendix B does not follow from its proof: what is proven is that the value of S(w) lies in an interval [1-e..1+e] with a certain probability for all w. It says nothing about a solution of an optimization problem of the form f(w)/S(w) or its constrained version. Actually, the proof never makes any connection to optimization.
- What the appendix C basically claims is that it's not possible to get an unbiased estimate of a gradient for a certain class of non-convex ratios with a finite-sum structure. This would contradict some previously established convergence results for this type of problems: Reddi et al. (2016) Stochastic Variance Reduction for Nonconvex Optimization, ICML and Wang et al. 2013. Variance Reduction for Stochastic Gradient Optimization, NIPS. On the other hand, there seem to be no need to prove such a claim in the first claim, since the difficulty of performing self-normalized IPS on GPU should be evident, if one remembers that the normalization should run over the whole logged dataset (while only the current mini-batch is accessible to the GPU).


In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for automatically designing and training deep models. We propose an extensible and modular language that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters. The resulting search spaces are treestructured and therefore easy to traverse. Models can be automatically compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequential model-based optimization (SMBO). We present experiments comparing the different algorithms on CIFAR-10 and show that MCTS and SMBO outperform random search. We also present experiments on MNIST, showing that the same search space achieves near state-of-the-art performance with a few samples. These experiments show that our framework can be used effectively for model discovery, as it is possible to describe expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly available.

Monte-Carlo Tree Search is a reasonable and promising approach to hyperparameter optimization or algorithm configuration in search spaces that involve conditional structure.
This paper must acknowledge more explicitly that it is not the first to take a graph-search approach. The cited work related to SMAC and Hyperopt / TPE addresses this problem similarly. The technique of separating a description language from the optimization algorithm is also used in both of these projects / lines of research. The [mis-cited] paper titled “Making a science of model search …” is about using TPE to configure 1, 2, and 3 layer convnets for several datasets, including CIFAR-10. SMAC and Hyperopt have been used to search large search spaces involving pre-processing and classification algorithms (e.g. auto-sklearn, autoweka, hyperopt-sklearn). There have been near-annual workshops on AutoML and Bayesian optimization at NIPS and ICML (see e.g. automl.org).
There is a benchmark suite of hyperparameter optimization problems that would be a better way to evaluate MCTS as a hyperparameter optimization algorithm: http://www.ml4aad.org/automl/hpolib/


Published as a conference paper at ICLR 2018 UNDERSTANDING DEEP NEURAL NETWORKS WITH RECTIFIED LINEAR UNITS
In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to global optimality with runtime polynomial in the data size albeit exponential in the input dimension. Further, we improve on the known lower bounds on size (from exponential to super exponential) for approximating a ReLU deep net function by a shallower ReLU net. Our gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature. An example consequence of our gap theorems is the following: for every natural number k there exists a function representable by a ReLU DNN with k 2 hidden layers and total size k 3 , such that any ReLU DNN with at most k hidden layers will require at least 1 2 k k+1 − 1 total nodes. Finally, for the family of R n → R DNNs with ReLU activations, we show a new lowerbound on the number of affine pieces, which is larger than previous constructions in certain regimes of the network architecture and most distinctively our lowerbound is demonstrated by an explicit construction of a smoothly parameterized family of functions attaining this scaling. Our construction utilizes the theory of zonotopes from polyhedral theory.

The paper presents a series of definitions and results elucidating details about the functions representable by ReLU networks, their parametrisation, and gaps between deep and shallower nets.
The paper is easy to read, although it does not seem to have a main focus (exponential gaps vs. optimisation vs. universal approximation). The paper makes a nice contribution to the details of deep neural networks with ReLUs, although I find the contributed results slightly overstated. The 1d results are not difficult to derive from previous results. The advertised new results on the asymptotic behaviour assume a first layer that dominates the size of the network. The optimisation method appears close to brute force and is limited to 2 layers.
Theorem 3.1 appears to be easily deduced from the results from Montufar, Pascanu, Cho, Bengio, 2014. For 1d inputs, each layer will multiply the number of regions at most by the number of units in the layer, leading to the condition w’ \geq w^{k/k’}. Theorem 3.2 is simply giving a parametrization of the functions, removing symmetries of the units in the layers.
In the list at the top of page 5. Note that, the function classes might be characterized in terms of countable properties, such as the number of linear regions as discussed in MPCB, but still they build a continuum of functions. Similarly, in page 5 ``Moreover, for fixed n,k,s, our functions are smoothly parameterized''. This should not be a surprise.
In the last paragraph of Section 3 ``m = w^k-1'' This is a very big first layer. This also seems to subsume the first condition, s\geq  w^k-1 +w(k-1) for the network discussed in Theorem 3.9. In the last paragraph of Section 3 ``To the best of our knowledge''. In the construction presented here, the network’s size is essentially in the layer of size m. Under such conditions, Corollary 6 of MPCB also reads as s^n. Here it is irrelevant whether one artificially increases the depth of the network by additional, very narrow, layers, which do not contribute to the asymptotic number of units.
The function class Zonotope is a composition of two parts. It would be interesting to consider also a single construction, instead of the composition of two constructions.
Theorem 3.9 (ii) it would be nice to have a construction where the size becomes 2m + wk when k’=k.
Section 4, while interesting, appears to be somewhat disconnected from the rest of the paper.
In Theorem 2.3. explain why the two layer case is limited to n=1.
At some point in the first 4 pages it would be good to explain what is meant by ``hard’’ functions (e.g. functions that are hard to represent, as opposed to step functions, etc.)


Deep neural networks have led to a series of breakthroughs, dramatically improving the state-of-the-art in many domains. The techniques driving these advances, however, lack a formal method to account for model uncertainty. While the Bayesian approach to learning provides a solid theoretical framework to handle uncertainty, inference in Bayesian-inspired deep neural networks is difficult. In this paper, we provide a practical approach to Bayesian learning that relies on a regularization technique found in nearly every modern network, batch normalization. We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty. Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure. Our approach is thoroughly validated in a series of empirical experiments on different tasks and using various measures, showing it to outperform baselines on a majority of datasets with strong statistical significance.

The authors show how the regularization procedure called batch normalization,
currently being used by most deep learning systems, can be understood as
performing approximate Bayesian inference. The authors compare this approach to
Monte Carlo dropout (another regularization technique which can also be
considered to perform approximate Bayesian inference). The experiments
performed show that the Bayesian view of batch normalization performs similarly
as MC dropout in terms of the estimates of uncertainty that it produces.
Quality:
I found the quality to be low in some aspects. First, the description of what
is the prior used by batch normalization in section 3.3 is unsatisfactory. The
authors basically refer to Appendix 6.4 for the case in which the weight decay
penalty is not zero. The details in that Appendix are almost none, they just
say "it is thus possible to derive the prior...".
The results in Table 2 are a bit confusing. The authors should highlight in
bold face the results of the best performing method.
The authors indicate that they do not need to compare to variational methods
because Gal and Ghahramani 2015 compare already to those methods. However, Gal
and Ghahramani's code used Bayesian optimization methods to tune
hyper-parameters and this code contains a bug that optimizes hyper-parameters
by maximizing performance on the test data. In particular for hyperparameter
selection, they average performance across (subsets of) 5 of the training sets
from the 20x train/test split, and then using the tau which got the best
average performance for all of 20x train/test splits to evaluate performance:
https://github.com/yaringal/DropoutUncertaintyExps/blob/master/bostonHousing/net/experiment_BO.py#L54
Therefore, the claim that
"Since we have established that MCBN performs on par with MCDO, by proxy we
might conclude that MCBN outperforms those VI methods as well."
is not valid.
At the beginning of section 4.3 the authors indicate that they follow in their
experiments the setup of Gal and Ghahramani (2015). However, Gal and Ghahramani
(2015) actually follow Hernández-Lobato and Adams, 2015 so the correct
reference should be the latter one.
Clarity:
The paper is clearly written and easy to follow and understand.
I found confusing how to use the proposed method to obtain estimates of
uncertainty for a particular test data point x_star. The paragraph just above
section 4 says that the authors sample a batch of training data for this, but
assume that the test point x_star has to be included in this batch.
How is this actually done in practice?
Originality:
The proposed contribution is original. This is the first time that a Bayesian
interpretation has been given to the batch normalization regularization
proposal.
Significance:
The paper's contributions are significant. Batch normalization is a very
popular regularization technique and showing that it can be used to obtain
estimates of uncertainty is relevant and significant. Many existing deep
learning systems can use this to produce estimates of uncertainty in their
predictions.


THE POWER OF DEEPER NETWORKS FOR EXPRESSING NATURAL FUNCTIONS
It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only linearly with n for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from 1 to k, the neuron requirement grows exponentially not with n but with n 1/k , suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with n.

The paper investigates the representation of polynomials by neural networks up to a certain degree and implied uniform approximations. It shows exponential gaps between the width of shallow and deep networks required for approximating a given sparse polynomial.
By focusing on polynomials, the paper is able to use of a variety of tools (e.g. linear algebra) to investigate the representation question. Results such as Proposition 3.3 relate the representation of a polynomial up to a certain degree, to the approximation question. Here it would be good to be more specific about the domain, however, as approximating the low order terms certainly does not guarantee a global uniform approximation.
Theorem 3.4 makes an interesting claim, that a finite network size is sufficient to achieve the best possible approximation of a polynomial (the proof building on previous results, e.g. by Lin et al that I did not verify). The idea being to construct a superposition of Taylor approximations of the individual monomials. Here it would be good to be more specific about the domain. Also, in the discussion of Taylor series, it would be good to mention the point around which the series is developed, e.g. the origin.
The paper mentions that ``the theorem is false for rectified linear units (ReLUs), which are piecewise linear and do not admit a Taylor series''. However, a ReLU can also be approximated by a smooth function and a Taylor series.
Theorem 4.1 seems to be implied by Theorem 4.2. Similarly, parts of Section 4.2 seem to follow directly from the previous discussion.
In page 1 ```existence proofs' without explicit constructions'' This is not true, with numerous papers providing explicit constructions of functions that are representable by neural networks with specific types of activation functions.


Curriculum learning and Self paced learning are popular topics in the machine learning that suggest to put the training samples in order by considering their difficulty levels. Studies in these topics show that starting with a small training set and adding new samples according to difficulty levels improves the learning performance. In this paper we experimented that we can also obtain good results by adding the samples randomly without a meaningful order. We compared our method with classical training, Curriculum learning, Self paced learning and their reverse ordered versions. Results of the statistical tests show that the proposed method is better than classical method and similar with the others. These results point a new training regime that removes the process of difficulty level determination in Curriculum and Self paced learning and as successful as these methods.

This paper addresses an interesting problem of curriculum/self-paced versus random order of samples for faster learning. Specifically, the authors argue that adding samples in random order is as beneficial as adding them with some curriculum strategy, i.e. from easiest to hardest, or reverse.
The main learning strategy considered in this work is learning with growing sets, i.e. at each next stage a new portion of samples is added to the current available training set. At the last stage, all training samples are considered. The classifier is re-learned on each stage, where optimized weights in the previous stage are given as initial weights in the next stage.
The work has several flaws.
-First of all, it is not surprising that learning with more training samples at each next stage (growing sets) gets better - this is the basic principle of learning. The question is how fast the current classifier converges to the optimal Bayes level when using Curriculum strategy versus Random strategy. The empirical evaluations do not show evidence/disprove regarding this matter. For example, it could happen that the classifier converges to the optimal on the first stage already, so there is no difference when training in random versus curriculum order with growing sets.
-Secondly, easyness/hardness of the samples are defined w.r.t. some pre-trained (external) ensemble method. It is not clear how this definition of easiness/hardness translates when training the 3-layer neural network (final classifier). For example, it could well happen that all the samples are equally easy for training the final classifier, so the curriculum order would be the same as random order. In the original work on self-paced learning, Kumar et al (2010), easiness of the samples is re-computed on each stage of the classifier learning.
-The empirical evaluations are not clear. Just showing the wins across datasets without actual performance is not convincing (Table 2).
-I wonder whether the section with theoretical explanation is needed. What is the main advantage of learning with growing sets (when re-training the classifier)  and (traditional) learning when using the whole training dataset (last stage, in this work)?


Published as a conference paper at ICLR 2018 FEARNET: BRAIN-INSPIRED MODEL FOR INCREMENTAL LEARNING
Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall. FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.

Quality: The paper presents a novel solution to an incremental classification problem based on a dual memory system. The proposed solution is inspired by the memory storage mechanism in brain.
Clarity: The problem has been clearly described and the proposed solution is described in detail. The results of numerical experiments and the real data analysis are satisfactory and clearly shows the superior performance of the method compared to the existing ones.
Originality: The solution proposed is a novel one based on a dual memory system inspired by the memory storage mechanism in brain. The memory consolidation is inspired by the mechanisms that occur during sleep. The numerical experiments showing the FearNet performance with sleep frequency also validate the comparison with the brain memory system.
Significance: The work discusses a significant problem of incremental classification. Many of the shelf deep neural net methods require storage of previous training samples too and that slows up the application to larger dataset. Further the traditional deep neural net also suffers from the catastrophic forgetting. Hence, the proposed work provides a novel and scalable solution to the existing problem.
pros: (a) a scalable solution to the incremental classification problem using a brain inspired dual memory system
          (b) mitigates the catastrophic forgetting problem using a memory consolidation by pseudorehearsal.
          (c) introduction of a subsystem that allows which memory system to use for the classification
cons: (a)  How FearNet would perform if imbalanced classes are seen in more than one study sessions?
          (b) Storage of class statistics during pseudo rehearsal could be computationally expensive. How to cope with that?
          (c) How FearNet would handle if there are multiple data sources?


Convolutional neural networks (CNNs) have been generally acknowledged as one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes very hard to obtain, and it is desirable to be able to learn them from few examples. In this work, we address these limitations of CNNs by developing novel, simple, and interpretable models for few-shot learning. Our models are based on the idea of encoding objects in terms of visual concepts, which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of visual concepts to the few-shot setting, and then uncover two key properties of feature encoding using visual concepts, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being much more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using visual concepts helps expose the natural capability of CNNs for few-shot learning.

My main concern for this paper is that the description of the Visual Concepts is completely unclear for me. At some point I thought I did understand it, but then the next equation didnt make sense anymore... If I understand correctly, f_p is a representation of *all images* of a specific layer *k* at/around pixel "p", (According to last line of page 3). That would make sense, given that then the dimensions of the vector f_p is a scalar (activation value) per image for that image, in layer k, around pixel p. Then f_v is one of the centroids (named VCs). However, this doesnt seem to be the case, given that it is impossible to construct VC activations for specific images from this definition. So, it should be something else, but it does not become clear, what this f_p is. This is crucial in order to follow / judge the rest of the paper. Still I give it a try.
Section 4.1 is the second most important section of the paper, where properties of VCs are discussed. It has a few shortcomings. First, iIt is unclear why coverage should be >=0.8 and firerate ~ 1, according to the motivation firerate should equal to coverage: that is each pixel f_p is assigned to a single VC centroid. Second, "VCs tent to occur for a specific class", that seems rather a bold statement from a 6 class, 3 VCs experiment, where the class sensitivity is in the order 40-77%. Also the second experiment, which shows the spatial clustering for the "car wheel" VC, is unclear, how is the name "car wheel" assigned to the VC? That has have to be named after the EM process, given that EM is unsupervised. Finally the cost effectiveness training (3c), how come that the same "car wheel" (as in 3b) is discovered by the EM clustering? Is that coincidence? Or is there some form of supervision involved?
Minor remarks
- Table 1: the reported results of the Matching Network are different from the results in the paper of Vinyals (2016).
- It is unclear what the influence of the smoothing is, and how the smoothing parameter is estimated / set.
- The VCs are introduced for few-shot classification, unclear how this is different from "previous few-shot methods" (sect 5).
- 36x36 patches have a plausible size within a 84x84 image, this is rather large, do semantic parts really cover 20% of the image?
- How are the networks trained, with what objective, how validated, which training images? What is the influence of the layer on the performance?
- Influence of the clustering method on VCs, eg k-means, gaussian, von-mises (the last one is proposed)?
On a personal note, I've difficulties with part of the writing. For example, the introduction is written rather "arrogant" (not completely the right word, sorry for that), with a sentence, like "we have only limited insights into why CNNs are effective" seems overkill for the main research body. The used Visual Concepts (VCs) were already introduced by other works (Wangt'15), and is not a novelty. Also the authors refer to another paper (about using VCs for detection) which is also under submission (somewhere). Finally, the introduction paragraph of Section 5 is rather bold, "resembles the learning process of human beings"? Not so sure that is true, and it is not supported by a reference (or an experiment).
In conclusion:
This paper presents a method for creating features from a (pre-trained) ConvNet.
It clusters features from a specific pooling layer, and then creates a binary assignment between per image extracted feature vectors and the cluster centroids. These are used in a 1-NN classifier and a (smoothed) Naive Bayes classifier. The results show promising results, yet lack exploration of the model, at least to draw conclusions like "we address the challenge of understanding the internal visual cues of CNNs". I believe this paper needs to focus on the working of the VCs for few-shot experiments, showing the influences of some of the choices (layer, network layout, smoothing, clustering, etc). Moreover, the introduction should be rewritten, and the the background section of VCs (Sect 3) should be clarified. Therefore, I rate the current manuscript as a reject.
After rebuttal:
The writing of the paper greatly improved, still missing insights (see comments below). Therefore I've upgraded my rating, and due to better understanding now, als my confidence.


Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models combining the strengths of both models. Our contribution is to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder. In contrast, prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our approach results in models with meaningful latent variable representations, and which rely on powerful autoregressive decoders to model image details. Our model generates qualitatively convincing samples, and yields stateof-the-art quantitative results.

Summary:
This paper attempts to solve the problem of meaningfully combining variational autoencoders (VAEs) and PixelCNNs. It proposes to do this by simultaneously optimizing a VAE with PixelCNN++ decoder, and a VAE with factorial decoder. The model is evaluated in terms of log-likelihood (with no improvement over a PixelCNN++) and the visual appearance of samples and reconstructions.
Review:
Combining density networks (like VAEs) and autoregressive models is an unsolved problem and potentially very useful. To me, the most interesting bit of information in this paper was the realization that you can weight the reconstruction and KL terms of a VAE and interpret it as variational inference in a generative model with multiple copies of pixels (below Equation 7). Unfortunately the authors were unable to make any good use of this insight, and I will explain below why I don’t see any evidence of an improved generative model in this paper.
As the paper is written now, it is not clear what the goal of the authors is. Is it density estimation? Then the addition of the VAE had no measurable effect on the PixelCNN++’s performance, i.e., it seems like a bad idea due to the added complexity and loss of tractability. Is it representation learning? Then the paper is missing experiments to support the idea that the learned representations are in any way an improvement. Is it image synthesis (not a real application by itself), then the paper should have demonstrated the usefulness of the model on a real task and probably involve human subjects in a quantitative evaluation.
Much of the authors’ analysis is based on a qualitative evaluation of samples. However, samples can be very misleading. A lookup table storing the training data generates samples containing objects and perfect details, but obviously has not learned anything about either objects or the low-level statistics of natural images.
In contrast to the authors, I fail to see a meaningful difference between the groups of samples in Figure 1.
The VAE samples in Figure 3b) look quite smooth. Was independent Gaussian noise added to the VAE samples or are those (as is sometimes done) sampled means? If the former, what was sigma and how was it chosen?
On page 7, the authors conclude that “the pixelCNN clearly takes into account the output of the VAE decoder” based on the samples. Being a mixture model, a PixelCNN++ could easily represent the following mixture:
p(x | z) = 0.01 \prod_i p(x_i | x_{<i}) + 0.99 \prod_i p(x_i | z)
The first term is just like a regular PixelCNN++, ignoring the latent variables. The second term is just like a variational autoencoder with factorial decoder. The samples in this case would be dominated by the VAE, which depends on the latent state. The log-likelihood would be dominated by the first term and would be minimally effected (see Theis et al., 2016). Note that I am not saying that this is exactly what the model has learned. I am merely providing a possible counter example to the notion that the PixelCNN++ has learned to use of the latent representation in a meaningful way.
What happens if the KL term is simply downweighted but the factorial decoder is not included? This seems like it would be a useful control to include.
The paper is well written and clear.


mixup: BEYOND EMPIRICAL RISK MINIMIZATION
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

I enjoyed reading this well-written and easy-to-follow paper. The paper builds on the rather old idea of minimizing the empirical vicinal risk (Chapelle et al., 2000) instead of the empirical risk. The authors' contribution is to provide a particular instance of vicinity distribution, which amounts to linear interpolation between samples. This idea of linear interpolation on the training sample to generate additional (adversarial, in the words of the authors) data is definitely appealing to prevent overfitting and improve generalization performance at a mild computational cost (note that this comment does not just apply to deep learning). This notion is definitely of interest to machine learning, and to the ICLR community in particular. I have several comments and remarks on the concept of mixup, listed below in no particular order. My overall opinion on the paper is positive and I stand for acceptance, provided the authors answer the points below. I would especially be interested in discussing those with the authors.
1 - While data augmentation literature is well acknowledged in the paper, I would also like to see a comment on domain adaptation, which is a very closely related topic and of particular interest to the ICLR community.
2 - Paragraph after Eq. (1), starting with "Learning" and ending with "(Szegedy et al., 2014)": I am not so familiar with the term memorization, is this just a fancy way of talking about overfitting? If so, you might want to rephrase this paragraph with terms more used in the machine learning community. When you write "one trivial way to minimize [the empirical risk] is to memorize the training data", do you mean output a predictor which only delivers predictions on $X_i$, equal to $Y_i$? If so, this is again not specific to deep learning and I feel this should be a bit more discussed.
3 - I have not found in the paper a clear heuristics about how pairs of training samples should be picked to create interpolations. Picking at random is the simplest however I feel that a proximity measure on the space $\mathcal{X}$ on which samples live would come in handy. For example, sampling with a probability decreasing as the Euclidean distance seems a natural idea. In any case, I strongly feel this discussion is missing in the paper.
4 - On a related note, I would like to see a discussion on how many "adversarial" examples should be used. Since the computational overhead cost of computing one new sample is reasonable (sampling from a Beta distribution + one addition), I wonder why $m$ is not taken very large, yielding more accurate estimates of the empirical risk. A related question: under what conditions does the vicinal risk converge (in expectation for example) to the empirical risk? I think some comments would be nice.
5 - I am intrigued by the last paragraph of Section 5. What do the authors exactly have in mind when they suggest that mixup could be generalized to regression problems? As far as I understood the paper, since $\tilde{y}$ is defined as a linear interpolation between $y_i$ and $y_j$, this formulation only works for continuous $y$s, like in regression. This formulation is not straightforwardly transposable to classification for example. I therefore am quite confused about the fact that the authors present experiments on classification tasks, with a method that writes for regression.
6 - Writing linear interpolations to generate new data points implicitly makes the assumption that the input and output spaces ($\mathcal{X}$ and $\mathcal{Y}$) are convex. I have no clear intuition wether this is a limitation of the authors' proposed method but I strongly feel this should be carefully addressed by a comment in Section 2.


Understanding physical phenomena is a key component of human intelligence and enables physical interaction with previously unseen environments. In this paper, we study how an artificial agent can autonomously acquire this intuition through interaction with the environment. We created a synthetic block stacking environment with physics simulation in which the agent can learn a policy end-to-end through trial and error. Thereby, we bypass to explicitly model physical knowledge within the policy. We are specifically interested in tasks that require the agent to reach a given goal state that may be different for every new trial. To this end, we propose a deep reinforcement learning framework that learns policies for stacking tasks which are parametrized by a target structure -departing from conventional approaches based on simulation and planning. We validated the model on a toy example navigating in a grid world with different target positions and in a block stacking task with different target structures of the final tower. In contrast to prior work, our policies show better generalization across different goals.

Summary: This paper proposes to use deep Q-learning to learn how to reconstruct a given tower of blocks, where DQN is also parameterized by the desired goal state in addition to the current observed state.
Pros:
- Impressive results on a difficult block-stacking task.
Cons:
- The idea of parameterizing an RL algorithm by goals is not particularly novel.
Quality and Clarity:
The paper is extremely well-written, easy to follow, and largely technically correct, though I am somewhat concerned about how the results were obtained as it does not seem like the vanilla DQN agent could do so well, even on the 2-block scenes. Even just including stable scenes, I estimated based on Figure 5 that there must be about 70 different configurations that are stable (and this is likely an underestimate). So, if each of these scenes occurs equally often and the vanilla DQN agent does not receive any information about the target goal and just acts based on an "average" policy, I would expect it to only achieve success about 1/70th of the time. Am I missing something here?
Another thing that was unclear to me is how the rotation of the blocks is chosen: is the agent given the next block with the correct rotation, or can it also choose to rotate the block? In the text it is implied that the only actions are {left, right, down}, which seems to simplify the task immensely. It would be interesting to include results where the agent additionally has to choose from actions of {rotate left by 90 degrees, rotate right by 90 degrees}.
Also: are the scenes used during testing separate from those used during training? If not, it's not obvious that the agent isn't just learning to memorize the solution (which somewhat defeats the idea behind parameterizing the Q-network with new goals every time).
Originality and Significance:
The block-stacking task is very cool and is more complex than many other physics-based RL tasks in the literature, which often involve just stacking square blocks in a single tower. I think it is a useful contribution to introduce this task and the GDQN agent as a baseline. However, the notion of parameterizing the policy by the goal state is not particularly novel. While it is true that many RL papers do train to optimize just a single reward function for a single goal, it is also very straightforward to modify the state space to include a goal and indeed [1-4] are just a few examples of recent papers that have done this. In general, any time there is a procedurally generated environment (e.g. Sokoban, as in [5]) the goal necessarily is included as part of the state space---so the idea of GDQN isn't really that new.
[1] Oh, J., Singh, S., Lee, H., & Kohli, P. (2017). Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning. arXiv Preprint arXiv:1706.05064.
[2] Dosovitskiy, A., & Koltun, V. (2017). Learning to act by predicting the future. Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).
[3] Hamrick, J. B., Ballard, A. J., Pascanu, R., Vinyals, O., Heess, N., & Battaglia, P. W. (2017). Metacontrol for adaptive imagination-based optimization. Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).
[4] Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racanière, S., … Battaglia, P. (2017). Learning model-based planning from scratch. arXiv Preprint arXiv: 1707.06170. Retrieved from https://arxiv.org/abs/1707.06170
[5] Weber, T., Racanière, S., Reichert, D. P., Buesing, L., Guez, A., Rezende, D. J., … Wierstra, D. (2017). Imagination-Augmented Agents for Deep Reinforcement Learning. arXiv Preprint arXiv: 1707.06203. Retrieved from http://arxiv.org/abs/1707.06203


State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15 − 40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.

The paper presentes a small extension to the Neural Transformer model of Vaswani et al 2017:
the multi-head attention computation (eq. 2,3):
head_i = Attention_i(Q,K,W)
MultiHead = Concat_i(head_i) * W = \sum_i head_i * W_i
is replaced with the so-called BranchedAttention (eq. 5,6,7,4):
head_i = Attention_i(Q,K,W)       // same as in the base model
BranchedAttention = \sum_i \alpha_i max(0, head_i * W_i * kappa_i * W^1 + b^1) W^2 + b^2
The main difference is that the results of application of each attention head is post-processed with a 2-layer ReLU network before being summed into the aggregated attention vector.
My main problem with the paper is understanding what really is implemented: the paper states that with alpha_i=1 and kappa_i=1 the two attention mechanism are equivalent. The equations, however, tell a different story: the original MultiHead attention quickly aggregates all attention heads, while the proposed BranchedAttention adds another processing step, effectively adding depth to the model.
Since the BranchedAttention is the key novelty of the paper, I am confused by this contradiction and treat it as a fatal flaw of this paper (I am willing to revise my score if the authors explain the equations) - the proposed attention either adds a small amount of parameters (the alphas and kappas) that can be absorbed by the other weights of the network, and the added alphas and kappas are easier/faster to optimize, as the authors state in the text, or the BranchedAttention works as shown in the equations, and effectively adds depth to the network by processing each attention's result with a small MLP before combining multiple attention heads. This has to be clarified before the paper is published.
The experiment show that the proposed change speeds convergence and improves the results by about 1 BLEU point. However, this requires a different learning rate schedule for the introduced parameters and some non-standard tricks, such as freezing the alphas and kappas during the end of the training.
I also have a questions about the presented results:
1) The numbers for the original transformer match the ones in Vaswani et al 2017, am I correct to assume that the authors did not rerun the tensor2tensor code and simply copied them from the paper?
2) Is all of the experimental setup the same as in Vaswani et al 2017? Are the results obtained using their tensor2tensor implementation, or are some hyperparameters different?
Detailed review:
Quality:
The equations and text in the paper contradict each other.
Clarity:
The language is clear, but the main contribution could be better explained.
Originality:
The proposed change is a small extension to the Neural Transformer model.
Significance:
Rather small, the proposed addition adds little modeling power to the network and its advantage may vanish with more data/different learning rate schedule.
Pros and cons:
+ the proposed approach is a simple way to improve the performance of multihead attentional models.
- it is not clear from the paper how the proposed extension works: does it regularize the model or dies it increase its capacity?


Published as a conference paper at ICLR 2018 RESIDUAL CONNECTIONS ENCOURAGE ITERATIVE IN- FERENCE
Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively, overfitting, and we show that simple existing strategies can help alleviating this problem.

This paper shows that residual networks can be viewed as doing a sort of iterative inference, where each layer is trained to use its “nonlinear part” to push its values in the negative direction of the loss gradient.  The authors demonstrate this using a Taylor expansion of a standard residual block first, then follow up with several experiments that corroborate this interpretation of iterative inference.  Overall the strength of this paper is that the main insight is quite interesting — though many people have informally thought of residual networks as having this interpretation — this paper is the first one to my knowledge to explain the intuition in a more precise way.
Some weaknesses of the paper on the other hand — some of the parts of the paper (e.g. on weight sharing) are only somewhat related to the main topic of the paper. In fact, the authors moved the connection to SGD to the appendix, which I thought would be *more* related.   Additionally, parts of the paper are not as clearly written as they could be and lack rigor.  This includes the mathematical derivation of the main insight — some of the steps should be spelled out more explicitly.  The explanation following is also handwavey despite claims to being formal.
Some other lower level thoughts:
* Regarding weight sharing for residual layers, I don’t understand why we can draw the conclusion that the initial gradient explosion is responsible for the lower generalization capability of the model with shared weights.  Are there other papers in literature that have shown this connection?
* The name “cosine loss” suggests that this function is actually being minimized by a training procedure, but it is just a value that is being plotted… perhaps just call it the cosine?
* I recommend that the authors also check out Figurnov et al CVPR 2017 ("Spatially Adaptive Computation Time for Residual Networks") which proposes an “adaptive” version of ResNet based on the intuition of adaptive inference.
* The plots in the later parts of the paper are quite small and hard to read.  They are also spaced together too tightly (horizontally), making it difficult to immediately see what each plot is supposed to represent via the y-axis label.
* Finally, the citations need to be fixed (use \citep{} instead of \cite{})


Published as a conference paper at ICLR 2018 SGD LEARNS OVER-PARAMETERIZED NETWORKS THAT PROVABLY GENERALIZE ON LINEARLY SEPARA- BLE DATA
Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations. Nonetheless, current generalization bounds for neural networks fail to explain this phenomenon. In an attempt to bridge this gap, we study the problem of learning a two-layer over-parameterized neural network, when the data is generated by a linearly separable function. In the case where the network has Leaky ReLU activations and only the first layer is trained, we provide both optimization and generalization guarantees for over-parameterized networks. Specifically, we prove convergence rates of SGD to a global minimum, and provide generalization guarantees for this global minimum that are independent of the network size. Therefore, our result clearly shows that the use of SGD for optimization both finds a global minimum, and avoids overfitting despite the high capacity of the model. This is the first theoretical demonstration that SGD can avoid overfitting, when learning over-specified neural network classifiers.

This paper shows that on linearly separable data, SGD on a overparametrized network (one hidden layer, with leaky ReLU activations) can still lean a classifier that provably generalizes. The assumption on data and structure of network is a bit strong, but this is the first result that achieves a number of desirable properties
``1. Works for overparametrized network
2. Finds global optimal solution for a non-convex network.
3. Has generalization guarantees (and generalization is related to the SGD algorithm).
4. Number of samples need not depend on the number of neurons.
There have been several papers achieving 1 and 2 (with much weaker assumptions), but they do not have 3 and 4. The proof of the optimization part is very similar to the proof of perceptron algorithm, and really relies on linear separability. The proof of generalization is based on a compression argument, where if an algorithm does not take many nonzero steps, then it must have good generalization. Ideally, one would also want to see a result where overparametrization actually helps (in the main result the whole data can be learned by a linear classifier). This is somewhat achieved when the activation is replaced with standard ReLU, where the paper showed with a small number of hidden units the algorithm is likely to get stuck at a local minima, but with enough hidden units the algorithm is likely to converge (but even in this case, the data is still linearly separable and can be learned just by a perceptron).
The main concern about the paper is the possibility of generalizing the result. The algorithm part seems to heavily rely on the linear separable assumption. The generalization part relies on not making many non-zero updates, which is not really true in realistic settings (where the data is accessed in multiple passes) [After author response: Yes in the linearly separable case with hinge loss it is quite possible that the number of updates is sublinear. However what I meant here is that with more complicated data and different loss functions it is hard to believe that this can still hold.]. The related work section is also a bit unfair to some of the other generalization results (e.g. Bartlett et al. Neyshabur et al.): those results work on more general network settings, and it's not completely clear that they cannot be related to the algorithm because they rely on certain solution specific quantities (such as spectral/Frobenius norms of the weight matrices) and it could be possible that SGD tends to find a solution with small norm (which can be proved in linear setting and might also be provable for the setting of this paper) [This is addressed in the author response].
Overall, even though the assumptions might be a bit strong, I think this is an interesting result working towards a good direction and should be accepted.


In this work we propose a novel approach for learning graph representation of the data using gradients obtained via backpropagation. Next we build a neural network architecture compatible with our optimization approach and motivated by graph filtering in the vertex domain. We demonstrate that the learned graph has richer structure than often used nearest neighbors graphs constructed based on features similarity. Our experiments demonstrate that we can improve prediction quality for several convolution on graphs architectures, while others appeared to be insensitive to the input graph.

Learning adjacency matrix of a graph with sparsely connected undirected graph with nonnegative edge weights is the goal of this paper. A projected sub-gradient descent algorithm is used. The UPS optimizer by itself is not new.
Graph Polynomial Signal (GPS) neural network is proposed to address two shortcomings of GSP using linear polynomial graph filter. First, a nonlinear function sigma in (8) is used, and second, weights are shared among neighbors of every data points. There are some concerns about this network that need to be clarified:
1. sigma is never clarified in the main context or experiments
2. the shared weights should be relevant to the ordering of neighbors, instead of the set of neighbors without ordering, in which case, the sharing looks random.
3. another explanation about the weights as the rescaling to matrix A needs to further clarified. As authors mentioned that the magnitude of |A| from L1 norm might be detrimental for the prediction. What is the disagreement between L1 penalty and prediction quality? Why not apply these weights to L1 norm as a weighted L1 norm to control the scaling of A?
4. Authors stated that the last step is to build a mapping from the GPS features into the response Y. They mentioned that linear fully connected layer or a more complex neural network can be build on top of the GPS features. However, no detailed information is given in the paper. In the experiments, authors only stated that “we fit the GPS architecture using UPS optimizer for varying degree of the neighborhood of the graph”, and then the graph is used to train existing models as the input of the graph. Which architecture is used for building the mapping ?
In the experimental results, detailed definition or explanation of the compared methods and different settings should be clarified. For example, what is GPS 8, GCN_2 Eq. 9 in Table 1, and GCN_3 9 and GPS_1, GPS_2, GPS_3 and so on. More explanations of Figure 2 and the visualization method can be great helpful to understand the advantages of the proposed algorithm.


Existing multi-agent reinforcement learning (MARL) communication methods have relied on a trusted third party (TTP) to distribute reward to agents, leaving them inapplicable in peer-to-peer environments. This paper proposes reward distribution using Neuron as an Agent (NaaA) in MARL without a TTP with two key ideas: (i) inter-agent reward distribution and (ii) auction theory. Auction theory is introduced because inter-agent reward distribution is insufficient for optimization. Agents in NaaA maximize their profits (the difference between reward and cost) and, as a theoretical result, the auction mechanism is shown to have agents autonomously evaluate counterfactual returns as the values of other agents. NaaA enables representation trades in peer-to-peer environments, ultimately regarding unit in neural networks as agents. Finally, numerical experiments (a single-agent environment from OpenAI Gym and a multi-agent environment from ViZDoom) confirm that NaaA framework optimization leads to better performance in reinforcement learning.
To the best of our knowledge, no existing literature discusses reward distributions in the configuration described above. Because CommNet assumes an environment that distributes a uniform reward to all the agents, if the distributed reward is in limited supply (such as money), it causes the Tragedy of the Commons (Lloyd, 1833), where the reward of contributing agents will be reduced due to the participation of free riders. Although there are several MARL methods for distributing rewards ac-  (Agogino & Tumer, 2006;Sukhbaatar et al., 2016;Foerster et al., 2016;2017). They should suppose TTP to distribute the optimal reward to the agents. (b) Inter-agent reward distribution model (our model). Some agents receive reward from the environment directly, and redistribute to other agents. The idea to determine the optimal reward without TTP is playing auction game among the agents.
cording to agents' contribution such as QUICR (Agogino & Tumer, 2006) and COMA (Sukhbaatar et al., 2016), they suppose the existence of TTP and hence cannot be applied to the situation investigated here.
The proposed method, Neuron as an Agent (NaaA), extends CommNet to actualize reward distributions in MARL without TTP based on two key ideas: (i) inter-agent reward distribution and (ii) auction theory. Auction theory was introduced because inter-agent reward distributions were insufficient for optimization. Agents in NaaA maximize profit, the difference between their received rewards and the costs which they redistribute to other agents. If the framework is naively optimized, a trivial solution is obtained where agents reduce their costs to zero to maximize profits. Then, NaaA employs the auction theory in game design to prevent costs from dropping below their necessary level. As a theoretical result, we show that agents autonomously evaluate the counterfactual return as values of other agents. The counterfactual return is equal to the discounted cumulative sum of counterfactual reward (Agogino & Tumer, 2006) distributed by QUICR and COMA.
NaaA enables representation trades in peer-to-peer environments and, ultimately, regards neural network units as agents. As NaaA is capable of regarding units as agents without losing generality, this setting was utilized in the current study. The concept of the proposed method is illustrated in Figure 1.
An environment extending ViZDoom (Kempka et al., 2016), a POMDP environment, to MARL was used for the experiment. Two agents, a cameraman sending information and a main player defeating enemies with a gun, were placed in the environment. Results confirmed that the cameraman learned cooperative actions for sending information from dead angles (behind the main player) and outperformed CommNet in score.
Interestingly, NaaA can apply to single-and multi-agent settings, since it learns optimal topology between the units. Adaptive DropConnect (ADC), which combines DropConnect (Wan et al., 2013) (randomly masking topology) with an adaptive algorithm (which has a higher probability of pruning connections with lower counterfactual returns) was proposed as a further application for NaaA. Experimental classification and reinforcement learning task results showed ADC outperformed DropConnect.
The remainder of this paper is organized as follows. In the next section, we show the problem setting. Then, we show proposed method with two key ideas: inter-agent reward distribution and auction theory in Section 3. After related works are introduced in Section 4, the experimental results are shown in classification, single-agent RL and MARL in Section 5. Finally, a conclusion ends the paper.

This paper proposed a novel framework Neuron as an Agent (NaaA) for training neural networks to perform various machine learning tasks, including classification (supervised learning) and sequential decision making (reinforcement learning). The NaaA framework is based on the idea of treating all neural network units as self-interested agents and optimizes the neural network as a multi-agent RL problem. This paper also proposes adaptive dropconnect, which extends dropconnect (Wan et al., 2013) by using an adaptive algorithm for masking network topology.

This work attempts to bring several fundamental principles in game theory to solve neural network optimization problems in deep learning. Although the ideas are interest and technically sound, and the proposed algorithms are demonstrated to outperform several baselines in various machine learning tasks, there several major problems with this paper, including lacking clarity of presentation, insights and substantiations of many claims. These issues may need a significant amount of effort to fix as I will elaborate more below.

1. Introduction
There are several important concepts, such as reward distribution, credit assignment, which are used (from the very beginning of the paper) without explanation until the final part of the paper.

The motivation of the work is not very clear. There seems to be a gap between the first paragraph and the second paragraph. The authors mentioned that “From a micro perspective, the abstraction capability of each unit contribute to the return of the entire system. Therefore, we address the following questions. Will reinforcement learning work even if we consider each unit as an autonomous agent ”
Is there any citation for the claim “From a micro perspective, the abstraction capability of each unit contribute to the return of the entire system” ?  It seems to me this is a very general claim. Even RL methods with linear function approximations use abstractions.  Also, it is unclear to me why this is an interest question. Does it have anything to do with existing issues in DRL? Moreover, The definition of autonomous agent is not clear, do you mean learning agent or policy execution agent?

“it uses \epsilon-greedy as a policy, …” Do you mean exploration policy?
I also have some concerns regarding the claim that “We confirm that optimization with the framework of NaaA leads to better performance of RL”. Since there are only two baselines are compared to the proposed method, this claim seems too general to be true.

It is not clear to why the authors mention that “negative result that the return decreases if we naively consider units as agents”. What is the big picture behind this claim?

“The counterfactual return is that by which we extend reward …” need to be rewritten.

The last paragraph of introduction discussed the possible applications of the proposed methods without any substantiation, especially neither citations nor any related experiments of the authors are provided.

2 Related Work

“POSG, a class of reinforcement learning with multiple ..” -> reinforcement learning framework

“Another one is credit assignment. Instead of reward.. ” Two sentences are disconnected and need to be rewritten.

“This paper unifies both issues” sounds very weird. Do you mean “solves/considers both issues in a principled way”?

The introduction of GAN is very abrupt. Rather than starting from introducing those new concepts directly, it might be better to mention that the proposed method is related to many important concepts in game theory and GANs.

“,which we propose in a later part of this paper” -> which we propose in this paper

3. Background

“a function from the state and the action of an agent to the real value” -> a reward function

Should provide a citation for DRQN

There is a big gap between the last two paragraphs of section 3.

4. Neuro as an agent

“We add the following assumption for characteristics of the v_i” -> assumptions for characterizing v_i

“to maximize toward maximizing its own return” -> to maximize its own return

We construct the framework of NaaA from the assumptions -> from these assumptions

“indicates that the unit gives additional value to the obtained data. …” I am not sure what this sentence means, given that \rho_ijt is not clearly defined.

5. Optimization

“NaaA assumes that all agents are not cooperative but selfish” Why? Is there any justification for such a claim?

What is the relation between \rho_jit and q_it ?

“A buyer which cannot receive the activation approximates x_i with …” It is unclear why a buyer need to do so given that it cannot receive the activation anyway.

“Q_it maximizing the equation is designated as the optimal price.” Which equation?

e_j and 0 are not defined in equation 8

6 Experiment
setare -> set are

what is the std for CartPole in table 1

It is hard to judge the significance of the results on the left side of figure 2. It might be better to add errorbars to those curves

More description should be provided to explain the reward visualization on the right side of figure 2. What reward? External/internal?

“Specifically, it is applicable to various methods as described below …” Related papers should be cited.


Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable. In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers. The key challenging task in learningbased motion planning is to learn a transformation from terrain observations to a suitable navigation reward function. In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN). SVIN is designed to produce more effective training gradients through the value iteration network. It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action. We demonstrate the effectiveness of the proposed method in robot motion planning scenarios. In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the Curiosity rover that is currently operating on Mars.

Summary:
The submission proposes a simple modification to the Value Iteration Networks (VIN) method of Tamar et al., basically consisting of assuming a stochastic policy and replacing the max-over-actions in value iteration with an expectation that weights actions proportional to their exponentiated Q-values. Since this change removes the main nondifferentiability of VINs, it is hypothesized that the resulting method will be easier to train than VINs, and experiments seem to support this hypothesis.
Pros:
+ The proposed modification to VIN is simple, well-motivated, and addresses the nondifferentiability of VIN
+ Experiments on synthetic data demonstrate a significant improvement over the standard VIN method
Cons:
+ Some important references are missing (e.g., MaxEnt IOC with deep-learned features)
+ Although intuitive, more detailed justification could be provided for replacing the max-over-actions with an exponentially-weighted average
+ No baselines are provided for the experiments with real data
+ All the experimental scenarios are fairly simple (2D grid-worlds with discrete actions, 1-channel input features)
The proposed method is simple, well-motivated, and addresses a real concern in VINs, which is their nondifferentiability. Although many of the nonlinearities used in CNNs for computer vision applications are nondifferentiable, the theoretical grounds for using these in conjunction with gradient-based optimization is obviously questionable. Despite this, they are widely used for such applications because of strong empirical results showing that such nonlinearities are beneficial in image-processing applications. However, it would be incorrect to assume that because such nonlinearities work for image processing, they are also beneficial in the context of unrolling value iteration.
Replacing the max-over-actions with an exponentially-weighted average is an intuitively well-motivated alternative because, as the authors note, it incorporates the values of suboptimal actions during the training procedure. We would therefore expect better or faster training, as the values of these suboptimal actions can be updated more frequently. The (admittedly limited) experiments bear out this hypothesis.
Perhaps the most significant downside of this work is that it fails to acknowledge prior work in the RL and IOC literature that result in similar  “smoothed” or “softmax" Bellman updates: in particular, MaxEnt IOC [A] and linearly-solvable MDPs [B] both fall in this category. Both of those papers clearly derive approximate Bellman equations from modified optimal control principles; although I believe this is also possible for the proposed update (Eq. 11), along the lines of the sentence after Eq. 11, this should be made more explicit/rigorous, and the result compared to [A,B].
Another important missing reference is [C], which learned cost maps with deep neural networks in a MaxEnt IOC framework. As far as I can tell, the application is identical to that of the present paper, and [C] may have some advantages: for instance, [C] features a principled, fully-differentiable training objective while also avoiding having to backprop through the inference procedure, as in VIN. Again, this raises the question of how the proposed method compares to MaxEnt IOC, both theoretically and experimentally.
The experiments are also a bit lacking in a few ways. First, a baseline is only provided for the experiments with synthetic data. Although that experiment shows a promising, significant advantage over VIN, the lack of baselines for the experiment with real data is disappointing. Furthermore, the setting for the experiments is fairly simple, consisting of a grid-world with 1-channel input features. The setting is simple enough that even shallow IOC methods (e.g., [D]) would probably perform well; however, the deep IOC methods of [C] is also applicable and should probably also be evaluated as a baseline.
In summary, although the method proposes an intuitively reasonable modification to VIN that seems to outperform it in limited experiments, the submission fails to acknowledge important related work (especially the MaxEnt IOC methods of [A,D]) that may have significant theoretical and practical advantages. Unfortunately, I believe the original VIN paper also failed to articulate the precise advantages of VIN over this prior work—which is not to say there are none, but it is clear that VINs applied to problems as simple as the one considered here have real competitors in prior work. Clarifying this connection, both theoretically and experimentally, would make this work much stronger and would be a valuable contribution to the literature.
[A] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
[B] Todorov, Emanuel. "Linearly-solvable Markov decision problems." Advances in neural information processing systems. 2007.
[C] Wulfmeier et al. Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environments. IROS 2016
[D] Ratliff, Nathan D., David Silver, and J. Andrew Bagnell. "Learning to search: Functional gradient techniques for imitation learning." Autonomous Robots 27.1 (2009): 25-53.


L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions. However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks. To understand this phenomenon, we investigate how and why learning fails for strong regularization. Specifically, we examine how gradients change over time for different regularization strength and provide an analysis why the gradients diminish so fast when strong regularization is imposed. We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it. We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level. Experiment results show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets. Our source code is published.

The work was prompted by  an interesting observation: a phase transition can be observed in deep learning with stochastic gradient descent and Tikhonov regularization. When the regularization parameter exceeds a (data-dependent) threshold, the parameters of the model are driven to zero, thereby preventing any learning. The authors then propose to moderate this problem by letting the regularization parameter to be zero for 5 to 10 epochs, and then applying the "strong" penalty parameter. In their experimental results, the phase transition is not observed anymore with their protocol. This leads to better performances, by using penalty parameters that would have prevent learning with the usual protocol.
The problem targeted is important, in the sense that it reveals that some of the difficulties related to non-convexity and the use of SGD that are often overlooked. The proposed protocol is reported to work well, but since it is really ad hoc, it fails to convince the reader that it provides the right solution to the problem. I would have found much more satisfactory to either address the initialization issue by a proper warm-start strategy, or to explore standard optimization tools such as constrained optimization (i.e. Ivanov regularization) , that could be for example implemented by stochastic projected gradient or barrier functions. I think that the problem would be better handled that way than with the proposed strategy, which seems to rely only on a rather limited amount of experiments, and which may prove to be inefficient when dealing with big databases.
To summarize, I believe that the paper addresses an important point, but that the tools advocated are really rudimentary compared with what has been already proposed elsewhere.
Details :
- there is a typo in the definition of the proximal operator in Eq. (9)
- there are many unsubstantiated speculations in the comments of the experimental section that do not add value to the paper
- the figure showing the evolution of the magnitude of parameters arrives too late and could be completed by the evolution of the data-fitting term of the training criterion


The design of small molecules with bespoke properties is of central importance to drug discovery. However significant challenges yet remain for computational methods, despite recent advances such as deep recurrent networks and reinforcement learning strategies for sequence generation, and it can be difficult to compare results across different works. This work proposes 19 benchmarks selected by subject experts, expands smaller datasets previously used to approximately 1.1 million training molecules, and explores how to apply new reinforcement learning techniques effectively for molecular design. The benchmarks here, built as OpenAI Gym environments, will be open-sourced to encourage innovation in molecular design algorithms and to enable usage by those without a background in chemistry. Finally, this work explores recent development in reinforcement-learning methods with excellent sample complexity (the A2C and PPO algorithms) and investigates their behavior in molecular generation, demonstrating significant performance gains compared to standard reinforcement learning techniques.

The paper proposes a set of benchmarks for molecular design, and compares different deep models against them. The main contributions of the paper are 19 molecular design benchmarks (with chembl-23 dataset), including two molecular design evaluation criterias and comparison of some deep models using these benchmarks. The paper does not seem to include any method development.
The paper suffers from a lack of focus. Several existing models are discussed to some length, while the benchmarks are introduced quite shortly. The dataset is not very clearly defined: it seems that there are 1.2 million training instance, does this apply for all benchmarks? The paper's title also does not seem to fit: this feels like a survey paper, which is not reflected in the title. Biologically lots of important atoms are excluded from the dataset, for instance natrium, calcium and kalium. I don't see any reason to exlude these. What does "biological activities on 11538 targets" mean?
The paper discussed molecular generation and reinforcement learning, but it is somewhat unclear how it relates to the proposed dataset since a standard training/test setting is used. Are the test molecules somehow generated in a directed or undirected fashion? Shouldn't there also be experiments on comparing ways to generate suitable molecules, and how well they match the proposed criterion? There should be benchmarks for predicting molecular properties (standard regression), and for generating molecules with certain properties. Currently it's unclear which type of problems are solved here.
Table 1 lists 5 models, while fig 3 contains 7, why the discrepancy? In table 1 the plotted runs seem to differ a lot from average results (e.g. -0.43 to 0.15, or 0.32 to 0.83). Variances should be added, and preferably more than 3 initialisations used.
Overall this is an interesting paper, but does not have any methodological contribution, and there is also few insightful results about the compared methods, nor is there meaningful analysis of the problem domain of molecules either.


Inference models, which replace an optimization-based inference procedure with a learned model, have been fundamental in advancing Bayesian deep learning, the most notable example being variational auto-encoders (VAEs). In this paper, we propose iterative inference models, which learn how to optimize a variational lower bound through repeatedly encoding gradients. Our approach generalizes VAEs under certain conditions, and by viewing VAEs in the context of iterative inference, we provide further insight into several recent empirical findings. We demonstrate the inference optimization capabilities of iterative inference models, explore unique aspects of these models, and show that they outperform standard inference models on typical benchmark data sets.

Instead of either optimization-based variational EM or an amortized inference scheme implemented via a neural network as in standard VAE models, this paper proposes a hybrid approach that essentially combines the two.  In particular, the VAE inference step, i.e., estimation of q(z|x), is conducted via application of a recent learning-to-learn paradigm (Andrychowicz et al., 2016), whereby direct gradient ascent on the ELBO criteria with respect to moments of q(z|x) is replaced with a neural network that iteratively outputs new parameter estimates using these gradients.  The resulting iterative inference framework is applied to a couple of small datasets and shown to produce both faster convergence and a better likelihood estimate.
Although probably difficult for someone to understand that is not already familiar with VAE models, I felt that this paper was nonetheless clear and well-presented, with a fair amount of useful background information and context.  From a novelty standpoint though, the paper is not especially strong given that it represents a fairly straightforward application of (Andrychowicz et al., 2016).  Indeed the paper perhaps anticipates this perspective and preemptively offers that "variational inference is a qualitatively different optimization problem" than that considered in (Andrychowicz et al., 2016), and also that non-recurrent optimization models are being used for the inference task, unlike prior work.  But to me, these are rather minor differentiating factors, since learning-to-learn is a quite general concept already, and the exact model structure is not the key novel ingredient.  That being said, the present use for variational inference nonetheless seems like a nice application, and the paper presents some useful insights such as Section 4.1 about approximating posterior gradients.
Beyond background and model development, the paper presents a few experiments comparing the proposed iterative inference scheme against both variational EM, and pure amortized inference as in the original, standard VAE.  While these results are enlightening, most of the conclusions are not entirely unexpected.  For example, given that the model is directly trained with the iterative inference criteria in place, the reconstructions from Fig. 4 seem like exactly what we would anticipate, with the last iteration producing the best result.  It would certainly seem strange if this were not the case.  And there is no demonstration of reconstruction quality relative to existing models, which could be helpful for evaluating relative performance.  Likewise for Fig. 6, where faster convergence over traditional first-order methods is demonstrated; but again, these results are entirely expected as this phenomena has already been well-documented in (Andrychowicz et al., 2016).
In terms of Fig. 5(b) and Table 1, the proposed approach does produce significantly better values of the ELBO critera; however, is this really an apples-to-apples comparison?  For example, does the standard VAE have the same number of parameters/degrees-of-freedom as the iterative inference model, or might eq. (4) involve fewer parameters than eq. (5) since there are fewer inputs?  Overall, I wonder whether iterative inference is better than standard inference with eq. (4), or whether the recurrent structure from eq. (5) just happens to implicitly create a better neural network architecture for the few examples under consideration.  In other words, if one plays around with the standard inference architecture a bit, perhaps similar results could be obtained.
Other minor comment:
* In Fig. 5(a), it seems like the performance of the standard inference model is still improving but the iterative inference model has mostly saturated.
* A downside of the iterative inference model not discussed in the paper is that it requires computing gradients of the objective even at test time, whereas the standard VAE model would not.