ESPRESSO: EFFICIENT FORWARD PROPAGATION FOR BINARY DEEP NEURAL NETWORKS
There are many applications scenarios for which the computational performance and memory footprint of the prediction phase of Deep Neural Networks (DNNs) need to be optimized. Binary Deep Neural Networks (BDNNs) have been shown to be an effective way of achieving this objective. In this paper, we show how Convolutional Neural Networks (CNNs) can be implemented using binary representations. Espresso is a compact, yet powerful library written in C/CUDA that features all the functionalities required for the forward propagation of CNNs, in a binary file less than 400KB, without any external dependencies. Although it is mainly designed to take advantage of massive GPU parallelism, Espresso also provides an equivalent CPU implementation for CNNs. Espresso provides special convolutional and dense layers for BCNNs, leveraging bit-packing and bitwise computations for efficient execution. These techniques provide a speed-up of matrix-multiplication routines, and at the same time, reduce memory usage when storing parameters and activations. We experimentally show that Espresso is significantly faster than existing implementations of optimized binary neural networks (≈ 2 orders of magnitude). Espresso is released under the Apache 2.0 license and is available at http://github.com/fpeder/espresso.

The paper presents a library written in C/CUDA that features all the functionalities required for the forward propagation of BCNNs. The library is significantly faster than existing implementations of optimized binary neural networks (≈ 2 orders of magnitude), and will be released on github.
BCNNs have been able to perform well on large-scale datasets with increased speed and decreased  energy consumption, and implementing efficient kernels for them can be very useful for mobile applications. The paper describes three implementations CPU, GPU and GPU_opt, but it is not entirely clear what the differences are and why GPU_opt is faster than GPU implementation.
Are BDNN and BCNN used to mean the same concept? If yes, could you please use only one of them?
The subsection title “Training Espresso” should be changed to “Converting a network to Espresso”, or “Training a network for Espresso”.
What is the main difference between GPU and GPU_opt implementations?
The unrolling and lifting operations are shown in Figure 2. Isn’t accelerating convolution by this method a very well known one which is implemented in many deep learning frameworks for both CPU and GPU?
What is the main contribution that makes the framework here faster than the other compared work? In Figure1, Espresso implementations are compared with other implementations in (a)dense binary matrix multiplication and (b)BMLP and not (c)BCNN. Can others ( BinaryNet
(Hubara et al., 2016) or Intel Nervana/neon (NervanaSystems)) run CNNs?
6.2 MULTI-LAYER PERCEPTRON ON MNIST – FIGURE 1B AND FIGURE 1E. It should be Figure 1d instead of 1e?
All in all, the novelty in this paper is not very clear to me. Is it bit-packing?
UPDATE:
Thank you for the revision and clarifications. I increase my rating to 6.


In this paper we propose a generalization of deep neural networks called deep function machines (DFMs). DFMs act on vector spaces of arbitrary (possibly infinite) dimension and we show that a family of DFMs are invariant to the dimension of input data; that is, the parameterization of the model does not directly hinge on the quality of the input (eg. high resolution images). Using this generalization we provide a new theory of universal approximation of bounded non-linear operators between function spaces. We then suggest that DFMs provide an expressive framework for designing new neural network layer types with topological considerations in mind. Finally, we introduce a novel architecture, RippLeNet, for resolution invariant computer vision, which empirically achieves state of the art invariance.

This paper deals with the problem of learning nonlinear operators using deep learning. Specifically, the authors propose to extend deep neural networks to the case where hidden layers can be infinite-dimensional. They give results on the quality of the approximation using these operator networks, and show how to build neural network layers that are able to take into account topological information from data. Experiments on MNIST using the proposed deep function machines (DFM) are provided.
The paper attempts to make progress in the region between deep learning and functional data analysis (FDA). This is interesting. Unfortunately, the paper requires significant improvements, both in terms of substance and in terms of presentation. My main concerns are the following:
1) One motivation of DFM is that in many applications data is a discretization of a continuous process and then can be represented by a function. FDA is the research field that formulated the ideas about the statistical data analysis of data samples consisting of continuous functions, where each function is viewed as one sample element. This paper fails to consider properly the work in its FDA context. Operator learning has been already studied in FDA. See for e.g. the problem of functional regression with functional responses. Indeed the functional model considered in the linear case is very similar to Eq. 2.5 or Eq. 3.2. Moreover, extension to nonparametric/nonlinear situations were also studied. The authors should add more information about previous work on this topic so that their results can be understood with respect to previous studies.
2) The computational aspects of DFM are not clear in the paper. From a practical computational perspective, the algorithm will be implemented on a machine which processes on finite representations of data. The paper does not clearly provide information about how the functional nature and the infinite dimensional can be handled in practice. In FDA, generally this is achieved via basis function approximations.
3) Some parts of the paper are hard to read. Sections 3 and 4 are not easy to understand. Maybe adding a section about the notation and developing more the intuition will improve the reading of the manuscript.
4) The experimental section can be significantly improved. It will be interesting to compare more DFM with its discrete counterpart. Also, other FDA approaches for operator learning should be discussed and compared to the proposed approach.


We consider tackling a single-agent RL problem by distributing it to n learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the egocentric planning overestimates values of states where the other advisors disagree, and the agnostic planning is inefficient around danger zones. We introduce a novel approach called empathic and discuss its theoretical aspects. We empirically examine and validate our theoretical findings on a fruit collection task.

Summary
The paper is well-written but does not make deep technical contributions and does not present a comprehensive evaluation or highly insightful empirical results.
Abstract / Intro
I get the entire focus of the paper is some variant of Pac-Man which has received attention in the RL literature for Atari games, but for the most part the impressive advances of previous Atari/RL papers are in the setting that the raw video is provided as input, which is much different than solving the underlying clean mathematically abstracted problem (as a grid world with obstacles) as done here and evident in the videos.  Further it is honestly hard for me to be strongly motivated about a paper that focuses on the need to decompose Pac-man into sub-agents/advisor value functions.
Section 2
Another historically well-cited paper for MDP decomposition:
  Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems, Ronald Parr. UAI 98.
  https://dslpitt.org/uai/papers/98/p422-parr.pdf
Section 3
Is the additive reward decomposition a required part of the problem specification?  It seems so, i.e., there is no obvious method for automatically decomposing a monolithic reward function over advisors.
Section 4
* Egocentric:
Definition 1: Sure, the problem will have local optima (attractors) when decomposed suboptimally -- I'm not sure what new insight we've gained from this analysis... it is a general problem with any function approximation scheme that does not guarantee that the rank ordering of actions for a state is preserved.
* Agnostic
Other than approximating some type of myopic rollout, I really don't see why this approach would be reasonable?  I am surprised it works at all though my guess is that this could simply be an artifact of evaluating on a single domain with a specific structure.
* Empathic
This appears to be the key contribution though related work certainly infringes on its novelty.  Is this paper then an empirical evaluation of previous methods in a single Pac-man grid world variant?
I wonder if the theory of DEC-MDPs would have any relevance for novel analysis here?
Section 5
I'm disappointed that the authors only evaluate on a single domain; presumably the empathic approach has applications beyond Pac-Man?
The fact that empathic generally performs better is not at all surprising.  The fact that a modified discount factor for egocentric can also perform well is not surprising given that lower discount factors have often been shown to improve approximated MDP solutions, e.g.,
  Biasing Approximate Dynamic Programming with a Lower Discount Factor
  Marek Petrik, Bruno Scherrer (NIPS-08).
  http://marek.petrik.us/pub/Petrik2009a.pdf
***
Side note:
The following part is somewhat orthogonal to the review above in that I would not expect the authors to address this on revision, *but* at the same time I think it provides a connection to the special case of concurrent action decomposition into advisors, which could potentially provide a high impact direction of application for this work (i.e., concurrent problems are hard and show up in numerous operations research problems covering inventory control, logistics, epidemic response).
For the special case that each advisor is assigned to one action in a factored space of concurrent actions, the egocentric algorithm would be very close to the Hindsight approximation in Section 6 of this paper (including an additive decomposition of rewards):
  Planning in Factored Action Spaces with Symbolic Dynamic Programming
  Aswin Nadamuni Raghavan, Alan Fern, Prasad Tadepalli, Roni Khardon, and Saket Joshi (AAAI-12).
  https://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/5012/5336
This simple algorithm is hard to beat for the following reason that connects some details of your egocentric and empathic settings: rather than decomposing a concurrent MDP into independent problems per concurrent action, the optimization of each action (by each advisor) is done in sequence (advisors are ordered) and gets to condition on the previously selected advisor actions.  So it provides an alternate paradigm where advisors actually get to see and condition their policy on what other advisors are doing.  In my own work comparing optimal concurrent solutions to this approach, I have found this approach to be near-optimal and much more efficient to solve since it exploits decomposition.
Why is this relevant to this work?  Because (a) it suggests another variant of the advisor decomposition that at least makes sense in the case of concurrent actions (and perhaps shared actions though this would require some extension) and (b) it suggests there are more options than just the full egocentric and empathic settings in this important class of concurrent action problems that are necessarily solved in practice for large action spaces by some form of decomposition.  This could be an interesting direction for future exploration of the ideas in this work, where there might be additional technical novelty and more space for empirical contributions and observations.


Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.

Summary:
This paper presents a derivation which links a DNN to recursive application of
maximum entropy model fitting. The mathematical notation is unclear, and in
one cases the lemmas are circular (i.e. two lemmas each assume the other is
correct for their proof). Additionally the main theorem requires complete
independence, but the second theorem provides pairwise independence, and the
two are not the same.
Major comments:
- The second condition of the maximum entropy equivalence theorem requires
  that all T are conditionally independent of Y. This statement is unclear, as
it could mean pairwise independence, or it could mean jointly independent
(i.e. for all pairs of non-overlapping subsets A & B of T I(T_A;T_B|Y) = 0).
This is the same as saying the mapping X->T is making each dimension of T
orthogonal, as otherwise it would introduce correlations. The proof of the
theorem assumes that pairwise independence induces joint independence and this
is not correct.
- Section 4.1 makes an analogy to EM, but gradient descent is not like this
  process as all the parameters are updated at once, and only optimised by a
single (noisy) step. The optimisation with respect to a single layer is
conditional on all the other layers remaining fixed, but the gradient
information is stale (as it knows about the previous step of the parameters in
the layer above). This means that gradient descent does all 1..L steps in
parallel, and this is different to the definition given.
- The proofs in Appendix C which are used for the statement I(T_i;T_j) >=
  I(T_i;T_j|Y) are incomplete, and in generate this statement is not true, so
requires proof.
- Lemma 1 appears to assume Lemma 2, and Lemma 2 appears to assume Lemma 1.
  Either these lemmas are circular or the derivations of both of them are
unclear.
- In Lemma 3 what is the minimum taken over for the left hand side? Elsewhere
  the minimum is taken over T, but T does not appear on the left hand side.
Explicit minimums help the reader to follow the logic, and implicit ones
should only be used when it is obvious what the minimum is over.
- In Lemma 5, what does "T is only related to X" mean? The proof states that
  Y -> T -> X forms a Markov chain, but this implies that T is a function of
Y, not X.
Minor comments:
- I assume that the E_{P(X,Y)} notation is the expectation of that probability
  distribution, but this notation is uncommon, and should be replaced with a
more explicit one.
- Markov is usually romanized with a "k" not a "c".
- The paper is missing numerous prepositions and articles, and contains
  multiple spelling mistakes & typos.


LARGE-SCALE OPTIMAL TRANSPORT AND MAPPING ESTIMATION
This paper presents a novel two-step approach for the fundamental problem of learning an optimal map from one distribution to another. First, we learn an optimal transport (OT) plan, which can be thought as a one-to-many map between the two distributions. To that end, we propose a stochastic dual approach of regularized OT, and show empirically that it scales better than a recent related approach when the amount of samples is very large. Second, we estimate a Monge map as a deep neural network learned by approximating the barycentric projection of the previously-obtained OT plan. This parameterization allows generalization of the mapping outside the support of the input measure. We prove two theoretical stability results of regularized OT which show that our estimations converge to the OT plan and Monge map between the underlying continuous measures. We showcase our proposed approach on two applications: domain adaptation and generative modeling.

This paper proposes a new method for estimating optimal transport plans and maps among continuous distributions, or discrete distributions with large support size. First, the paper proposes a dual algorithm to estimate Kantorovich plans, i.e. a coupling between two input distributions minimizing a given cost function, using dual functions parameterized as neural networks. Then an algorithm is given to convert a generic plan into a Monge map, a deterministic function from one domain to the other, following the barycenter of the plan. The algorithms are shown to be consistent, and demonstrated to be more efficient than an existing semi-dual algorithm. Initial applications to domain adaptation and generative modeling are also shown.
These algorithms seem to be an improvement over the current state of the art for this problem setting, although more of a discussion of the relationship to the technique of Genevay et al. would be useful: how does your approach compare to the full-dual, continuous case of that paper if you simply replace their ball of RKHS functions with your class of deep networks?
The consistency properties are nice, though they don't provide much insight into the rate at which epsilon should be decreased with n or similar properties. The proofs are clear, and seem correct on a superficial readthrough; I have not carefully verified them.
The proofs are mainly limited in that they don't refer in any way to the class of approximating networks or the optimization algorithm, but rather only to the optimal solution. Although of course proving things about the actual outcomes of optimizing a deep network is extremely difficult, it would be helpful to have some kind of understanding of how the class of networks in use affects the solutions. In this way, your guarantees don't say much more than those of Arjovsky et al., who must assume that their "critic function" reaches the global optimum: essentially you add a regularization term, and show that as the regularization decreases it still works, but under seemingly the same kind of assumptions as Arjovsky et al.'s approach which does not add an explicit regularization term at all. Though it makes sense that your regularization might lead to a better estimator, you don't seem to have shown so either in theory or empirically.
The performance comparison to the algorithm of Genevay et al. is somewhat limited: it is only on one particular problem, with three different hyperparameter settings. Also, since Genevay et al. propose using SAG for their algorithm, it seems strange to use plain SGD; how would the results compare if you used SAG (or SAGA/etc) for both algorithms?
In discussing the domain adaptation results, you mention that the L2 regularization "works very well in practice," but don't highlight that although it slightly outperforms entropy regularization in two of the problems, it does substantially worse in the other. Do you have any guesses as to why this might be?
For generative modeling: you do have guarantees that, *if* your optimization and function parameterization can reach the global optimum, you will obtain the best map relative to the cost function. But it seems that the extent of these guarantees are comparable to those of several other generative models, including WGANs, the Sinkhorn-based models of Genevay et al. (2017, https://arxiv.org/abs/1706.00292/), or e.g. with a different loss function the MMD-based models of Li, Swersky, and Zemel (ICML 2015) / Dziugaite, Roy, and Ghahramani (UAI 2015). The different setting than the fundamental GAN-like setup of those models is intriguing, but specifying a cost function between the source and the target domains feels exceedingly unnatural compared to specifying a cost function just within one domain as in these other models.
Minor:
In (5), what is the purpose of the -1 term in R_e? It seems to just subtract a constant 1 from the regularization term.


In this paper, we show a phenomenon, which we named "super-convergence", where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate. Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. The architectures to replicate this work will be made available upon publication.

In this paper, the authors analyze training of residual networks using large cyclic learning rates (CLR). The authors demonstrate (a) fast convergence with cyclic learning rates and (b) evidence of large learning rates acting as regularization which improves performance on test sets – this is called “super-convergence”. However, both these effects are only shown on a specific dataset, architecture, learning algorithm and hyper parameter setting.
Some specific comments by sections:
2. Related Work: This section loosely mentions other related works on SGD, topology of loss function and adaptive learning rates. The authors mention Loshchilov & Hutter in next section but do not compare it to their work. The authors do not discuss a somewhat contradictory claim from NIPS 2017 (as pointed out in the public comment): http://papers.nips.cc/paper/6770-train-longer-generalize-better-closing-the-generalization-gap-in-large-batch-training-of-neural-networks.pdf
3. Super-convergence: This is a well explained section where the authors describe the LR range test and how it can be used to understand potential for super-convergence for any architecture. The authors also provide sufficient intuition for super-convergence. Since CLRs were already proposed by Smith (2015), the originality of this work would be specifically tied to their application to residual units. It would be interesting to see a qualitative analysis on how the residual error is impacting super-convergence.
4. Regularization: While Fig 4 demonstrates the regularization property, the reference to Fig 1a with better test error compared to typical training methods could simply be a result of slower convergence of typical training methods.
5. Optimal LRs: Fig.5b shows results for 1000 iterations whereas the text says 10000 (seems like a typo in scaling the plot). Figs 1 and 5 illustrate only one cycle (one increase and one decrease) of CLR. It would be interesting to see cases where more than one cycle is required and to see what happens when the LR increases the second time.
6. Experiments: This is a strong section where the authors show extensive reproducible experimentation to identify settings under which super-convergence works or does not work. However, the fact that the results only applies to CIFAR-10 dataset and could not be observed for ImageNet or other architectures is disappointing and heavily takes away from the significance of this work.
Overall, the work is presented as a positive result in very specific conditions but it seems more like a negative result. It would be more appealing if the paper is presented as a negative result and strengthened by additional experimentation and theoretical backing.


Modern deep artificial neural networks have achieved impressive results through models with very large capacity-compared to the number of training examplesthat control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of results, they seem to waste capacity. In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone-without any other explicit regularization techniques-can achieve the same performance or higher as regularized models, especially when training with fewer examples.

The paper proposes data augmentation as an alternative to commonly used regularisation techniques like weight decay and dropout, and shows for a few reference models / tasks that the same generalization performance can be achieved using only data augmentation.
I think it's a great idea to investigate the effects of data augmentation more thoroughly. While it is a technique that is often used in literature, there hasn't really been any work that provides rigorous comparisons with alternative approaches and insights into its inner workings. Unfortunately I feel that this paper falls short of achieving this.
Experiments are conducted on two fairly similar tasks (image classification on CIFAR-10 and CIFAR-100), with two different network architectures. This is a bit meager to be able to draw general conclusions about the properties of data augmentation. Given that this work tries to provide insight into an existing common practice, I think it is fair to expect a much stronger experimental section. In section 2.1.1 it is stated that this was a conscious choice because simplicity would lead to clearer conclusions, but I think the conclusions would be much more valuable if variety was the objective instead of simplicity, and if larger-scale tasks were also considered.
Another concern is that the narrative of the paper pits augmentation against all other regularisation techniques, whereas more typically these will be used in conjunction. It is however very interesting that some of the results show that augmentation alone can sometimes be enough.
I think extending the analysis to larger datasets such as ImageNet, as is suggested at the end of section 3, and probably also to different problems than image classification, is going to be essential to ensure that the conclusions drawn hold weight.
Comments:
- The distinction between "explicit" and "implicit" regularisation is never clearly enunciated. A bunch of examples are given for both, but I found it tricky to understand the difference from those. Initially I thought it reflected the intention behind the use of a given technique; i.e. weight decay is explicit because clearly regularisation is its primary purpose -- whereas batch normalisation is implicit because its regularisation properties are actually a side effect. However, the paper then goes on to treat data augmentation as distinct from other explicit regularisation techniques, so I guess this is not the intended meaning. Please clarify this, as the terms crop up quite often throughout the paper. I suspect that the distinction is somewhat arbitrary and not that meaningful.
- In the abstract, it is already implied that data augmentation is superior to certain other regularisation techniques because it doesn't actually reduce the capacity of the model. But this ignores the fact that some of the model's excess capacity will be used to model out-of-distribution data (w.r.t. the original training distribution) instead. Data augmentation always modifies the distribution of the training data. I don't think it makes sense to imply that this is always preferable over reducing model capacity explicitly. This claim is referred to a few times throughout the work.
- It could be more clearly stated that the reason for the regularising effect of batch normalisation is the noise in the batch estimates for mean and variance.
- Some parts of the introduction could be removed because they are obvious, at least to an ICLR audience (like "the model would not be regularised if alpha (the regularisation parameter) equals 0").
- The experiments with smaller dataset sizes would be more interesting if smaller percentages were used. 50% / 80% / 100% are all on the same order of magnitude and this setting is not very realistic. In practice, when a dataset is "too small" to be able to train a network that solves a problem reliably, it will generally be one or more orders of magnitude too small, not 2x too small.
- The choices of hyperparameters for "light" and "heavy" motivation seem somewhat arbitrary and are not well motivated. Some parameters which are sampled uniformly at random should be probably be sampled log-uniformly instead, because they represent scale factors. It should also be noted that much more extreme augmentation strategies have been used for this particular task in literature, in combination with padding (for example by Graham). It would be interesting to include this setting in the experiments as well.
- On page 7 it is stated that "when combined with explicit regularization, the results are much worse than without it", but these results are omitted from the table. This is unfortunate because it is a very interesting observation, that runs counter to the common practice of combining all these regularisation techniques together (e.g. L2 + dropout + data augmentation is a common combination). Delving deeper into this could make the paper a lot stronger.
- It is not entirely true that augmentation parameters depend only on the training data and not the architecture (last paragraph of section 2.4). Clearly more elaborate architectures benefit more from data augmentation, and might need heavier augmentation to perform optimally because they are more prone to overfitting (this is in fact stated earlier on in the paper as well). It is of course true that these hyperparameters tend to be much more robust to architecture changes than those of other regularisation techniques such as dropout and weight decay. This increased robustness is definitely useful and I think this is also adequately demonstrated in the experiments.
- Phrases like "implicit regularization operates more effectively at capturing reality" are too vague to be meaningful.
- Note that weight decay has also been found to have side effects related to optimization (e.g. in "Imagenet classification with deep convolutional neural networks", Krizhevsky et al.)
REVISION: I applaud the effort the authors have put in to address many of my and the other reviewers' comments. I think they have done so adequately for the most part, so I've decided to raise the rating from 3 to 5, for what it's worth.
The reason I have decided not to raise it beyond that, is that I still feel that for a paper like this, which studies an existing technique in detail, the experimental side needs to be significantly stronger. While ImageNet experiments may be a lot of work, some other (smaller) additional datasets would also have provided more interesting evidence. CIFAR-10 and CIFAR-100 are so similar that they may as well be considered variants of the same dataset, at least in the setting where they are used here.
I do really appreciate the variety in the experiments in terms of network architectures, regularisation techniques, etc. but I think for real-world relevance, variety in problem settings (i.e. datasets) is simply much more important. I think it would be fine if additional experiments on other datasets were not varied along all these other axes, to cut down on the amount of work this would involve. But not including them at all unfortunately makes the results much less impactful.


In order to develop a scalable multi-task reinforcement learning (RL) agent that is able to execute many complex tasks, this paper introduces a new RL problem where the agent is required to execute a given task graph which describes a set of subtasks and dependencies among them. Unlike existing approaches which explicitly describe what the agent should do, our problem only describes properties of subtasks and relationships between them, which requires the agent to perform a complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural task graph solver (NTS) which encodes the task graph using a recursive neural network. To overcome the difficulty of training, we propose a novel non-parametric gradient-based policy that performs back-propagation over a differentiable form of the task graph to compute the influence of each subtask on the other subtasks. Our NTS is pre-trained to approximate the proposed gradientbased policy and fine-tuned through actor-critic method. The experimental results on a 2D visual domain show that our method to pre-train from the gradient-based policy significantly improves the performance of NTS. We also demonstrate that our agent can perform a complex reasoning to find the optimal way of executing the task graph and generalize well to unseen task graphs. In addition, we compare our agent with a Monte-Carlo Tree Search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of our agent can be further improved by combining with MCTS. The demo video is available at https://youtu.be/e_ZXVS5VutM.

This paper proposes to train recursive neural network on subtask graphs in order to execute a series of tasks in the right order, as is described by the subtask graph's dependencies. Each subtask execution is represented by a (non-learned) option. Reward shaping allows the proposed model to outperform simpler baselines, and experiments show the model generalizes to unseen graphs.
While this paper is as far as I can tell novel in how it does what it does, the authors have failed to convey to me why this direction of research is relevant.
- We know finding options is the hard part about options
- We already have good algorithms that take subtask graphs and execute them in the right order from the planning litterature
An interesting avenue would be if the subtask graphs were instead containing some level of uncertainty, or representing stochasticity, or anything that more traditional methods are unable to deal with efficiently, then I would see a justification for the use of neural networks. Alternatively, if the subtask graphs were learned instead of given, that would open the door to scaling an general learning. Yet, this is not discussed in the paper.
Another interesting avenue would be to learn the options associated with each task, possibly using the information from the recursive neural networks to help learn these options.
The proposed algorithm relies on fairly involved reward shaping, in that it is a very strong signal of supervision on what the next action should be. Additionaly, it's not clear why learning seems to completely "fail" without the pre-trained policy. The justification given is that it is "to address the difficulty of training due to the complex nature of the problem" but this is not really satisfying as the problems are not that hard. This also makes me question the generality of the approach since the pre-trained policy is rather simple while still providing an apparently strong score.
In your experiments, you do not compare with any state-of-the-art RL or hierarchical RL algorithm on your domain, and use a new domain which has no previous point of reference. It it thus hard to properly evaluate your method against other proposed methods.
What the authors propose is a simple idea, everything is very clearly explained, the experiments are somewhat lacking but at least show an improvement over more a naive approach, however, due to its simplicity, I do not think that this paper is relevant for the ICLR conference.
Comments:
- It is weird to use both a discount factor \gamma *and* a per-step penalty. While not disallowed by theory, doing both is redundant because they enforce the same mechanism.
- It seems weird that the smoothed logical AND/OR functions do not depend on the number of inputs; that is unless there are always 3 inputs (but it is not explained why; logical functions are usually formalised as functions of 2 inputs) as suggested by Fig 3.
- It does not seem clear how the whole training is actually performed (beyond the pre-training policy). The part about the actor-critic learning seems to lack many elements (whole architecture training? why is the policy a sum of "p^{cost}" and "p^{reward}"? is there a replay memory? How are the samples gathered?). (On the positive side, the appendix provides some interesting details on the tasks generations to understand the experiments.)
- The experiments cover different settings with different task difficulties. However, only one type of tasks is used. It would be good to motivate (in addition to the paragraph in the intro) the cases where using the algorithm described in the paper may be (or not?) the only viable option and/or compare it to other algorithms. Even tough not mandatory, it would also be a clear good addition to also demonstrate more convincing experiments in a different setting.
- "The episode length (time budget) was randomly set for each episode in a range such that 60% − 80% of subtasks are executed on average for both training and testing." --> this does not seem very precise: under what policy is the 60-80% defined? Is the time budget different for each new generated environment?
- why wait until exactly 120 epochs for NTS-RProp before fine-tuning with actor-critic? It seems that much less would be sufficient from figure 4?
- In the table 1 caption, it is written "same graph structure with training set" --> do you mean "same graph structure than the training set"?


Published as a conference paper at ICLR 2018 SYNTHETIC AND NATURAL NOISE BOTH BREAK NEURAL MACHINE TRANSLATION
Character-based neural machine translation (NMT) models alleviate out-ofvocabulary issues, learn morphology, and move us closer to completely end-toend translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

This paper investigates the impact of character-level noise on various flavours of neural machine translation. It tests 4 different NMT systems with varying degrees and types of character awareness, including a novel meanChar system that uses averaged unigram character embeddings as word representations on the source side. The authors test these systems under a variety of noise conditions, including synthetic scrambling and keyboard replacements, as well as natural (human-made) errors found in other corpora and transplanted to the training and/or testing bitext via replacement tables. They show that all NMT systems, whether BPE or character-based, degrade drastically in quality in the presence of both synthetic and natural noise, and that it is possible to train a system to be resistant to these types of noise by including them in the training data. Unfortunately, they are not able to show any types of synthetic noise helping address natural noise. However, they are able to show that a system trained on a mixture of error types is able to perform adequately on all types of noise.
This is a thorough exploration of a mostly under-studied problem. The paper is well-written and easy to follow. The authors do a good job of positioning their study with respect to related work on black-box adversarial techniques, but overall, by working on the topic of noisy input data at all, they are guaranteed novelty. The inclusion of so many character-based systems is very nice, but it is the inclusion of natural sources of noise that really makes the paper work. Their transplanting of errors from other corpora is a good solution to the problem, and one likely to be built upon by others. In terms of negatives, it feels like this work is just starting to scratch the surface of noise in NMT. The proposed meanChar architecture doesn’t look like a particularly good approach to producing noise-resistant translation systems, and the alternative solution of training on data where noise has been introduced through replacement tables isn’t extremely satisfying. Furthermore, the use of these replacement tables means that even when the noise is natural, it’s still kind of artificial. Finally, this paper doesn’t seem to be a perfect fit for ICLR, as it is mostly experimental with few technical contributions that are likely to be impactful; it feels like it might be more at home and have greater impact in a *ACL conference.
Regarding the artificialness of their natural noise - obviously the only solution here is to find genuinely noisy parallel data, but even granting that such a resource does not yet exist, what is described here feels unnaturally artificial. First of all, errors learned from the noisy data sources are constrained to exist within a word. This tilts the comparison in favour of architectures that retain word boundaries (such as the charCNN system here), while those systems may struggle with other sources of errors such as missing spaces between words. Second, if I understand correctly, once an error is learned from the noisy data, it is applied uniformly and consistently throughout the training and/or test data. This seems worse than estimating the frequency of the error and applying them stochastically (or trying to learn when an error is likely to occur). I feel like these issues should at least be mentioned in the paper, so it is clear to the reader that there is work left to be done in evaluating the system on truly natural noise.
Also, it is somewhat jarring that only the charCNN approach is included in the experiments with noisy training data (Table 6). I realize that this is likely due to computational or time constraints, but it is worth providing some explanation in the text for why the experiments were conducted in this manner. On a related note, the line in the abstract stating that “... a character convolutional neural network  is able to simultaneously learn representations robust to multiple kinds of noise” implies that the other (non-charCNN) architectures could not learn these representations, when in reality, they simply weren’t given the chance.
Section 7.2 on the richness of natural noise is extremely interesting, but maybe less so to an ICLR audience. From my perspective, it would be interesting to see that section expanded, or used as the basis for future work on improve architectures or training strategies.
I have only one small, specific suggestion: at the end of Section 3, consider deleting the last paragraph break, so there is one paragraph for each system (charCNN currently has two paragraphs).
[edited for typos]


Published as a conference paper at ICLR 2018 RESIDUAL LOSS PREDICTION: REINFORCEMENT LEARNING WITH NO INCREMENTAL FEEDBACK
We consider reinforcement learning and bandit structured prediction problems with very sparse loss feedback: only at the end of an episode. We introduce a novel algorithm, RESIDUAL LOSS PREDICTION (RESLOPE), that solves such problems by automatically learning an internal representation of a denser reward function. RESLOPE operates as a reduction to contextual bandits, using its learned loss representation to solve the credit assignment problem, and a contextual bandit oracle to trade-off exploration and exploitation. RESLOPE enjoys a no-regret reductionstyle theoretical guarantee and outperforms state of the art reinforcement learning algorithms in both MDP environments and bandit structured prediction settings.

After reading the other reviews and the authors' responses, I am satisfied that this paper is above the accept threshold.  I think there are many areas of further discussion that the authors can flesh out (as mentioned below and in other reviews), but overall the contribution seems solid.   I also appreciate the reviewers' efforts to run more experiments and flesh out the discussion in the revised version of the submission.
Final concluding thoughts:
-- Perhaps pi^ref was somehow better for the structured prediction problems than RL problems?
-- Can one show regret bound for multi-deviation if one doesn't have to learn x (i.e., we were given a good x a priori)?
---------------------------------------------
ORIGINAL REVIEW
First off, I think this paper is potentially above the accept threshold.  The ideas presented are interesting and the results are potentially interesting as well.   However, I have some reservations, a significant portion of which stem from not understanding aspects of the proposed approach and theoretical results, as outlined below.
The algorithm design and theoretical results in the appendix could be made substantially more rigorous. Specifically:
--  basic notations such as regret (in Theorem 1), the total reward (J), Q-value (Q), and value function (V) are not defined.  While these concepts are fairly standard, it would be highly beneficial to define them formally.
-- I'm not convinced that the "terms in the parentheses" (Eq. 7) are "exactly the contextual bandit cost".  I would like to see a more rigorous derivation of the connection.  For instance, one could imagine that the policy disadvantage should be the difference between the residual costs of the bandit algorithm and the reference policy, rather than just the residual cost of the bandit algorithm.
-- I'm a little unclear in the proof of Theorem 1 where Q(s,pi_n) from Eq 7 fits into Eq 8.
-- The residual cost used for CB.update depends on estimated costs at other time steps h!=h_dev.  Presumably, these estimated costs will change as learning progresses.  How does one reconcile that?  I imagine that it could somehow work out using a bandit algorithm with adversarial guarantees, but I can also imagine it not working out.  I would like to see a rigorous treatment of this issue.
-- It would be nice to see an end-to-end result that instantiates Theorem 1 (and/or Theorem 2) with a contextual bandit algorithm to see a fully instantiated guarantee.
With regards to the algorithm design itself, I have some confusions:
-- How does one create x in practice? I believe this is described in Appendix H, but it's not obvious.
-- What happens if we don't have a good way to generate x and it must be learned as well?  I'd imagine one would need larger RNNs in that case.
-- If x is actually learned on-the-fly, how does that impact the theoretical results?
-- I find it curious that there's no notion of future reward learning in the learning algorithm.  For instance, in Q learning, one directly models the the long-term (discounted) rewards during learning.  In fact, the theoretical analysis talks about advantage functions as well.  It would be nice to comment on this aspect at an intuitive level.
With regards to the experiments:
-- I find it very curious that the results are so negative for using only 1-dev compared to multi-dev (Figure 9 in Appendix).  Given that much of the paper is devoted to 1-dev, it's a bit disappointing that this issue is not analyzed in more detail, and furthermore the results are mostly hidden in the appendix.
-- It's not clear if a reference policy was used in the experiments and what value of beta was used.
-- Can the authors speculate about the difference in performance between the RL and bandit structured prediction settings?  My personal conjecture is that the bandit structured prediction settings are more easily decomposable additively, which leads to a greater advantage of the proposed approach, but I would like to hear the authors' thoughts.
Finally, the overall presentation of this paper could be substantially improved.  In addition to the above uncertainties, some more points are described below.  I don't view these points as "deal breakers" for determining accept/reject.
-- This paper uses too many examples, from part-of-speech tagging to credit assignment in determining paths.  I recommend sticking to one running example, which substantially reduces context switching for the reader.  In every such example, there are extraneous details are not relevant to making the point, and the reader needs to spend considerable effort figuring that out for each example used.
-- Inconsistent language. For instance, x is sometimes referred to as the "input example", "context" and "features".
-- At the end of page 4, "Internally, ReslopePolicy takes a standard learning to search step."  Two issues: 1) ReslopePolicy is not defined or referred to anywhere else.  2) is the remainder of that paragraph a description of a "standard learning to search step"?
-- As mentioned before, Regret is not defined in Theorem 1 & 2.
-- The discussion of the high-level algorithmic concepts is a bit diffuse or lacking.  For instance, one key idea in the algorithmic development is that it's sufficient to make a uniformly random deviation.  Is this idea from the learning to search literature?  If so, it would be nice to highlight this in Section 2.2.


Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and wellchosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers.

* In the "flat vs sharp" dilemma, the experiments display that the dilemma, if any, is subtle. Table 1 does not necessarily contradict this view. It would be a good idea to put the test results directly on Fig. 4 as it does not ease reading currently (and postpone ResNet-56 in the appendix).
How was Figure 5 computed ? It is said that *a* random direction was used from each minimiser to plot the loss, so how the 2D directions obtained ?
* On the convexity vs non-convexity (Sec. 6), it is interesting to see how pushing the Id through the net changes the look of the loss for deep nets. The difference VGG - ResNets is also interesting, but it would have been interesting to see how this affects the current state of the art in understanding deep learning, something that was done for the "flat vs sharp" dilemma, but is lacking here. For example, does this observation that the local curvature of the loss around minima is different for ResNets and VGG allows to interpret the difference in their performances ?
* On optimisation paths, the choice of PCA directions is wise compared to random projections, and results are nice as plotted. There is however a phenomenon I would have liked to be discussed, the fact that the leading eigenvector captures so much variability, which perhaps signals that optimisation happens in a very low dimensional subspace for the experiments carried, and could be useful for optimisation algorithms (you trade dimension d for a much smaller "effective" d', you only have to figure out a generating system for this subspace and carry out optimisation inside). Can this be related to the "flat vs sharp" dilemma ? I would suppose that flatness tends to increase the variability captured by leading eigenvectors ?
Typoes:
Legend of Figure 2: red lines are error -> red lines are accuracy
Table 1: test accuracy -> test error
Before 6.2: architecture effects -> architecture affects


Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world's most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) farmework for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method.

This paper presents a new reinforcement learning approach to handle environments with a mix of discrete and
continuous action spaces. The authors propose a parameterized deep Q-network (P-DQN) and leverage learning
schemes from existing algorithms such as DQN and DDPG to train the network. The proposed loss function and
alternating optimization of the parameters are pretty intuitive and easy to follow. My main concern is
with lack of sufficient depth in empirical evaluation and analysis of the method.
Pros:
1. The setup is an interesting and practically useful one to investigate. Many real-world environments require individual actions
 that are further parameterized over a continuous space.
2. The proposed method is simple and intuitive.
Cons:
1. The evaluation is performed only on a single environment in a restricted fashion. I understand the authors are restricted in the choice of environments which require a hybrid action space. However,
 even domains like Atari could be used in a setting where the continuous parameter x_k refers to the number of
 repetitions for action k. This is similar to the work of Lakshminarayanan et al. (2017). Could you test your algorithm in such a setting?
2. Comparison of the algorithm is performed only against DDPG. Have you tried other options like PPO (Schulman et al., 2017)?
 Also, considering that the action space is simplified in the experimental setup ("we use the default parameters of skills provided by the game environment, usually pointing to
the opponent hero's location"), with only the move(\alpha) action being a hybrid, one could imagine discretizing the move
direction \alpha and training a DQN (or any other algorithms over discrete action spaces) as another baseline.
3. The reward structure seems to be highly engineered. With so many components in the reward, it is not clear
what the individual contributions are and what policies are actually learned.
4. The authors don't provide any analysis of the empirical results. Do the P-DQN and DDPG converge to the same policy?
What factor(s) contribute most to the faster learning of P-DQN? Do the values of \alpha and \beta for the two-timescale
updates affect the results considerably?
5. (minor) The writing contains a lot of grammatical errors which makes this draft below par for an ICLR paper.
Other Questions:
1. In eq. 5.3, the loss over \theta is defined as the sum of Q values over different k. Did you try other formulations of
the loss? (say, product of the Q values for instance) One potential issue with the sum could be that if some values of k dominate this sum, Q(s, k, x_k; w) might not be maximized for all k.
2. Some terms of the reward function seem to be overly dependent on historic actions (ex. difference in gold and hitpoints). This could swamp the
influence of the other terms which are more dependent on the current action a_t, which might be an issue, especially with the Markovian assumption?
References:
 Lakshminarayanan et al, 2017; Dynamic Action Repetition for Deep Reinforcement Learning; AAAI
 Schulman et al., 2017; Proximal Policy Optimization Algorithms; Arxiv


Published as a conference paper at ICLR 2018 MEMORIZATION PRECEDES GENERATION: LEARNING UNSUPERVISED GANS WITH MEMORY NETWORKS
We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.

[Overview]
In this paper, the authors proposed a novel model called MemoryGAN, which integrates memory network with GAN. As claimed by the authors, MemoryGAN is aimed at addressing two problems of GAN training: 1) difficult to model the structural discontinuity between disparate classes in the latent space; 2) catastrophic forgetting problem during the training of discriminator about the past synthesized samples by the generator. It exploits the life-long memory network and adapts it to GAN. It consists of two parts, discriminative memory network (DMN) and Memory Conditional Generative Network (MCGN). DMN is used for discriminating input samples by integrating the memory learnt in the memory network, and MCGN is used for generating images based on random vector and the sampled memory from the memory network. In the experiments, the authors evaluated memoryGAN on three datasets, CIFAR-10, affine-MNIST and Fashion-MNIST, and demonstrated the superiority to previous models. Through ablation study, the authors further showed the effects of separate components in memoryGAN.
[Strengths]
1. This paper is well-written. All modules in the proposed model and the experiments were explained clearly. I enjoyed much to read the paper.
2. The paper presents a novel method called MemoryGAN for GAN training. To address the two infamous problems mentioned in the paper, the authors proposed to integrate a memory network into GAN. Through memory network, MemoryGAN can explicitly learn the data distribution of real images and fake images. I think this is a very promising and meaningful extension to the original GAN.
3. With MemoryGAN, the authors achieved best Inception Score on CIFAR-10. By ablation study, the authors demonstrated each part of the model helps to improve the final performance.
[Comments]
My comments are mainly about the experiment part:
1. In Table 2, the authors show the Inception Score of images generated by DCGAN at the last row. On CIFAR-10, it is ~5.35. As the authors mentioned, removing EM, MCGCN and Memory will result in a conventional DCGAN. However, as far as I know, DCGAN could achieve > 6.5 Inception Score in general.  I am wondering what makes such a big difference between the reported numbers in this paper and other papers?
2. In the experiments, the authors set N = 16,384, and M = 512, and z is with dimension 16. I did not understand why the memory size is such large. Take CIFAR-10 as the example, its training set contains 50k images. Using such a large memory size, each memory slot will merely count for several samples. Is a large memory size necessary to make MemoryGAN work? If not, the authors should also show ablated study on the effect of different memory size; If it is true, please explain why is that. Also, the authors should mention the training time compared with DCGAN. Updating memory with such a large size seems very time-consuming.
3. Still on the memory size in this model. I am curious about the results if the size is decreased to the same or comparable number of image categories in the training set. As the author claimed, if the memory network could learn to cluster training data into different category, we should be able to see some interesting results by sampling the keys and generate categoric images.
4. The paper should be compared with InfoGAN (Chen et al. 2016), and the authors should explain the differences between two models in the related work. Similar to MemoryGAN, InfoGAN also did not need any data annotations, but could learn the latent code flexibly.
[Summary]
This paper proposed a new model called MemoryGAN for image generation. It combined memory network with GAN, and achieved state-of-art performance on CIFAR-10. The arguments that MemoryGAN could solve the two infamous problem make sense. As I mentioned above, I did not understand why the authors used such large memory size. More explanations and experiments  should be conducted to justify this setting. Overall, I think MemoryGAN opened a new direction of GAN and worth to further explore.


Published as a conference paper at ICLR 2018 MANY PATHS TO EQUILIBRIUM: GANS DO NOT NEED TO DECREASE A DIVERGENCE AT EVERY STEP
Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion. Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize their own cost. GANs are designed to reach a Nash equilibrium at which each player cannot reduce their cost without changing the other players' parameters. One useful approach for the theory of GANs is to show that a divergence between the training distribution and the model distribution obtains its minimum value at equilibrium. Several recent research directions have been motivated by the idea that this divergence is the primary guide for the learning process and that every step of learning should decrease the divergence. We show that this view is overly restrictive. During GAN training, the discriminator provides learning signal in situations where the gradients of the divergences between distributions would not be useful. We provide empirical counterexamples to the view of GAN training as divergence minimization. Specifically, we demonstrate that GANs are able to learn distributions in situations where the divergence minimization point of view predicts they would fail. We also show that gradient penalties motivated from the divergence minimization perspective are equally helpful when applied in other contexts in which the divergence minimization perspective does not predict they would be helpful. This contributes to a growing body of evidence that GAN training may be more usefully viewed as approaching Nash equilibria via trajectories that do not necessarily minimize a specific divergence at each step.

This paper answers recent critiques about ``standard GAN'' that were recently formulated to motivate variants based on other losses, in particular using ideas from optimal transport.  It makes main points
1) ``standard GAN'' is an ill-defined term that may refer to two different learning criteria, with different properties
2) though the non-saturating variant (see Eq. 3) of ``standard GAN'' may converge towards a minimum of the Jensen-Shannon divergence, it does not mean that the minimization process follows gradients of the Jensen-Shannon divergence (and conversely, following gradient paths of the Jensen-Shannon divergence may not converge towards a minimum, but this was rather the point of the previous critiques about ``standard GAN'').
3) the penalization strategies introduced for ``non-standard GAN'' with specific motivations, may also apply successfully to the ``standard GAN'', improving robustness, thereby helping to set hyperparameters.
Note that item 2) is relevant in many other setups in the deep learning framework and is often overlooked.
Overall, I believe that the paper provides enough material to substantiate these claims, even if the message could be better delivered. In particular, the writing is sometimes ambiguous (e.g. in Section 2.3, the reader who did not follow the recent developments on the subject on arXiv will have difficulties to rebuild the cross-references between authors, acronyms and formulae). The answers to the critiques referenced in the
 paper are convincing, though I must admit that I don't know how crucial it is to answer these critics, since it is difficult to assess wether they reached or will reach a large audience.
Details:
- p. 4 please do not qualify KL as a distance metric
- Section 4.3: "Every GAN variant was trained for 200000 iterations, and 5 discriminator updates were done for each generator update" is ambiguous: what is exactly meant by "iteration" (and sometimes step elsewhere)?
- Section 4.3: the performance measure is not relevant regarding distributions. The l2 distance is somewhat OK for means, but it makes little sense for covariance matrices.


We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies. We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies. Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies. This framework for training deep RL policies involve a zero-sum dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region. Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm. Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a "worst" possible adversarial disturbances.

There are two anonymity violations in the paper. The first is in the sentence "The 7-DoF robot result presented shortly previously appeared in our abstract that introduced robust GPS [21]". The second is in the first linked video, which links to a non-anonymized youtube video. The second linked video, a dropbox link, does not have the correct permissions set, and thus cannot be viewed. Also, the citation style does not seem to follow the ICLR style guidelines.
Disregarding the anonymity and style violations, I will review the paper.  I do not have background in H_inf control theory, but I will review the paper to the best of my ability.
This paper proposes a guided policy search method for training deep neural network policies that are robust to worst-case additive disturbances. To my knowledge, the approach presented in the paper is novel, though some relevant references are missing. The experimental results demonstrate the method on two simulated experimental domains, demonstrating robustness to adversarial perturbations. The paper is generally well-written, but has some bugs and typos. The paper is substantially longer than the strongly suggested page limit. There are parts of the paper that should be moved to an appendix to accommodate the page limit.
Regarding the experiments:
My main concerns are with regard to the completeness of the experiments. First, the experimental results report performance in terms of cost/reward, which is extremely difficult to interpret. It would be helpful to also provide success rate for all experiments, where the authors can define success as, e.g. getting the peg in the hole or being within a certain threshold of the goal.
Second, the paper should provide a comparison of policy robustness between the proposed approach and (1) a policy trained with standard GPS, (2) a policy trained with GPS and random perturbations, and ideally, (3) prior approaches to robustness, e.g. Pinto et al., Madelkar et al. [1], or Rajeswaran et al. [2].
Regarding related work and clarity:
There are a few papers that consider the problem of building deep neural network policies that are robust [1,2] that should be discussed and ideally compared to.
Recent deep reinforcement learning work has studied the problem of robustness to adversarial perturbations in the observation space, e.g. [3,4,5,6]. As such, it would be helpful to clarify in the introduction that this paper is considering additive perturbations in the action space.
The paper switches between using rewards and costs. It would be helpful to pick one term and stick with it for the entire paper, rather than switching. Further, it seems like there are errors due to the switching. e.g. on page 3, \ell is defined as the expected reward and in equation 3, it seems like the protaganist policy is trying to minimize \ell, contradicting the earlier definition.
Lastly, section 5.1 is currently rather difficult to follow. It would help to see more top-down direction in the derivation and more details in section 5.1, 5.2, and 5.3 to be moved to an appendix.
Regarding correctness:
There seem to be some issues in the math and/or notation:
The most major issue is in Algorithm 2, which is probably the most important part of the paper to be correct, given that it provides a complete picture of the algorithm. I believe that steps 3 and 4 are incorrect and/or incomplete. Step 4 should be referring to the local policy p rather than the global policy pi (assuming the notation in sections 5.1 and 5.3). Also, the local policy p(v|x) appears nowhere in the algorithm, and probably should appear in step 4. In step 5, how is this regression different from step 7?
In equation 1 and elsewhere in section 4, there is a mix of notation, using u and z inter-changably. (e.g. in equation 1 and the following equation, I believe the u should be switched to z or the z should be switched to u).
Minor feedback:
> "requires fewer samples to generalize to the real world"
None of these experiments are in the real world, so the term "real world" should not be used.
> "algoruithm" -> "algorithm"
> "when unexpected such as when there exists" - bad grammar / typo
> "ertwhile sensy-sensitivity parameter" - typo
- reference 1 is missing authors
In summary, I think the paper would be significantly improved with further experimental comparisons, further discussion of related work, and clarifications/corrections on the notation, equations, and algorithms.  In its current form, I don't think the paper is fit for publication. My rating is between a 4 and a 5.
[1] http://vision.stanford.edu/pdf/mandlekar2017iros.pdf
[2] https://arxiv.org/abs/1610.01283
[2] https://arxiv.org/abs/1701.04143
[3] https://arxiv.org/abs/1702.02284
[4] https://www.ijcai.org/proceedings/2017/0525.pdf
[5] https://arxiv.org/abs/1705.06452


ONLINE LEARNING RATE ADAPTATION WITH HYPERGRADIENT DESCENT
We introduce a general method for improving the convergence rate of gradientbased optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

SUMMARY:
The authors reinvent a 20 years old technique for adapting a global or component-wise learning rate for gradient descent. The technique can be derived as a gradient step for the learning rate hyperparameter, or it can be understood as a simple and efficient adaptation technique.
GENERAL IMPRESSION:
One central problem of the paper is missing novelty. The authors are well aware of this. They still manage to provide added value.
Despite its limited novelty, this is a very interesting and potentially impactful paper. I like in particular the detailed discussion of related work, which includes some frequently overlooked precursors of modern methods.
CRITICISM:
The experimental evaluation is rather solid, but not perfect. It considers three different problems: logistic regression (a convex problem), and dense as well as convolutional networks. That's a solid spectrum. However, it is not clear why the method is tested only on a single data set: MNIST. Since it is entirely general, I would rather expect a test on a dozen different data sets. That would also tell us more about a possible sensitivity w.r.t. the hyperparameters \alpha_0 and \beta.
The extensions in section 5 don't seem to be very useful. In particular, I cannot get rid of the impression that section 5.1 exists for the sole purpose of introducing a convergence theorem. Analyzing the actual adaptive algorithm would be very interesting. In contrast, the present result is trivial and of no interest at all, since it requires knowing a good parameter setting, which defeats a large part of the value of the method.
MINOR POINTS:
page 4, bottom: use \citep for Duchi et al. (2011).
None of the figures is legible on a grayscale printout of the paper. Please do not use color as the only cue to identify a curve.
In figure 2, top row, please display the learning rate on a log scale.
page 8, line 7 in section 4.3: "the the" (unintended repetition)
End of section 4: an increase from 0.001 to 0.001002 is hardly worth reporting - or am I missing something?


THE REACTOR: A FAST AND SAMPLE-EFFICIENT ACTOR-CRITIC AGENT FOR REINFORCEMENT LEARNING
In this work, we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2017) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms, designed for expected value evaluation, into distributional algorithms. Next, we introduce the β-leave-one-out policy gradient algorithm, which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.

This paper proposes a novel reinforcement learning algorithm (« The Reactor ») based on the combination of several improvements to DQN: a distributional version of Retrace, a policy gradient update rule called beta-LOO aiming at variance reduction, a variant of prioritized experience replay for sequences, and a parallel training architecture. Experiments on Atari games show a significant improvement over prioritized dueling networks in particular, and competitive performance compared to Rainbow, at a fraction of the training time.
There are definitely several interesting and meaningful contributions in this submission, and I like the motivations behind them. They are not groundbreaking (essentially extending existing techniques) but are still very relevant to current RL research.
Unfortunately I also see it as a step back in terms of comparison to other algorithms. The recent Rainbow paper finally established a long overdue clear benchmark on Atari. We have seen with the « Deep Reinforcement Learning that Matters » paper how important (and difficult) it is to properly compare algorithms on deep RL problems. I assume that this submission was mostly written before Rainbow came out, and that comparisons to Rainbow were hastily added just before the ICLR deadline: this would explain why they are quite limited, but in my opinion it remains a major issue, which is the main reason why I am advocating for rejection.
More precisely, focusing on the comparison to Rainbow which is the main competitor here, my concerns are the following:
- There is almost no discussion on the differences between Reactor and Rainbow (actually the paper lacks a « related work » section). In particular Rainbow also uses a version of distributional multi-step, which as far as I can tell may not be as well motivated (from a mathematical point of view) as the one in this submission (since it does not correct for the « off-policyness » of the replay data), but still seems to work well on Atari.
- Rainbow is not distributed. This was a deliberate choice by its authors to focus on algorithmic comparisons. However, it seems to me that it could benefit from a parallel training scheme like Reactor’s. I believe a comparison between Reactor and Rainbow needs to either have them both parallelized or none of them (especially for a comparison on time efficiency like in Fig. 2)
- Rainbow uses the traditional feedforward DQN architecture while Reactor uses a recurrent network. It is not clear to which extent this has an impact on the results.
- Rainbow was stopped at 200M steps, at which point it seems to be overall superior to Reactor at 200M steps. The results as presented here emphasize the superiority of Reactor at 500M steps, but a proper comparison would require Rainbow results at 500M steps as well.
In addition, although I found most of the paper to be clear enough, some parts were confusing to me, in particular:
- « multi-step distributional Bellman operator » in 3.2: not clear exactly what the target distribution is. If I understand correctly this is the same as the Rainbow extension, but this link is not mentioned.
- 3.4.1 (network architecture): a simple diagram in the appendix would make it much easier to understand (Table 3 is still hard to read because it is not clear which layers are connected together)
- 3.3 (prioritized sequence replay): again a visual illustration of the partitioning scheme would in my opinion help clarify the approach
A few minor points to conclude:
- In eq. 6, 7 and the rest of this section, A does not depend (directly) on theta so it should probably be removed to avoid confusion. Note also that using the letter A may not be best since A is used to denote an action in 3.1.
- In 3.1: « Let us assume that for the chosen action A we have access to an estimate R(A) of Qπ(A) » => « unbiased estimate »
- In last equation of p.5 it is not clear what q_i^n is
- There is a lambda missing on p.6 in the equation showing that alphas are non-negative on average, just before the min
- In the equation above eq. 12 there is a sum over « i=1 »
- That same equation ends with some h_z_i that are not defined
- In Fig. 2 (left) for Reactor we see one worker using large batches and another one using many threads. This is confusing.
- 3.3 mentions sequences of length 32 but 3.4 says length 33.
- 3.3 says tree operations are in O(n ln(n)) but it should be O(ln(n))
- At very end of 3.3 it is not clear what « total variation » is.
- In 3.4 please specify the frequency at which the learner thread downloads shared parameters and uploads updates
- Caption of Fig. 3 talks about « changing the number of workers » for the left plot while it is in the right plot
- The explanation on what the variants of Reactor (ND and 500M) mean comes after results are shown in Fig. 2.
- Section 4 starts with Fig. 3 without explaining what the task is, how performance is measured, etc. It also claims that Distributional Retrace helps while this is not the case in Fig. 3 (I realize it is explained afterwards, but it is confusing when reading the sentence « We can also see... »). Finally it says priorization is the most important component while the beta-LOO ablation seems to perform just the same.
- Footnote 3 should say it is 200M observations except for Reactor 500M
- End of 4.1: « The algorithms that we compare Reactor against are » => missing ACER, A3C and Rainbow
- There are two references for « Sample efficient actor-critic with experience replay »
- I do not see the added benefit of the Elo computation. It seems to convey essentially the same information as average rank.
And a few typos:
- Just above 2.1.3: « increasing » => increasingly
- In 3.1: « where V is a baseline that depend » => depends
- p.7: « hight » => high, and « to all other sequences » => of all other sequences
- Double parentheses in Bellemare citation at beginning of section 4
- Several typos in appendix (too many to list)
Note: I did not have time to carefully read Appendix 6.3 (contextual priority tree)
Edit after revision: bumped score from 5 to 7 because (1) authors did many improvements to the paper, and (2) their explanations shed light on some of my concerns


SCAN: LEARNING HIERARCHICAL COMPOSITIONAL VISUAL CONCEPTS
The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.

Summary
---
This paper proposes a new model called SCAN (Symbol-Concept Association Network) for hierarchical concept learning. It trains one VAE on images then another one on symbols and aligns their latent spaces. This allows for symbol2image and image2symbol inference. But it also allows for generalization to new concepts composed from existing concepts using logical operators. Experiments show that SCAN generates images which correspond to provided concept labels and span the space of concepts which match these labels.
The model starts with a beta-VAE trained on images (x) from the relevant domain (in this case, simple scenes generated from DeepMind Lab which vary across a few known dimensions). This is complemented by the SCAN model, which is a beta-VAE trained to reconstruct symbols (y; k-hot encoded concepts like {red, suitcase}) with a slightly modified objective. SCAN optimizes the ELBO plus a KL term which pushes the latent distribution of the y VAE toward the latent distribution of the x (image) VAE. This aligns the latent representations so now a symbol can be encoded into a latent distribution z and decoded as an image.
One nice property of the learned latent representation is that more specific concepts have more specific latent representations. Consider latent distributions z1 and z2 for a more general symbol {red} and a more specific symbol {red, suitcase}. Fewer dimensions of z2 have high variance than dimensions of z1. For example, the latent space could encode red and suitcase in two dimensions (as binary attributes). z1 would have high variance on all dimensions but the one which encodes red and z2 would have high variance on all dimensions but red and suitcase. In the reported experiments some of the dimensions do seem to be interpretable attributes (figure 5 right).
SCAN also pays particular attention to hierarchical concepts. Another very simple model (1d convolution layer) is learned to mimic logical operators. Normally a SCAN encoder takes {red} as input and the decoder reconstructs {red}. Now another model is trained that takes "{red} AND {suitcase}" as input and reconstructs {red, suitcase}. The two input concepts {red} and {suitcase} are each encoded by a pre-trained SCAN encoder and then those two distributions are combined into one by a simple 1d convolution module trained to implement the AND operator (or IGNORE/IN COMMON). This allows images of concepts like {small, red, suitcase} to be generated even if small red suitcases are not in the training data.
Experiments provide some basic verification and analysis of the method:
1) Qualitatively, concept samples are correct and diverse, generating images with all configurations of attributes not specified by the input concept.
2) As SCAN sees more diverse examples of a concept (e.g. suitcases of all colors instead of just red ones) it starts to generate more diverse image samples of that concept.
3) SCAN samples/representations are more accurate (generate images of the right concept) and more diverse (far from a uniform prior in a KL sense) than JMVAE and TELBO baselines.
4) SCAN is also compared to SCAN_U, which uses an image beta-VAE that learned an entangled (Unstructured) representation. SCAN_U performed worse than SCAN
and baselines.
5) Concepts expressed as logical combinations of other concepts generalize well for both the SCAN representation and the baseline representations.
Strengths
---
The idea of concept learning considered here is novel and satisfying. It imposing logical, hierarchical structure on latent representations in a general way. This suggests opportunities for inserting prior information and adds interpretability to the latent space.
Weaknesses
---
I think this paper is missing some important evaluation.
Role/Nature of Disentangled Features not Clear (major):
* Disentangled features seem to be very important for SCAN to work well (SCAN vs SCAN_U). It seems that the only difference between the unstructured (entangled) and the structured (disentangled) visual VAE is the color space of the input (RGB vs HSV). If so, this should be stated more clearly in the main paper. What role did beta-VAE (tuning beta) as opposed to plain VAE play in learning disentangled features?
* What color space was used for the JMVAE and TELBO baselines? Training these with HSV seems especially important for establishing a good comparison, but it would be good to report results for HSV and RGB for all models.
* How specific is the HSV trick to this domain? Would it matter for natural images?
* How would a latent representation learned via supervision perform? (Maybe explicitly align dimensions of z to red/suitcase/small with supervision through some mechanism. c.f. "Discovering Hidden Factors of Variation in Deep Networks" by Cheung et al.)
Evaluation of sample complexity (major):
* One of the main benefits of SCAN is that it works with less training data. There should be a more systematic evaluation of this claim. In particular, I would like to see a Number of Examples vs Performance (Accuracy/Diversity) plot for both SCAN and the baselines.
Minor questions/comments/concerns:
* What do the logical operators learn that the hand-specified versions do not?
* Does training SCAN with the structure provided by the logical operators lead to improved performance?
* There seems to be a mistake in figure 5 unless I interpreted it incorrectly. The right side doesn't match the left side. During the middle stage of training object hues vary on the left, but floor color becomes less specific on the right. Shouldn't object color become less specific?
Prelimary Evaluation
---
This clear and well written paper describes an interesting and novel way of learning a model of hierarchical concepts. It's missing some evaluation that would help establish the sample complexity benefit more precisely (a claimed contribution) and add important details about unsupervised disentangled representations. I would happy to increase my rating if these are addressed.


Deep neural networks (DNNs) had great success on NLP tasks such as language modeling, machine translation and certain question answering (QA) tasks. However, the success is limited at more knowledge intensive tasks such as QA from a big corpus. Existing end-to-end deep QA models (Miller et al., 2016;Weston et al., 2014) need to read the entire text after observing the question, and therefore their complexity in responding a question is linear in the text size. This is prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web. We propose to solve this scalability issue by using symbolic meaning representations, which can be indexed and retrieved efficiently with complexity that is independent of the text size. More specifically, we use sequence-to-sequence models to encode knowledge symbolically and generate programs to answer questions from the encoded knowledge. We apply our approach, called the N-Gram Machine (NGM), to the bAbI tasks  and a special version of them ("life-long bAbI") which has stories of up to 10 million sentences. Our experiments show that NGM can successfully solve both of these tasks accurately and efficiently. Unlike fully differentiable memory models, NGM's time complexity and answering quality are not affected by the story length. The whole system of NGM is trained end-to-end with REINFORCE (Williams, 1992). To avoid high variance in gradient estimation, which is typical in discrete latent variable models, we use beam search instead of sampling. To tackle the exponentially large search space, we use a stabilized auto-encoding objective and a structure tweak procedure to iteratively reduce and refine the search space.

This paper presents the n-gram machine, a model that encodes sentences into simple symbolic representations ("n-grams") which can be queried efficiently. The authors propose a variety of tricks (stabilized autoencoding, structured tweaking) to deal with the huge search space, and they evaluate NGMs on five of the 20 bAbI tasks. I am overall a fan of the general idea of this paper; scaling up to huge inputs is definitely a necessary research direction for QA. However, I have some concerns about the specific implementation and model discussed here. How much of the proposed approach is specific to getting good results on bAbI (e.g., conditioning the knowledge encoder on only the previous sentence, time stamps in the knowledge tuple, super small RNNs, four simple functions in the n-gram machine, structure tweaking) versus having a general-purpose QA model for natural language? Addressing some of these issues would likely prevent scaling to millions of (real) sentences, as the scalability is reliant on programs being efficiently executed (by simple string matching) against a knowledge storage. The paper is missing a clear analysis of NGM's limitations... the examples of knowledge storage from bAbI in the supplementary material are also underwhelming as the model essentially just has to learn to ignore stopwords since the sentences are so simple. In its current form, I am borderline but leaning towards rejecting this paper.
Other questions:
- is "n-gram" really the most appropriate term to use for the symbolic representation? N-grams are by definition contiguous sequences... The authors may want to consider alternatives.
- why focus only on extractive QA? The evaluations are only conducted on 5 of the 20 bAbI tasks, so  it is hard to draw any conclusions from the results as to the validity of this approach. Can the authors comment on how difficult it will be to add functions to the list in Table 2 to handle the other 15 tasks? Or is NGM strictly for extractive QA?
- beam search is performed on each sentence in the input story to obtain knowledge tuples... while the answering time may not change (as shown in Figure 4) as the input story grows, the time to encode the story into knowledge tuples certainly grows, which likely necessitates the tiny RNN sizes used in the paper. How long does the encoding time take with 10 million sentences?
- Need more detail on the programmer architecture, is it identical to the one used in Liang et al., 2017?


Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks (RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training. Specifically, we parameterize the transition matrix by its singular value decomposition (SVD), which allows us to explicitly track and control its singular values. We attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. By explicitly controlling the singular values, our proposed svdRNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent. We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron. Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it potentially makes the optimization process easier. Our extensive experimental results also demonstrate that the proposed framework converges faster, and has good generalization, especially in capturing long range dependencies, as shown on the synthetic addition and copying tasks.

This paper suggests a reparametrization of the transition matrix. The proposed reparametrization which is based on Singular Value Decomposition can be used for both recurrent and feedforward networks.
The paper is well-written and authors explain related work adequately. The paper is a follow up on Unitary RNNs which suggest a reparametrization that forces the transition matrix to be unitary. The problem of vanishing and exploding gradient in deep network is very challenging and any work that shed lights on this problem can have a significant impact.
I have two comments on the experiment section:
- Choice of experiments. Authors have chosen UCR datasets and MNIST for the experiments while other experiments are more common. For example, the adding problem, the copying problem and the permuted MNIST problem and language modeling are the common experiments in the context of RNNs. For feedforward settings, classification on CIFAR10 and CIFAR100 is often reported.
- Stopping condition. The plots suggest that the optimization has stopped earlier for some models. Is this because of some stopping condition or because of gradient explosion? Is there a way to avoid this?
- Quality of figures. Figures are very hard to read because of small font. Also, the captions need to describe more details about the figures.