Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum. This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning. We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space. In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards.

This paper proposes a new method for reverse curriculum generation by gradually reseting the environment in phases and classifying states that tend to lead to success. It additionally proposes a mechanism for learning from human-provided "key states".
The ideas in this paper are quite nice, but the paper has significant issues with regard to clarity and applicability to real-world problems:
First, it is unclear is the proposed method requires access only high-dimensional observations (e.g. images) during training or if it additionally requires low-dimensional states (e.g. sufficient information to reset the environment). In most compelling problems settings where a low-dimensional representation that sufficiently explains the current state of the world is available during training, then it is also likely that one can write down a nicely shaped reward function using that state information during training, in which case, it makes sense to use such a reward function. This paper seems to require access to low-dimensional states, and specifically considers the sparse-reward setting, which seems contrived.
Second, the paper states that the assumption "when resetting, the agent can be reset to any state" can be satisfied in problems such as real-world robotic manipulation. This is not correct. If the robot could autonomously reset to any state, then we would have largely solved robotic manipulation. Further, it is not always realistic to assume access to low-dimensional state information during training on a real robotic system (e.g. knowing the poses of all of the objects in the world).
Third, the experiments section lacks crucial information needed to understand the experiments. What is the state, observation, and action space for each problem setting? What is the reward function for each problem setting? What reinforcement learning algorithm is used in combination with the curriculum and tendency rewards? Are the states and actions continuous or discrete? Without this information, it is difficult to judge the merit of the experimental setting.
Fourth, the proposed method seems to lack motivation, making the proposed scheme seem a bit ad hoc. Could each of the components be motivated further through more discussion and/or ablative studies?
Finally, the main text of the paper is substantially longer than the recommended page limit. It should be shortened by making the writing more concise.
Beyond my feedback on clarity and significance, here are further pieces of feedback with regard to the technical content, experiments, and related work:
I'm wondering -- can the reward shaping in Equation 2 be made to satisfy the property of not affecting the final policy? (see Ng et al. '09) If so, such a reward shaping would make the method even more appealing.
How do the experiments in section 5.4 compare to prior methods and ablations? Without such a comparison, it is impossible to judge the performance of the proposed method and the level of difficulty of these tasks. At the very least, the paper should compare the performance of the proposed method to the performance a random policy.
The paper is missing some highly relevant references. First, how does the proposed method compare to hindsight experience replay? [1] Second, learning from keyframes (rather than demonstrations) has been explored in the past [1]. It would be preferable to use the standard terminology of "keyframe".
[1] Andrychowicz et al. Hindsight Experience Replay. 2017
[2] Akgun et al. Keyframe-based Learning from Demonstration. 2012
In summary, I think this paper has a number of promising ideas and experimental results, but given the significant issues in clarity and significance to real world problems, I don't think that the current version of this paper is suitable for publication in ICLR.
More minor feedback on clarity and correctness:
- Abstract: "Deep RL algorithms have proven successful in a vast variety of domains" -- This is an overstatement.
- The introduction should be more clear with regard to the assumptions. In particular, it would be helpful to see discussion of requiring human-provided keyframes. As is, it is unclear what is meant by "checkpoint scheme", which is not commonly used terminology.
- "This kind of spare reward, goal-oriented tasks are considered the most difficult challenges" -- This is also an overstatement. Long-horizon tasks and high-dimensional observations are also very difficult. Also, the sentence is not grammatically correct.
- "That is, environment" -> "That is, the environment"
- In the last paragraph of the intro, it would be helpful to more clearly state what the experiments can accomplish. Can they handle raw pixel inputs?
- "diverse domains" -> "diverse simulated domains"
- "a robotic grasping task" -> "a simulated robotic grasping task"
- There are a number of issues and errors in citations, e.g. missing the year, including the first name, incorrect reference
- Assumption 1: \mathcal{P} has not yet been defined.
- The last two paragraphs of section 3.2 are very difficult to understand without reading the method yet
- "conventional RL solver tend" -> "conventional RL tend", also should mention sparse reward in this sentence.
- Algorithm 1 and Figure 1 are not referenced in the text anywhere, and should be
- The text in Figure 1 and Figure 3 is extremely small
- The text in Figure 3 is extremely small


Published as a conference paper at ICLR 2018 IMPROVING THE UNIVERSALITY AND LEARNABIL- ITY OF NEURAL PROGRAMMER-INTERPRETERS WITH COMBINATOR ABSTRACTION
To overcome the limitations of Neural Programmer-Interpreters (NPI) in its universality and learnability, we propose the incorporation of combinator abstraction into neural programing and a new NPI architecture to support this abstraction, which we call Combinatory Neural Programmer-Interpreter (CNPI). Combinator abstraction dramatically reduces the number and complexity of programs that need to be interpreted by the core controller of CNPI, while still allowing the CNPI to represent and interpret arbitrary complex programs by the collaboration of the core with the other components. We propose a small set of four combinators to capture the most pervasive programming patterns. Due to the finiteness and simplicity of this combinator set and the offloading of some burden of interpretation from the core, we are able construct a CNPI that is universal with respect to the set of all combinatorizable programs, which is adequate for solving most algorithmic tasks. Moreover, besides supervised training on execution traces, CNPI can be trained by policy gradient reinforcement learning with appropriately designed curricula.

Quality
The paper is very interesting and clearly motivated. The idea of importing concepts from functional programming into neural programming looks very promising, helping to address a bit the somewhat naive approach taken so far in the deep learning community towards program induction. However, I found the model description difficult to fully understand and have significant unresolved questions - especially *why* exactly the model should be expected to have better universality compared to NPI and RNPI, given than applier memory is unbounded just like NPI/RNPI program memories are unbounded.
Clarity
The paper does a good job of summarizing NPI and motivating the universality property of the core module.
I had a lot of questions while reading:
What is the purpose of detectors? It is not clear what is being detected. From the context it seems to be encoding observations from the environment, which can vary according to the task and change during program execution. The detector memory is also confusing. In the original NPI, it is assumed that the caller knows which encoder is needed for each program. In CNPI, is this part learned or more general in some way?
Appliers - is it the case that *every* program apart from the four combinators must be written as an applier? For example ADD1, BSTEP, BUBBLESORT, etc all must be implemented as an applier, and programs that cannot be implemented as appliers are not expressible by CNPI?
Memory - combinator memory looks like a 4-way softmax over the four combinators, right? The previous NPI program memory is analogous then to the applier memory.
Eqn 3 - binarizing the detector output introduces a non-differentiable operation. How is the detector then trained e.g. from execution traces? Later I see that there is a notion of a “correct condition” for the detector to regress on, which makes me confused again about what exactly the output of a detector means.
Computing the next subprogram - since the size of applier memory is unbounded, the core still needs to be aware of an unlimited number of subprograms. I must be missing something here - how does the proposed model therefore achieve better universality than the original NPI and RNPI models?
Analysis - for the claim of perfect generalization, I think this will not generally hold true for perceptual inputs. Will the proposed model only be useful in discrete domains for algorithmic tasks, or could it be more broadly applicable, e.g. to robotics tasks?
Originality
This methods proposed in this paper are quite novel and start to bridge an important gap between neural program induction and functional programming, by importing the concept of combinator abstraction into NPI.
Significance
The paper will be significant to people interested in NPI-related models and neural program induction generally, but on the other hand, there is currently not yet a “killer application” to this line of work.
The experiments appear to show significant new capabilities of CNPI compared to NPI and RNPI in terms of better generalization and universality, as well as being trainable by reinforcement learning.
Pros
- Learns new programs without catastrophic forgetting in the NPI core, in particular where previous NPI models fail.
- Detector training is decoupled from core and memory training, so that perfect generalization does not have to be re-verified after learning new behaviors.
Cons
- So far lacking useful applications in the real world. Could the techniques in this paper help in robotics extensions to NPI? (see e.g. https://arxiv.org/abs/1710.01813)
- Adds a significant amount of further structure into the NPI framework, which could potentially make broader applications more complex to implement. Do the proposed modifications reduce generality in any way?


Partial differential equations (PDEs) play a prominent role in many disciplines such as applied mathematics, physics, chemistry, material science, computer science, etc. PDEs are commonly derived based on physical laws or empirical observations. However, the governing equations for many complex systems in modern applications are still not fully known. With the rapid development of sensors, computational power, and data storage in the past decade, huge quantities of data can be easily collected and efficiently stored. Such vast quantity of data offers new opportunities for data-driven discovery of hidden physical laws. Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulfill two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models. The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (filters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses. Comparing with existing approaches, which either assume the form of the nonlinear response is known or fix certain finite difference approximations of differential operators, our approach has the most flexibility by learning both differential operators and the nonlinear responses. A special feature of the proposed PDE-Net is that all filters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network. These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of filters (an important concept originated from wavelet theory). We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet). Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

Authors propose a neural network based algorithm for learning from data that arises from dynamical systems with governing equations that can be written as partial differential equations.  The network architecture is constrained such that regardless of the parameters, it always implements discretization of an arbitrary PDE.  Through learning, the network adapts itself to solve a specific PDE.  Discretization is finite difference in space and forward Euler in time.
The article is quite novel in my opinion.  To the best of my knowledge, it is the first article that implements a generic method for learning arbitrary PDE models from data.  In using networks, the method differs from previously proposed approaches for learning PDEs.  Experiments are only presented with synthetic data but given the potential for the method and its novelty, I believe this can be accepted.  However, it would have been a stronger article if authors have applied to a real life model with real initial and boundary conditions, and real observations.
I have three main criticism about the article:
 1.   Authors do not cite Chen and Pock’s article on learning diffusion filters with networks that first published in CVPR 2015 and then authors published a PAMI article this year.  To the best of my knowledge, they are the first to show the connection between res-net type architecture and numerical solutions of PDEs. I think proper credit should be given. [I need to emphasize that I am not an author in that article.]
 2.   Authors emphasize the importance of interpretability, however, the constraint on the moment matrices might cripple this aspect.  The frozen filters have clear interpretations. They are in the end finite difference approximations with some level of accuracy depending on the size and the number of zeros.  When M(q) matrix is free to change, it is unclear what the effect will be on the filters.  Are the numbers that replace stars in Equation 6 for instance, will be absorbed in the O(\epsilon) term?  Can one really interpret the final c_{ij} for filters whose M(q) have many non-zeros?
 3.   The introduction and results sections are well written.  The method section on the other hand, needs improvement.  The notation is not easy to follow due to missing definitions.  I believe with proper definitions — amounting to small modifications — readability of the article can substantially improve.
In addition to the main criticisms, I have some other questions and concerns:
  1.  How sensitive is the model?  In real life, one cannot expect to get observations every delta t.  Data is most often very sparse.  Can the model learn in that regime?  Can it differentiate between different PDEs and find the correct one with sparse data?
  2.  The average operations decreases the interpretability of the proposed model as a PDE.  Depending on the filter size, D_{0}u can deviate from u, which should be the term that should be used in the residual block.  Why do authors need this?  How does the model behave without it?
  3.  The statement “Thus, the PDE-Net with bigger n owns a longer time stability.” is a very vague statement.  I understand with larger n, training would be easier since more data would be used to estimate parameters.  However, it is not clear how this relates to “time stability”, which is also not defined in the article.
 4.  How is the relative error computed?  Values in relative error plots goes as high as 10^2.  That would be a huge error if it is relative.


We present a radial basis function solver for convolutional neural networks that can be directly applied to both distance metric learning and classification problems. Our method treats all training features from a deep neural network as radial basis function centres and computes loss by summing the influence of a feature's nearby centres in the embedding space. Having a radial basis function centred on each training feature is made scalable by treating it as an approximate nearest neighbour search problem. End-to-end learning of the network and solver is carried out, mapping high dimensional features into clusters of the same class. This results in a well formed embedding space, where semantically related instances are likely to be located near one another, regardless of whether or not the network was trained on those classes. The same loss function is used for both the metric learning and classification problems. We show that our radial basis function solver outperforms state-of-the-art embedding approaches on the Stanford Cars196 and CUB-200-2011 datasets. Additionally, we show that when used as a classifier, our method outperforms a conventional softmax classifier on the CUB-200-2011, Stanford Cars196, Oxford 102 Flowers and Leafsnap fine-grained classification datasets.

The authors propose a loss that is based on a RBF loss for metric learning and incorporates additional per exemplar weights in the index for classification. Significant improvements over softmax are shown on several datasets.
IMHO, this could be a worthwhile paper, but the framing of the paper into existing literature is lacking and thus it appears as if the authors are re-inventing the wheel (NCA loss) under a different name (RBF solver).
The specific problems are:
- The authors completely miss the connection to NCA loss (https://papers.nips.cc/paper/2566-neighbourhood-components-analysis.pdf) and thus appear to be re-inventing the wheel.
  - The proposed metric learning scenario is exactly as proposed in the NCA loss works, while the classification approach adds an interesting twist by learning per exemplar weights. I haven't encountered this before and it could make an interesting proposal. Of course the benefit of this should be evaluated in ablation studies( Tab 3 shows one experiment with marginal improvements).
- The authors' use of 'solver' seems uncommon and confusing. What is proposed is a loss in addition to building a weighted index in the case of classification.
- In the metric learning comparison with softmax (end of page 9) the authors mentions that a Gaussian standard deviation for softmax is learned. It appears as if the authors use the softmax logits as embedding whereas the more common approach is to use the bottleneck layer. This is also indicated by the discussion at the end of page 10 where the authors mention that softmax is restricted to axis aligned embeddings. All softmax metric learning experiments should be carried out on appropriately sized bottleneck layers.
- Some of the motivations of what the various methods learn seem flawed, e.g. triplet loss CAN learn multiple modes per class and there is nothing in the Softmax loss that encourages the classes to fill a large region of the space.
- Why don't the authors compare on ImageNet?
Some positive points:
- The authors mention in Sec 3.3 that updating the RBF centres is not required. This is a crucial point that should be made a centerpiece of this work, as there are many metric learning works that struggle with this. Additional experiments that can investigate this point would greatly contribute to a well rounded paper.
- The numbers reported in Tab 1 show very significant improvements
If the paper was re-framed and builds on top of the already existing NCA loss, there could be valuable contributions in this paper. The experimental comparisons are lacking in some respect, as the comparison with Softmax as a metric learning method seems uncommon, i.e. using the logits instead of the bottleneck layer. I encourage the authors to extend the paper and flesh out some of the experiments and then submit it again.


Published as a conference paper at ICLR 2018 MONOTONIC CHUNKWISE ATTENTION
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-theart results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.

The paper proposes an extension to a previous monotonic attention model (Raffel et al 2017) to attend to a fixed-sized window up to the alignment position. Both the soft attention approximation used for training the monotonic attention model, and the online decoding algorithm is extended to the chunkwise model. In terms of the model this is a relatively small extention of Raffel et al 2017.
Results show that for online speech recognition the model matches the performance of an offline soft attention baseline, doing significantly better than the monotonic attention model. Is the offline attention baseline unidirectional or bidirectional? In case it is unidirectional it cannot really be claimed that the proposed model's performance is competitive with an offline model.
My concern with the statement that all hyper-parameters are kept the same as the monotonic model is that the improvement might partly be due to the increase in total number of parameters in the model. Especially given that w=2 works best for speech recognition, it not clear that the model extension is actually helping. My other concern is that in speech recognition the time-scale of the encoding is somewhat arbitrary, so possibly a similar effect could be obtained by doubling the time frame through the convolutional layer. While the empirical result is strong it is not clear that the proposed model is the best way to obtain the improvement.
For document summarization the paper presents a strong result for an online model, but the fact that it is still less accurate than the soft attention baseline makes it hard to see the real significance of this. If the contribution is in terms of speed (as shown with the synthetic benchmark in appendix B) more emphesis should be placed on this in the paper.
Sentence summarization tasks do exhibit mostly monotonic alignment, and most previous models with monotonic structure were evaluated on that, so why not test that here?
I like the fact that the model is truely online, but that contribution was made by Raffel et al 2017, and this paper at best proposes a slightly better way to train and apply that model.
---
 The additional experiments in the new version gives stronger support in favour of the proposed model architecture (vs the effect of hyperparameter choices). While I'm still on the fence on whether this paper is strong enough to be accepted for ICLR, this version is certainly improves the quality of the paper.


PROGRAM SYNTHESIS WITH PRIORITY QUEUE TRAINING
We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards. We introduce an iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far. Then, we synthesize new programs and add them to the priority queue by sampling from the RNN. We benchmark our algorithm, called priority queue training (or PQT), against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF. Our experimental results show that our simple PQT algorithm significantly outperforms the baselines. By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.

This paper focuses on using RNNs to generate straightline computer programs (ie. code strings) using reinforcement learning.  The basic setup assumes a setting where we do not have access to input/output samples, but instead only have access to a separate reward function for each desired program that indicates how close a predicted program is to the correct one.  This reward function is used to train a separate RNN for each desired program.
The general space of generating straight-line programs of this form has been explored before, and their main contribution is the use of a priorty queue of highest scoring programs during training.  This queue contains the highest scoring programs which have been observed at any point in the training so far, and they consider two different objectives:  (1) the standard policy-gradient objective which tries to maximize the expected reward and (2) a supervised learning objective which tries to maximize the average probability of the top-K samples.  They show that this priority queue algorithm significantly improves the stability of the resulting synthesis procedure such that when synthesis succeeds at all, it succeeds for most of the random seeds used.
This is a nice result, but I did not feel as though their algorithm was sufficently different from the algorithm used by Liang et. al. 2017.  In Liang et. al. they keep around the best observed program for each input sample.  They argue that their work is different from Liang et. al. because they show that they can learn effectively using only objective (2) while completely dropping objective (1).  However I'm quite worried these results only apply in very specific setups.  It seems that if the policy gradient objective is not used, and there are not K different programs which generate the correct output, then the Top-K objective alone will encourage the model to continue to put equal probability on the programs in the Top-K which do not generate an incorrect output.
I also found the setup itself to be poorly motivated.  I was not able to imagine a reasonable setting where we would have access to a reward function of this form without input/output examples.  The paper did not provide any such examples, and in their experiments they implement the proposed reward function by assuming access to a set of input/output examples.  I feel as though the restriction to the reward function in this case makes the problem uncessarily hard, and does not represent an important use-case.
In addition I had the following more minor concerns:
1.  At the end of section 4.3 the paper is inconsistent about whether the test cases are randomly generated or hand picked, and whether they use 5 test cases for all problems, or sometimes up to 20 test cases.  If they are hand picked (and the number of test cases is hand chosen for each problem), then how dependant are the results on an appropriate choice of test cases?
2.  They argue that they don't need to separate train and test, but I think it is important to be sure that the generated programs work on test cases that are not a part of the reward function.  They say that "almost always" the synthesizer does not overfit, but I would have liked them to be clear about whether their reported results include any cases of overfitting (i.e. did they ensure they the final generate program always generalized)?
3.  It is worth noting that while their technique succeeds much more consistently than the baseline genetic algorithm, the genetic algorithm actually succeeds at least once, on more tasks (19 vs. 17).  The success rate is probably a good indicator of whether the technique will scale to more complex problems, but I would have prefered to see this in the results, rather than just hoping it will be true (i.e. by including my complicated problems where the genetic algorithm never succeeds).


Long short-term memory networks (LSTMs) were introduced to combat vanishing gradients in simple recurrent neural networks (S-RNNs) by augmenting them with additive recurrent connections controlled by gates. We present an alternate view to explain the success of LSTMs: the gates themselves are powerful recurrent models that provide more representational power than previously appreciated. We do this by showing that the LSTM's gates can be decoupled from the embedded S-RNN, producing a restricted class of RNNs where the main recurrence computes an element-wise weighted sum of context-independent functions of the inputs. Experiments on a range of challenging NLP problems demonstrate that the simplified gate-based models work substantially better than S-RNNs, and often just as well as the original LSTMs, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

This paper presents an analysis of LSTMS showing that they have a from where the memory cell contents at each step is a weighted combination of the “content update” values computed at each time step. The weightings are defined in terms of an exponential decay on each dimension at each time step (given by the forget gate), which lets the cell be computed sequentially in linear time rather than in the exhaustive quadratic time that would apparently be necessary for this definition. Second, the paper offers a simplification of LSTMs that compute the value by which the memory cell at each time step in terms of a deterministic function of the input rather than a function of the input and the current context. This reduced form of the LSTM is shown to perform comparably to “full” LSTMs.
The decomposition of the LSTM in terms of these weights is useful, and suggests new strategies for comparing existing quadratic time attention-based extensions to RNNs. The proposed model variations (which replaces the “content update” that has a recurrent network in terms of context-independent update) and their evaluations seem rather more arbitrary. First, there are two RNNs present in the LSTM- one controls the gates, one controls the content update. You get rid of one, not the other. You can make an argument for why the one that was ablated was “more interesting”, but really this is an obvious empirical question that should be addressed. The second problem of what tasks to evaluate on is a general problem with comparing RNNs. One non-language task (e.g., some RL agent with an LSTM, or learning to execute or something) and one synthetic task (copying or something) might be sensible. Although I don’t think this is the responsibility of this paper (although something that should be considered).
Finally, there are many further simplifications of LSTMs that could have been explored in the literature: coupled input-forget gates (Greff et al, 2015), diagonal matrices for gates, GRUs. When proposing yet another simplification, some sense for how these different reductions is useful, so I would recommend comparison to those.
Notes on clarity:
Before Eq 1 it’s hard to know what the antecedent of “which” is without reading ahead.
For componentwise multiplication, you have been using \circ, but then for the iterated component wise product, \prod is used. To be consistent, notation like \odot and \bigodot might be a bit clearer.
The discussion of dynamic programming: the dynamic program is also only available because the attention pattern is limited in a way that self attention is not. This might be worth mentioning.
When presenting Eq 11, the definition of w_j^t elides a lot of complexity. Indeed, w_j^t is only ever implicitly defined in Eq 8, whereas things like the input and forget gates are defined multiple times in the text. Since w_j^t can be defined iteratively and recursively (as a dynamic program), it’s probably worth writing both out, for expository clarity.
Eq 11 might be clearer if you show that Eq 8 can also be rewritten in the same wheat, provided, you make h_{t-1} an argument to output and content.
Table 4 is unclear. In a language model, the figure looks like it is attending to the word that is being generated, which is clearly not what you want to convey since language models don’t condition on the word they are predicting. Presumably the strong diagonal attention is attending to the previous word when computing the representation to generate the subsequent word? In any case, this figure should be corrected to reflect this. This objection also concerns the right hand figure, and the semantics of the meaning of the upper vs lower triangles should be clarified in the caption (rather than just in the text).


Published as a conference paper at ICLR 2018 ON THE DISCRIMINATION-GENERALIZATION TRADE- OFF IN GANS
Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.

In more detail, the analysis of the paper is as follows.  Firstly, it primarily focuses on GAN objective functions which are "integral probability metrics (IPMs)"; one way to define these is by way of similarity to the W-GAN, namely IPMs replace the 1-Lipschitz functions in W-GAN with a generic set of functions F.  The paper overall avoids computational issues and treats the suprema as though exactly solved by sgd or related heuristic (the results of the paper simply state supremum, but some of the prose seems to touch on this issue).
The key arguments of the paper are as follows.
1. It argues that the discriminator set should be not simply large, it should be dense in all bounded continuous functions; as a consequence of this, the IPM is 0 iff the distributions are equal (in the weak sense).  Due to this assertion, it says that it suffices to use two layer neural networks as the discriminator set (as a consequence of the "universal approximation" results well-known in the neural network literature).
2. It argues the discriminator set should be small in order to mitigate small-sample effects.  (Together, points 1 and 2 mimic a standard bias-variance tradeoff in statistics.)  For this step, the paper relies upon standard Rademacher results plus a little bit of algebraic glue.  Curiously, the paper chooses to argue (and forms as a key tenet, indeed in the abstract) that the size of the generator set is irrelevant for this, only the size of the discriminator matters.
Unfortunately, I find significant problems with the paper, in order from most severe to least severe.
A.  The calculation ruling out the impact of the generator in generalization calculations in 2 above is flawed.  Before pointing out the concrete bug, I note that this assertion runs completely counter to intuition, and thus should be made with more explanation (as opposed to the fortunate magic it is presented as).  Moreover, I'll say that if the authors choose to "fix" this bug by adding a generator generalization term, the bound is still a remedial application of Rademacher complexity, so I'm not exactly blown away.  Anyway, the bug is as follows.  The equation which drops the role of the generator in the generalization calculation is the equation (10).  The proof of this inequality is at the start of appendix E.  Looking at the derivation in that appendix, everything is correct up to the second-to-last display, the one with a supremum over nu in G.  First of all, this right hand side should set off alarm bells; e.g., if we make the generator class big, we can make this right hand side essentially as big as the IPM allows even when mu = mu_m.  Now the bug itself appears when going to the next display: if the definition of d_F is expanded, one obtains two suprema, each own over _their own_ optimization variable (in this case the variables are discriminator functions).  When going to the next equation, the authors accidentally made the two suprema have the same variable and invoke a fortuitous but incorrect cancellation.  As stated a few sentences back, one can construct trivial counterexamples to these inequalities, for instance by making mu and mu_m arbitrarily close (even exactly equal if you wish) and then making nu arbitrarily far away and the discriminator set large enough to identify this.
B. The assertions in 1, regarding sizes of discriminator sets needed to achieve the goal of the IPM being 0 iff the distributions are equal (in the weak sense), are nothing more than immediate corollaries of approximation results well-known for decades in the neural network literature.  It is thus hard to consider this a serious contribution.
C. I will add on a non-technical note that the paper's assertion on what a good IPM "should be" is arguably misled.  There is not only a meaning to specific function classes (as with Lip_1 in Wasserstein_1) beyond simply "many functions", but moreover there is an interplay between the size of the generator set and the size of the discriminator set.  If the generator set is simple, then the discriminator set can also get away with being simple (this is dicussed in the Arora et al 2017 ICML paper, amongst other places).  Perhaps I am the one that is misled, but even so the paper does not appear to give a good justification of its standpoint.
I will conclude with typos and minor remarks.  I found the paper to contain a vast number of small errors, to the point that I doubted a single proofread.
Abstract, first line: "a minimizing"?  general grammar issue in this sentence; this sort of issue throughout the paper.
Abstract, "this is a mild condition".  Optimizing over a function class which is dense in all bounded measurable functions is not a mild assumption.  In the particular case under discussion, the size of the network can not be bounded (even though it has just two layers, or as the authors say is the span of single neurons).
Abstract, "...regardless of the size of the generator or hypothesis set".  This really needs explanation in the abstract, it is such a bold claim.  For instance, I wrote "no" in the margin while reading the abstract the first time.
Intro, first line: its -> their.
Intro, #3 "energy-based GANs": 'm' clashes with sample size.
Intro, bottom of page 1, the sentence with "irrelenvant": I can't make any sense of this sentence.
Intro, bottom of page 1, "is a much smaller discriminator set": no, the Lip_1 functions are in general incomparable to arbitrary sets of neural nets.
From here on I'll comment less on typos.
Middle of page 2, point (i): this is the only place it is argued/asserted that the discriminator set should contain essentially everything?  I think this needs a much more serious justification.
Section 1.1: Lebegure -> Lebesgue.
Page 4, vicinity of equation 5: there should really be a mention that none of these universal approximation results give a meaningful bound on the size of the network (the bound given by Barron's work, while nice, is still massive).
Start of section 3.  To be clear, while one can argue that the Lipschitz-1 constraint has a regularization effect, the reason it was originally imposed is to match the Kantorovich duality for Wasserstein_1.  Moreover I'll say this is another instance of the paper treating the discriminator set as irrelevant other than how close it is to being dense in all bounded measurable functions.


GANITE: ESTIMATION OF INDIVIDUALIZED TREAT- MENT EFFECTS USING GENERATIVE ADVERSARIAL NETS
Estimating individualized treatment effects (ITE) is a challenging task due to the need for an individual's potential outcomes to be learned from biased data and without having access to the counterfactuals. We propose a novel method for inferring ITE based on the Generative Adversarial Nets (GANs) framework. Our method, termed Generative Adversarial Nets for inference of Individualized Treatment Effects (GANITE), is motivated by the possibility that we can capture the uncertainty in the counterfactual distributions by attempting to learn them using a GAN. We generate proxies of the counterfactual outcomes using a counterfactual generator, G, and then pass these proxies to an ITE generator, I, in order to train it. By modeling both of these using the GAN framework, we are able to infer based on the factual data, while still accounting for the unseen counterfactuals. We test our method on three real-world datasets (with both binary and multiple treatments) and show that GANITE outperforms state-of-the-art methods.

This paper presents GANITE, a Generative Adversarial Network (GAN) approach for estimating Individualized Treatment Effects (ITE). This is achieved by utilising a GAN to impute the `missing` counterfactuals, i.e. the  outcomes of the treatments that were not observed in the training (i.e. factual) sample, and then using another GAN to estimate the ITE based on this `complete` dataset. The authors then proceed in combining the two GAN objectives with extra supervised losses to better account for the observed data; the GAN loss for the `G` network has an extra term for the `G` network to better predict the factual outcome `y_f` (which should be easy to do given the fact that y_f is an input to the network) and the GAN loss for the `I` network has an extra term w.r.t. the corresponding performance metric used for evaluation, i.e. PEHE for binary treatment and MSE for multiple treatments. This model is then evaluated on extensive experiments.
The paper is reasonably well-written with clear background and diagrams for the overall architecture. The idea is novel and seems to be relatively effective in practice although I do believe that it has a lot of moving parts and introduces a considerable amount of hyperameters (which generally are problematic to tune in causal inference tasks). Other than that, I have the following questions and remarks:
- I might have misunderstood the motivation but the GAN objective for the `G` network is a bit weird; why is it a good idea to push the counterfactual outcomes close to the factual outcomes (which is what the GAN objective is aiming for)? Intuitively, I would expect that different treatments should have different outcomes and the distribution of the factual and counterfactual `y` should differ.
- According to which metric did you perform hyper-parameter optimization on all of the experiments?
- From the first toy experiment that highlights the importance of each of the losses it seems that the addition of the supervised loss greatly boosts the performance, compared to just using the GAN objectives. What was the relative weighting on those losses in general?
- From what I understand the `I` network is necessary for out-of-sample predictions where you don’t have the treatment assignment, but for within sample prediction you can also use the `G` network. What is the performance gap between the `I` and `G` networks on the within-sample set? Furthermore, have you experimented with constructing `G` in a way that can represent `I` by just zeroing the contribution of `y_f` and `t`? In this way you can tie the parameters and avoid the two-step process (since `G` and `I` represent similar things).
- For figure 2 what was the hyper parameters for CFR? CFR includes a specific knob to account for the larger mismatches between treated and control distributions. Did you do hyper-parameter tuning for all of the methods in this task?
- I would also suggest to not use “between” when referring to the KL-divergence as it is not a symmetric quantity.
Also it should be pointed out that for IHDP the standard evaluation protocol is 1000 replications (rather than 100) so there might be some discrepancy on the scores due to that.


This paper proposes a novel approach for learning discriminative and sparse representations. It consists of utilizing two different models. A predefined number of non-linear transform models are used in the learning stage, and one sparsifying transform model is used at test time. The non-linear transform models have discriminative and minimum information loss priors. A novel measure related to the discriminative prior is proposed and defined on the support intersection for the transform representations. The minimum information loss prior is expressed as a constraint on the conditioning and the expected coherence of the transform matrix. An equivalence between the non-linear models and the sparsifying model is shown only when the measure that is used to define the discriminative prior goes to zero. An approximation of the measure used in the discriminative prior is addressed, connecting it to a similarity concentration. To quantify the discriminative properties of the transform representation, we introduce another measure and present its bounds. Reflecting the discriminative quality of the transform representation we name it as discrimination power. To support and validate the theoretical analysis a practical learning algorithm is presented. We evaluate the advantages and the potential of the proposed algorithm by a computer simulation. A favorable performance is shown considering the execution time, the quality of the representation, measured by the discrimination power and the recognition accuracy in comparison with the state-of-the-art methods of the same category.

Summary:
The paper proposes a model to estimate a non-linear transform of data with labels, trying to increase the discriminative power of the transformation while preserving information.
Quality:
The quality is potentially good but I misunderstood too many things (see below) to emit a confident judgement.
Clarity:
Clarity is poor. The paper is overall difficult to follow (at least for me) for different reasons. First, with 17 pages + references + appendix, the length of the paper is way above the « strongly suggested limit of 8 pages ». Second, there are a number of small typos and the citations are not well formatted (use \citep instead of \citet). Third, and more importantly, several concepts are only vaguely formulated (or wrong), and I must say I certainly misunderstood some parts of the manuscript.
A few examples of things that should be clarified:
- p4: « per class c there exists an unknown nonlinear function […] that separates the data samples from different classes in the transformed domain ». What does « separate » mean here? There exists as many functions as there are classes, do you mean that each function separates all classes? Or that when you apply each class function to the elements of its own class, you get a separation between the classes? (which would not be very useful)
- p5: what do you mean exactly by « non-linear thresholding function »?
- p5: « The goal of learning a nonlinear transform (2) is to estimate… »: not sure what you mean by « accurate approximation » in this sentence.
- p5 equation 3: if I understand correctly, you not only want to estimate a single vector of thresholds for all classes, but also want it to be constant. Is there a reason for the second constraint?
- p5 After equation 4, you define different « priors ». The one on z_k seems to be Gaussian; however in equation 4, z_k seems to be the difference between the input and the output of the nonlinear threshold operator. If this is correct, then the nonlinear threshold operator is just the addition of a Gaussian random noise, which is really not a thresholding operator. I suppose I misunderstood something here, some clarification is probably needed.
- p5 before equation 5, you define a conditional distribution of \tau_c given y_{c,k} ; however if you define a prior on \tau_c given each point (i.e., each $k$), how do you define the law of \tau_c given all points?
- p5 Equation 6: I dont understand what the word « approximation » refers to, and how equation 6 is derived.
-p6 equation 7 : missing exponent in the Gaussian distribution
- p6-7: to combine equations 7-8-9 and obtain 10, I suppose in equation 7 the distributions should be conditioned on A (at least the first one), and in equation 9 I suppose the second line should be removed and the third line should be conditioned on A; otherwise more explanations are needed.
- Lemma 1 is just unreadable. Please at least split equations in several lines.
Originality:
As far as I can tell, the approach is quite original and the results proved are new.
Significance:
The method provides a new way to learn discriminative features; as such it is a variant of several existing methods, which could have some impact if clarity is improved and a public code is provided.


The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements. Motivated by the potential for an inthe-field cell sorting detector, we examine a Synechocystis sp. PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures. We use deep learning techniques to both successfully classify cells and generate a mask segmenting the cells/condition from the background. Further, we use the classification accuracy to guide a data-driven, iterative feature selection method, allowing the design neural networks requiring 90% fewer input features with little accuracy degradation.

This paper explores the use of neural networks for classification and segmentation of hypersepctral imaging (HSI) of cells. The basic set-up of the method and results seem correct and will be useful to the specific application described. While the narrow scope might limit this work's significance, my main issue with the paper is that while the authors describe prior art in terms of HSI for biological preps, there is a very rich literature addressing HSI images in other domains: in particular for remote sensing. I think that this work can (1) be a lot clearer as to the novelty and (2) have a much bigger impact if this literature is addressed. In particular, there are a number of methods of supervised and unsupervised feature extraction used for classification purposes (e.g. endmember extraction or dictionary learning). It would be great to know how the features extracted from the neural network compare to these methods, as well as how the classification performance compares to typical methods, such as performing SVM classification in the feature space. The comparison with the random forests is nice, but that is not a standard method. Putting the presented work in context for these other methods would help make there results more general, and hopefully increase the applicability to more general HSI data (thus increasing significance).
An additional place where this comparison to the larger HSI literature would be useful is in the section where the authors describe the use of the network weights to isolate sub-bands that are more informative than others. Given the high correlation in many spectra, typically something like random sampling might be sufficient (as in compressive sensing). This type of compression which can be applied at the sensor -- a benefit the authors mention of their band-wise sub-sampling. It would be good to acknowledge this prior work and to understand if the features from the network are superior to the random sampling scheme.
For these comments, I suggest the authors look at the following papers (and especially the references therein):
[1] Li, Chengbo, et al. "A compressive sensing and unmixing scheme for hyperspectral data processing." IEEE Transactions on Image Processing 21.3 (2012): 1200-1210.
[2] Bioucas-Dias, José M., et al. "Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches." IEEE journal of selected topics in applied earth observations and remote sensing 5.2 (2012): 354-379.
[3] Charles, Adam S., Bruno A. Olshausen, and Christopher J. Rozell. "Learning sparse codes for hyperspectral imagery." IEEE Journal of Selected Topics in Signal Processing 5.5 (2011): 963-978.


Published as a conference paper at ICLR 2018 EMERGENT TRANSLATION IN MULTI-AGENT COMMUNICATION
While most machine translation systems to date are trained on large parallel corpora, humans learn language in a different way: by being grounded in an environment and interacting with other humans. In this work, we propose a communication game where two agents, native speakers of their own respective languages, jointly learn to solve a visual referential task. We find that the ability to understand and translate a foreign language emerges as a means to achieve shared goals. The emergent translation is interactive and multimodal, and crucially does not require parallel corpora, but only monolingual, independent text and corresponding images. Our proposed translation model achieves this by grounding the source and target languages into a shared visual modality, and outperforms several baselines on both word-level and sentence-level translation tasks. Furthermore, we show that agents in a multilingual community learn to translate better and faster than in a bilingual communication setting.

Summary: The authors show that using visual modality as a pivot they can train a model to translate from L1 to L2.
Please find my detailed comments/questions/suggestions below:
1) IMO, the paper could have been written much better. At the core, this is simply a model which uses images as a pivot for learning to translate between L1 and L2 by learning a common representation space for {L1, image} or {L2, image}. There are several works on such multimodal representation learning but the authors present their work in a way which makes it look very different from these works. IMO, this leads to unnecessary confusion and does more harm than good. For example, the abstract gives an impression that the authors have designed a game to collect data (and it took me a while to set this confusion aside).
2) Continuing on the above point, this is essentially about learning a common multimodal representation and then decode from this common representation. However, the authors do not cite enough work on such multimodal representation learning (for example, look at Spandana et. al.: Image Pivoting for Learning Multilingual Multimodal Representations, EMNLP 2017 for a good set of references)
3) This omission of related work also weakens the experimental section. At least for the word translation task many of these common representation learning frameworks could have been easily evaluated. For example, find the nearest german neighbour of the word "dog" in the common representation space. The authors instead compare with very simple baselines.
4) Even when comparing with simple baselines, the proposed model does not convincingly outperform them. In particular,  the P@5 and P@20 numbers are only slightly better.
5) Some of the choices made in the Experimental setup seem questionable to me:
   - Why  use a NMT model without attention? That is not standard and does not make sense to use when a better baseline model (with attention) is available ?
   - It is mentioned that "While their model unit-normalizes the output of every encoder, we found this to consistently hurt performance, so do not use normalization for fair comparison with our models." I don't think this is a fair comparison. The authors can mention their results without normalization if that works well for them but it is not fair to drop normalization from the model of N&N if that gives better performance. Please mention the numbers with unit normalization to give a better picture. It does not make sense to weaken an existing baseline and then compare with it.
6) It would be good to mention the results of the NMT model in Table 1 itself instead of mentioning them separately in a paragraph. This again leads to poor readability and it is hard to read and compare the corresponding numbers from Table 1.  I am not sure why this cannot be accommodated in the Table itself.
7) In Figure 2, what exactly do you mean by "Results are averaged over 30 translation scenarios". Can you please elaborate ?


Despite many second-order methods have been proposed to train neural networks, most of the results were done on smaller single layer fully connected networks, so we still cannot conclude whether it's useful in training deep convolutional networks. In this study, we conduct extensive experiments to answer the question "whether second-order method is useful for deep learning?". In our analysis, we find out although currently second-order methods are too slow to be applied in practice, it can reduce training loss in fewer number of iterations compared with SGD. In addition, we have the following interesting findings: (1) When using a large batch size, inexact-Newton methods will converge much faster than SGD. Therefore inexact-Newton method could be a better choice in distributed training of deep networks. (2) Quasi-newton methods are competitive with SGD even when using ReLu activation function (which has no curvature) on residual networks. However, current methods are too sensitive to parameters and not easy to tune for different settings. Therefore, quasi-newton methods with more selfadjusting mechanisms might be more useful than SGD in training deeper networks.

The paper conducts an empirical study on 2nd-order algorithms for deep learning, in particular on CNNs to answer the question whether 2nd-order methods are useful for deep learning.  More modestly and realistically, the authors compared stochastic Newton method (SHG) and stochastic Quasi- Newton method (SR1, SQN) with stochastic gradient method (SGD).  The activation function ReLu is known to be singular at 0, which may lead to poor curvature information, but the authors gave a good numerical comparison between the performances of 2nd-order methods with ReLu and the smooth function, Tanh.  The paper presented a reasonably good overview of existing 2nd-order methods, with clear numerical examples and reasonably well written.
The paper presents several interesting empirical findings, which will no doubt lead to follow up work. However, there are also a few critical issues that may undermine their claims, and that need to be addressed before we can really answer the original question of whether 2nd-order methods are useful for deep learning.
1. There is no complexity comparison, e.g. what is the complexity for a single step of different method.
2. Relatedly, the paper reports the performance over epochs, but it is not clear what "per epoch" means for 2nd-order methods.  In particular, it seems to me that they did not count the inner CG iterations, and it is known that this is crucial in running time and important for quality.  If so, then the comparison between 1st-order and 2nd-order methods are not fair or incomplete.
3. The results on 2nd-order methods behave similarly to 1st-order methods, which makes me wonder how many CG iterations they used for 2nd-order method in their experiment, and also the details of the data.  In particular, are they looking at parameter/hyperparameter settings for which 2nd-order methods aren't really necessary.
4. In deep learning setting, the training objective is non-convex, which means the Hessian can be non-PSD.  It is not clear how the stochastic inexact-Newton method mentioned in Section 2.1 could work.  Details on implementations of 2nd-order methods are important here.
5. For 2nd-order methods, the author used line search to tune the step size.  It is not clear in the line search, the author used the whole training objective or batch loss.  Assuming using the batch loss, I suspect the training curve will be very noisy (depending on how large the batch size is).  But the paper only show the average training curves, which might be misleading.
Here are other points.
1. There is no figure showing training/ test accuracy.  Aside from being interested in test error, it is also of interest to see how 2nd order methods are similar/different than 1st order methods on training versus test.
2. Since it is a comparison paper, it only compares three 2nd-order methods with SGD.  The choices made were reasonable, but 2nd-order methods are not as trivial to implement as SGD, and it isn't clear whether they have really "spanned the space" of second order methods
3. In the paper, the settings of LeNet, AlexNet are different with those in the original paper.  The authors did not give a reason.
4. The quality of figures is not good.
5. The setting of optimization is not clear, e.g. the learning rate of SGD, the parameter of backtrack line search.  It's hard to reproduce results when these are not described.


Many state-of-the-art word embedding techniques involve factorization of a cooccurrence based matrix. We aim to extend this approach by studying word embedding techniques that involve factorization of co-occurrence based tensors (Nway arrays). We present two new word embedding techniques based on tensor factorization, and show that they outperform multiple commonly used embedding methods when used for various NLP tasks on the same training data. Also, to train one of the embeddings, we present a new joint tensor factorization problem and an approach for solving it. Furthermore, we modify the performance metrics for the Outlier Detection task (Camacho-Collados & Navigli (2016)) to measure the quality of higher-order relationships that a word embedding captures. Our tensor-based methods significantly outperform existing methods at this task when using our new metric. Finally, we demonstrate that vectors in our embeddings can be composed multiplicatively to create different vector representations for each meaning of a polysemous word in a way that cannot be done with other common embeddings. We show that this property stems from the higher order information that the vectors contain, and thus is unique to our tensor based embeddings.

The paper presents the word embedding technique which consists of: (a) construction of a positive (i.e. with truncated negative values) pointwise mutual information order-3 tensor for triples of words in a sentence and (b) symmetric tensor CP factorization of this tensor. The authors propose the CP-S (stands for symmetric CP decomposition) approach which tackles such factorization in a "batch" manner by considering small random subsets of the original tensor. They also consider the JCP-S approach, where the ALS (alternating least squares) objective is represented as the joint objective of the matrix and order-3 tensor ALS objectives. The approach is evaluated experimentally on several tasks such as outlier detection, supervised analogy recovery, and sentiment analysis tasks.
CLARITY: The paper is very well written and is easy to follow. However, some implementation details are missing, which makes it difficult to assess the quality of the experimental results.
QUALITY: I understand that the main emphasis of this work is on developing faster computational algorithms, which would handle large scale problems, for factorizing this tensor. However, I have several concerns about the algorithms proposed in this paper:
  - First of all, I do not see why using small random subsets of the original tensor would give a desirable factorization. Indeed, a CP decomposition of a tensor can not be reconstructed from CP decompositions of its subtensors. Note that there is a difference between batch methods in stochastic optimization where batches are composed of a subset of observations (which then leads to an approximation of desirable quantities, e.g. the gradient, in expectation) and the current approach where subtensors are considered as batches. I would expect some further elaboration of this question in the paper. Although similar methods appeared in the tensor literature before, I don't see any theoretical ground for their correctness.
  - Second, there is a significant difference between the symmetric CP tensor decomposition and the non-negative symmetric CP tensor decomposition. In particular, the latter problem is well posed and has good properties (see, e.g., Lim, Comon. Nonengative approximations of nonnegative tensors (2009)). However, this is not the case for the former (see, e.g., Comon et al., 2008 as cited in this paper). Therefore, (a) computing the symmetric and not non-negative symmetric decomposition does not give any good theoretical guarantees (while achieving such guarantees seems to be one of the motivations of this paper) and (b) although the tensor is non-negative, its symmetric factorization is not guaranteed to be non-negative and further elaboration of this issue seem to be important to me.
  - Third, the authors claim that one of their goals is an experimental exploration of tensor factorization approaches with provable guarantees applied to the word embedding problem. This is an important question that has not been addressed in the literature and is clearly a pro of the paper. However, it seems to me that this goal is not fully implemented. Indeed, (a) I mentioned in the previous paragraph the issues with the symmetric CP decomposition and (b) although the paper is motivated by the recent algorithm proposed by Sharan&Valiant (2017), the algorithms proposed in this paper are not based on this or other known algorithms with theoretical guarantees. This is therefore confusing and I would be interested in the author's point of view to this issue.
  - Further, the proposed joint approach, where the second and third order information are combined requires further analysis. Indeed, in the current formulation the objective is completely dominated by the order-3 tensor factor, because it contributes O(d^3) terms to the objective vs O(d^2) terms contributed by the matrix part. It would be interesting to see further elaboration of the pros and cons of such problem formulation.
  - Minor comment. In the shifted PMI section, the authors mention the parameter alpha and set specific values of this parameter based on experiments. However, I don't think that enough information is provided, because, given the author's approach, the value of this parameter most probably depends on other parameters, such as the bach size.
  - Finally, although the empirical evaluation is quite extensive and outperforms the state-of the art, I think it would be important to compare the proposed algorithm to other tensor factorization approaches mentioned above.
ORIGINALITY: The idea of using a pointwise mutual information tensor for word embeddings is not new, but the authors fairly cite all the relevant literature. My understanding is that the main novelty is the proposed tensor factorization algorithm and extensive experimental evaluation. However, such batch approaches for tensor factorization are not new and I am quite skeptical about their correctness (see above). The experimental evaluation presents indeed interesting results. However, I think it would also be important to compare to other tensor factorization approaches. I would also be quite interested to see the performance of the proposed algorithm for different values of parameters (such as the butch size).
SIGNIFICANCE: I think the paper addresses very interesting problem and significant amount of work is done towards the evaluation, but there are some further important questions that should be answered before the paper can be published.  To summarize, the following are the pros of the paper:
  - clarity and good presentation;
  - good overview of the related literature;
  - extensive experimental comparison and good experimental results.
While the following are the cons:
  - the mentioned issues with the proposed algorithm, which in particular does not have any theoretical guarantees;
  - lack of details on how experimental results were obtained, in particular, lack of the details on the values of the free parameters in the proposed algorithm;
  - lack of comparison to other tensor approaches to the word embedding problem (i.e. other algorithms for the tensor decomposition subproblem);
  - the novelty of the approach is somewhat limited, although the idea of the extensive experimental comparison is good.


Published as a conference paper at ICLR 2018 BEYOND SHARED HIERARCHIES: DEEP MULTITASK LEARNING THROUGH SOFT LAYER ORDERING
Existing deep multitask learning (MTL) approaches align layers shared between tasks in a parallel ordering. Such an organization significantly constricts the types of shared structure that can be learned. The necessity of parallel ordering for deep MTL is first tested by comparing it with permuted ordering of shared layers. The results indicate that a flexible ordering can enable more effective sharing, thus motivating the development of a soft ordering approach, which learns how shared layers are applied in different ways for different tasks. Deep MTL with soft ordering outperforms parallel ordering methods across a series of domains. These results suggest that the power of deep MTL comes from learning highly general building blocks that can be assembled to meet the demands of each task.

Summary: This paper proposes a different approach to deep multi-task learning using “soft ordering.”  Multi-task learning encourages the sharing of learned representations across tasks, thus using less parameters and tasks help transfer useful knowledge across. Thus enabling the reuse of universally learned representations and reuse them by assembling them in novel ways for new unseen tasks. The idea of “soft ordering” enforces the idea that there shall not be a rigid structure for all the tasks, but a soft structure would make the models more generalizable and modular.
The methods reviewed prior work which the authors refer to as “parallel order”, which assumed that subsequences of the feature hierarchy align across tasks and sharing between tasks occurs only at aligned depths whereas in this work the authors argue that this shouldn’t be the case. They authors then extend the approach to “permuted order” and finally present their proposed “soft ordering” approach. The authors argue that their proposed soft ordering approach increase the expressivity of the model while preserving the performance.
The “soft ordering” approach simply enable task specific selection of layers, scaled with a learned scaling factor, to be combined in which order to result for the best performance for each task. The authors evaluate their approach on MNIST, UCI, Omniglot and CelebA datasets and compare their approach to “parallel ordering” and “permuted ordering” and show the performance gain.
Positives:
- The paper is clearly written and easy to follow
- The idea is novel and impactful if its evaluated properly and consistently
- The authors did a great job summarizing prior work and motivating their approach
Negatives:
- Multi-class classification problem is one incarnation of Multi-Task Learning, there are other problems where the tasks are different (classification and localization) or auxiliary (depth detection for navigation). CelebA dataset could have been a good platform for testing different tasks, attribute classification and landmark detection.

(TODO) I would recommend that the authors test their approach on such setting.
- Figure 6 is a bit confusing, the authors do not explain why the “Permuted Order” performs worse than “Parallel Order”. Their assumptions and results as of this section should be consistent that soft order>permuted order>parallel order>single task.

(TODO) I would suggest that the authors follow up on this result, which would be beneficial for the reader.
- Figure 4(a) and 5(b), the results shown on validation loss, how about testing error similar to Figure 6(a)? How about results for CelebA dataset, it could be useful to visualize them as was done for MNIST, Omniglot and UCL.

(TODO) I would suggest that the authors make the results consistent across all datasets and use the same metric such that its easy to compare.
Notation and Typos:
- Figure 2 is a bit confusing, how come the accuracy decreases with increasing number of training samples? Please clarify.
1- If I assume that the Y-Axis is incorrectly labeled and it is Training Error instead, then the permuted order is doing worse than the parallel order.

2- If I assume that the X-Axis is incorrectly labeled and the numbering is reversed (start from max and ending at 0), then I think it would make sense.
- Figure 4 is very small and not easy to read the text. Does single task mean average performance over the tasks?
- In eq.(3) Choosing \sigma_i for a task-specific permutation of the network is a bit confusing, since it could be thought of as a sigmoid function, I suggest using a different symbol.

Conclusion: I would suggest that the authors address the concerns mentioned above. Their approach and idea is very interesting and relevant, and addressing these suggestions will make the paper strong for publication.


PROGRESSIVE GROWING OF GANS FOR IMPROVED QUALITY, STABILITY, AND VARIATION
We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CELEBA images at 1024 2 . We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CELEBA dataset.

This paper proposes a number of ideas for improving GANs for image generation, highlighting in particular a curriculum learning strategy to progressively increase the resolution of the generated images, resulting in GAN generators capable of producing samples with unprecedented resolution and visual fidelity.
Pros:
The paper is well-written and the results speak for themselves! Qualitatively they’re an impressive and significant improvement over previous results from GANs and other generative models.  The latent space interpolations shown in the video (especially on CelebA-HQ) demonstrate that the generator can smoothly transition between modes and convince me that it isn’t simply memorizing the training data. (Though I think this issue could be addressed a bit better -- see below.) Though quantification of GAN performance is difficult and rapidly evolving, there is a lot of quantitative analysis all pointing to significant improvements over previous methods.
A number of new tricks are proposed, with the ablation study (tab 1 + fig 3) and learning curves (fig 4) giving insight into their effects on performance.  Though the field is moving quickly, I expect that several of these tricks will be broadly adopted in future work at least in the short to medium term.
The training code and data are released.
Cons/Suggestions:
It would be nice to see overfitting addressed and quantified in some way.  For example, the proposed SWD metric could be recomputed both for the training and for a held-out validation/test set, with the difference between the two scores measuring the degree of overfitting.  Similarly, Danihelka et al. [1] show that an independently trained Wasserstein critic (with one critic trained on G samples vs. train samples, and another trained on G samples vs. val samples) can be used to measure overfitting.  Another way to go could be to generate a large number of samples and show the nearest neighbor for a few training set samples and for a few val set samples.  Doing this in pixel space may not work well especially at the higher resolutions, but maybe a distance function in the space of some high-level hidden layer of a trained discriminator could show good semantic nearest neighbors.
The proposed SWD metric is interesting and computationally convenient, but it’s not clear to me that it’s an improvement over previous metrics like the independent Wasserstein critic proposed in [1].  In particular the use of 7x7 patches would seem to limit the metric’s ability to capture the extent to which global structure has been learned, even though the patches are extracted at multiple levels of the Laplacian pyramid.
The ablation study (tab 1 + fig 3) leaves me somewhat unsure which tricks contribute the most to the final performance improvement over previous work.  Visually, the biggest individual improvement is easily when going from (c) to (d), which adds the “Revised training parameters”, with the improvement from (a) to (b) which adds the highlighted progressive training schedule appearing relatively minor in comparison.  However, I realize the former large improvement is due to the arbitrary ordering of the additions in the ablation study, with the small minibatch addition in (c) crippling results on its own.  Ablation studies with large numbers of tweaks are always difficult and this one is welcome and useful despite the ambiguity.
On a related note, it would be nice if there were more details on the “revised training hyperparameters” improvement ((d) in the ablation study) -- which training hyperparameters are adjusted, and how?
“LAPGAN” (Denton et al., 2015) should be cited as related work: LAPGAN’s idea of using a separate generator/discriminator at each level of a Laplacian pyramid conditioned on the previous level is quite relevant to the progressive training idea proposed here.  Currently the paper is only incorrectly cited as “DCGAN” in a results table -- this should be fixed as well.
Overall, this is a well-written paper with striking results and a solid effort to analyze, ablate, and quantify the effect of each of the many new techniques proposed. It’s likely that the paper will have a lot of impact on future GAN work.
[1] Danihelka et al., “Comparison of Maximum Likelihood and GAN-based training of Real NVPs” https://arxiv.org/abs/1705.05263


Published as a conference paper at ICLR 2018 THERMOMETER ENCODING: ONE HOT WAY TO RESIST ADVERSARIAL EXAMPLES
It is well known that it is possible to construct "adversarial examples" for neural networks: inputs which are misclassified by the network yet indistinguishable from true data. We propose a simple modification to standard neural network architectures, thermometer encoding, which significantly increases the robustness of the network to adversarial examples. We demonstrate this robustness with experiments on the MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, and show that models with thermometer-encoded inputs consistently have higher accuracy on adversarial examples, without decreasing generalization. State-of-the-art accuracy under the strongest known white-box attack was increased from 93.20% to 94.30% on MNIST and 50.00% to 79.16% on CIFAR-10. We explore the properties of these networks, providing evidence that thermometer encodings help neural networks to find more-non-linear decision boundaries.

The authors present an in-depth study of discretizing / quantizing the input as a defense against adversarial examples. The idea is that the threshold effects of discretization make it harder to find adversarial examples that only make small alterations of the image, but also that it introduces more non-linearities, which might increase robustness. In addition, discretization has little negative impact on the performance on clean data. The authors also propose a version of single-step or multi-step attacks against models that use discretized inputs, and present extensive experiments on MNIST, CIFAR-10, CIFAR-100 and SVHN, against standard baselines and, on MNIST and CIFAR-10, against a version of quantization in which the values are represented by a small number of bits.
The merits of the paper is that the study is rather comprehensive: a large number of datasets were used, two types of discretization were tried, and the authors propose an attack mechanism better that seems reasonable considering the defense they consider. The two main claims of the paper, namely that discretization doesn't hurt performance on natural test examples and that better robustness (in the author's experimental setup) is achieved through the discretized encoding, are properly backed up by the experiments.
Yet, the applicability of the method in practice is still to be demonstrated. The threshold effects might imply that small perturbations of the input (in the l_infty sense) will not have a large effect on their discritized version, but it may also go the other way: an opponent might be able to greatly change the discretized input without drastically changing the input. Figure 8 in the appendix is a bit worrysome on that point, as the performance of the discretized version drops rapidly to 0 when the opponents gets a bit stronger. Did the authors observe the same kind of bahavior on other datasets? What would the authors propose to mitigate this issue? To what extend the good results that are exhibited in the paper are valid over the wide range of opponent's strengths?
minor comment:
- the experiments on CIFAR-100 in Appendix E are carried out by mixing adversarial / clean examples while training, whereas those on SVHN in Appendix F use adversarial examples only.


Published as a conference paper at ICLR 2018 BEYOND WORD IMPORTANCE: CONTEXTUAL DE- COMPOSITION TO EXTRACT INTERACTIONS FROM LSTMS
The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.

In this paper, the authors propose a new LSTM variant that allows to produce interpretations by capturing the contributions of the words to the final prediction and the way their learned representations are combined in order to yield that prediction. They propose a new approach that they call Contextual Decomposition (CD). Their approach consists of disambiguating interaction between LSTM’s gates where gates are linearized so the products between them is over linear sums of contributions from different factors. The hidden and cell states are also decomposed in terms of contributions to the “phrase” in question and contributions from elements outside of the phrase. The motivation of the proposed decomposition using LSTMs is that the latter are powerful at capturing complex non-linear interactions, so, it would be useful to observe how these interactions are handled and to interpret the LSTM’s predictions. As the authors intention is to build a way of interpreting LSTMs output and not to increase the model’s accuracy, the empirical results illustrate the ability of their decomposition of giving a plausible interpretation to the elements of a sentence. They compare their method with different existing method by illustrating samples from the Yelp Polarity and SST datasets. They also show the ability of separating the distribution of CD scores related to positive and negative phrases on respectively Yelp Polarity and SST.
The proposed approach is potentially of great benefit as it is simple and elegant and could lead to new methods in the same direction of research. The sample illustrations, the scatter plots and the CD score distributions are helpful to asses the benefit of the proposed approach.
The writing could be improved as it contains parts where it leads to confusion. The details related to the linearization (section 3.2.2), the training (4.1) could be improved. In equation 25, it is not clear what  π_{i}^(-1) and x_{π_{i}} represent but the example in equation 26 makes it clearer. The section would be clearer if each index and notation used is explained explicitly.
(CD in known for Contrastive Divergence in the deep learning community. It would be better if Contextual Decomposition is not referred by CD.)
Training details are given in section 4.1 where the authors mention the integrated gradient baseline without mentioning the reference paper to it (however they do mention the reference paper at each of table 1 and 2). it would be clearer for the reader if the references are also mentioned in section 4.1 where integrated gradient is introduced. Along with the reference, a brief description of that baseline could be given.
The “Leave one out” baseline is never mentioned in text before section 4.4 (and tables 1 and 2). Neither the reference nor the description of the baseline are given. It would have been clearer to the reader if this had been the case.
Overall, the paper contribution is of great benefit. The quality of the paper could be improved if the above explanations and details are given explicitly in the text.


Generative Adversarial Networks (GANs) have achieved remarkable results in the task of generating realistic natural images. In most applications, GAN models share two aspects in common. On the one hand, GANs training involves solving a challenging saddle point optimization problem, interpreted as an adversarial game between a generator and a discriminator functions. On the other hand, the generator and the discriminator are parametrized in terms of deep convolutional neural networks. The goal of this paper is to disentangle the contribution of these two factors to the success of GANs. In particular, we introduce Generative Latent Optimization (GLO), a framework to train deep convolutional generators without using discriminators, thus avoiding the instability of adversarial optimization problems. Throughout a variety of experiments, we show that GLO enjoys many of the desirable properties of GANs: learning from large data, synthesizing visually-appealing samples, interpolating meaningfully between samples, and performing linear arithmetic with noise vectors.

Summary: The authors observe that the success of GANs can be attributed to two factors; leveraging the inductive bias of deep CNNs and the adversarial training protocol. In order to disentangle the factors of success, and they propose to eliminate the adversarial training protocol while maintaining the first factor. The proposed Generative Latent Optimization (GLO) model maps a learnable noise vector to the real images of the dataset by minimizing a reconstruction loss. The experiments are conducted on CelebA and LSUN-Bedroom datasets.
Strengths:
The paper is well written and the topic is relevant for the community.
The notations are clear, as far as I can tell, there are no technical errors.
The design choices are well motivated in Chapter 2 which makes the main idea easy to grasp.
The image reconstruction results are good.
The experiments are conducted on two challenging datasets, i.e. CelebA and LSUN-Bedroom.
Weaknesses:
A relevant model is Generative Moment Matching Network (GMMN) which can also be thought of as a “discriminator-less GAN”. However, the paper does not contrast GLO with GMMN either in the conceptual level or experimentally.
Another relevant model is Variational Autoencoders (VAE) which also learns the data distribution through a learnable latent representation by minimizing a reconstruction loss. The paper would be more convincing if it provided a comparison with VAE.
In general, having no comparisons with other models proposed in the literature as improvements over GAN such as ACGAN, InfoGAN, WGAN weakens the experimental section.
The evaluation protocol is quite weak: CelebA images are 128x128 while LSUN images are 64x64. Especially since it is a common practice nowadays to generate much higher dimensional images, i.e. 256x256, the results presented in this paper appear weak.
Although the reconstruction examples (Figure 2 and 3) are good, the image generation results (Figure 4 and 5) are worse than GAN, i.e. the 3rd images in the 2nd row in Figure 4 for instance has unrealistic artifacts, the entire Figure 5 results are quite boxy and unrealistic. The authors mention in Section 3.3.2 that they leave the careful modeling of Z to future work, however the paper is quite incomplete without this.
In Section 3.3.4, the authors claim that the latent space that GLO learns is interpretable. For example, smiling seems correlated with the hair color in Figure 6. This is a strong claim based on one example, moreover the evidence of this claim is not as obvious (based on the figure) to the reader. Moreover, in Figure 8, the authors claim that the principal components of the GLO latent space is interpretable. However it is not clear from this figure what each eigenvector generates. The authors’ observations on Figure 8 and 9 are not clearly visible through manual inspection.
Finally, as a minor note, the paper has some vague statements such as
“A linear interpolation in the noise space will generate a smooth interpolation of visually-appealing images”
“Several works attempt at recovering the latent representation of an image with respect to a generator.”
Therefore, a careful proofreading would improve the exposition.


Mini-batch gradient descent and its variants are commonly used in deep learning. The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient. However, its batch size is a fixed hyper parameter requiring manual setting before training the neural network. Yin et al. (2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds. We extend the BA-SGD to momentum algorithm, and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classification. Experiments confirm that batch adaptive methods can achieve a lower loss compared with mini-batch methods and methods that manually increase the batch size, after scanning the same epochs of data. Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes. We also identified an interesting phenomenon, batch size boom. The code implementing the batch adaptive framework is now open source, applicable to any gradient-based optimization problems.

The paper proposes a generalization of an algorithm by Yin et al. (2017), which performs SGD with adaptive batch sizes. The present paper generalizes the algorithm to SGD with momentum. Since the original algorithm was already formulated with a general utility function, the proposed algorithm is similar in structure but replaces the utility function so that it takes momentum into account. Experiments on an image classification task show improvements in the training loss. However, no test accuracies are reported and the learning curves have suspicious artifacts, see below. Experiments on a relation extraction task show little improvement over SGD with momentum and constant batch size.
COMMENTS:
The paper discusses a relevant issue. While adaptive learning algorithms are popular in deep learning, most algorithms adapt the learning rate or the momentum coefficient, but not the batch size. It appears to me that the main idea and the overall structure of the proposed algorithm is the same as in the one published by Yin et al. (2017), and that only few changes were necessary to include momentum. Given the incremental process, I find the presentation unnecessarily involved, and experiments not convincing enough.
Concerning the presentation, the paper dedicates two full pages on a review of the algorithm by Yin et al. (2017). The first page of this review states that, for large enough batch sizes, the change of the objective function in SGD is normal distributed with a variance that is inversely proportional the batch size. It seems to me that this is a direct consequence of the central limit theorem. The derivation, however, is quite technical and introduces some quantities that are never used (e.g., $\vec{\xi}_j$ is never used individually, only the combined term $\epsilon_t$ defined below Eq. 12 is). The second page of the review seems to discuss the main part of the algorithm, but I could not follow it. First, a "state" $s_t$ (also written as $S$) is introduced, which, according to the text, is "the objective value", which was earlier denoted by $F$. Nevertheless, the change of $s_t$, Eq. 5, appears to obey a different probability distribution than the change of $F$. The paper provides a verbal explanation for this discrepancy, saying that it is possible that $S$ is first reduced to the minimum $S^*$ of the objective and then increased again. However, in my understanding, the minimum of the objective is only realized at a singular point in parameter space. Crossing this point in an update step should have zero probability as long as the model has more than one parameter. The explanation also does not make it clear why the argument should apply to $S$ (or $s$) but not to $F$.
Page 5 provides pseudocode for the proposed algorithm. However, I couldn't find an explanation of the code. The code suggests that, for each update step, one gradually increases the batch size until it becomes larger or equal than a running estimate of the optimal batch size. While this may be a plausible strategy in practice, it seems to have a bias that is not addressed in the paper: the algorithm recalculates a noisy estimate of the optimal batch size after each increase of the batch size, and it terminates as soon as the noisy estimate happens to be small enough, resulting in a bias towards a smaller than optimal batch size. A probably more important issue is that the algorithm is sequential and hard to parallelize, where parallelization is usually the main motivation to use larger batch sizes. As the gradient noise scales inversely proportional to the batch size, I don't see why increasing the batch size should be preferred over decreasing the learning rate unless optimizations with a larger batch size can be parallelized. The experiments don't compare the two alternatives.
Concerning the experiments, it seems peculiar that the learning curves in Figure 1 remain at a constant value for a long time at the beginning of the optimization before they begin to drop. Do the authors understand this behavior? It could indicate that the magnitude of the random initialization was chosen too small. I.e., the parameters might have been initialized too close to zero, where the loss is stationary due to symmetries. Also, absolute values of the training loss can be deceptive since there is often no natural scale. A better indicator of convergence would be the test accuracy. The identification of the "batch size boom" is interesting.