Published as a conference paper at ICLR 2018 ASK THE RIGHT QUESTIONS: ACTIVE QUESTION REFORMULATION WITH REINFORCEMENT LEARNING
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a stateof-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

This paper formulates the Jeopardy QA as a query reformulation task that leverages a search engine.  In particular, a user will try a sequence of alternative queries based on the original question in order to find the answer.  The RL formulation essentially tries to mimic this process.  Although this is an interesting formulation, as promoted by some recent work, this paper does not provide compelling reasons why it's a good formulation.  The lack of serious comparisons to baseline methods makes it hard to judge the value of this work.
Detailed comments/questions:
	1. I am actually quite confused on why it's a good RL setting. For a human user, having a series of queries to search for the right answer is a natural process, but it's not natural for a computer program.  For instance, each query can be viewed as different formulation of the same question and can be issued concurrently. Although formulated as an RL problem,  it is not clear to me whether the search result after each episode has been used as the immediate environment feedback. As a result, the dependency between actions seems rather weak.
	2. I also feel that the comparisons to other baselines (not just the variation of the proposed system) are not entirely fair. For instance, the baseline BiDAF model has only one shot, namely using the original question as query.  In this case, AQA should be allowed to use the same budget -- only one query.  Another more realistic baseline is to follow the existing work on query formulation in the IR community.  For example, 20 shorter queries generated by methods like [1] can be used to compare the queries created by AQA.
[1] Kumaran & Carvalho. "Reducing Long Queries Using Query Quality Predictors". SIGIR-09

Pros:
	1. An interesting RL formulation for query reformulation
Cons:
	1. The use of RL is not properly justified
	2. The empirical result is not convincing that the proposed method is indeed advantageous
---------------------------------------
After reading the author response and checking the revised paper, I'm both delighted and surprised that the authors improved the submission substantially and presented stronger results.  I believe the updated version has reached the bar and recommend accepting this paper.


Published as a conference paper at ICLR 2018 CERTIFYING SOME DISTRIBUTIONAL ROBUSTNESS WITH PRINCIPLED ADVERSARIAL TRAINING
Neural networks are vulnerable to adversarial examples and researchers have proposed many heuristic attack and defense mechanisms. We address this problem through the principled lens of distributionally robust optimization, which guarantees performance under adversarial input perturbations. By considering a Lagrangian penalty formulation of perturbing the underlying data distribution in a Wasserstein ball, we provide a training procedure that augments model parameter updates with worst-case perturbations of training data. For smooth losses, our procedure provably achieves moderate levels of robustness with little computational or statistical cost relative to empirical risk minimization. Furthermore, our statistical guarantees allow us to efficiently certify robustness for the population loss. For imperceptible perturbations, our method matches or outperforms heuristic approaches.

This paper proposes a principled methodology to induce distributional robustness in trained neural nets with the purpose of mitigating the impact of adversarial examples. The idea is to train the model to perform well not only with respect to the unknown population distribution, but to perform well on the worst-case distribution in some ball around the population distribution. In particular, the authors adopt the Wasserstein distance to define the ambiguity sets. This allows them to use strong duality results from the literature on distributionally robust optimization and express the empirical minimax problem as a regularized ERM with a different cost. The theoretical results in the paper are supported by experiments.
Overall, this is a very well-written paper that creatively combines a number of interesting ideas to address an important problem.


In this paper, we propose the Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input. In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information improves translation performance. However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input. Our approach generates (associates) visual information from input text and generates response text using context vector fusing associative visual information and sentence textual information. A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences. Furthermore, analysis experiment of visual association showed that our proposed model generates (associates) visual information effective for sentence generation.

The paper proposes to augment (traditional) text-based sentence generation/dialogue approaches by incorporating visual information.  The idea is that associating visual information with input text, and using that associated visual information as additional input will produce better output text than using only the original input text.
The basic idea is to collect a bunch of data consisting of both text and associated images or video.  Here, this was done using Japanese news programs.  The text+image/video is used to train a model that requires both as input and that encodes both as context vectors, which are then combined and decoded into output text.  Next, the image inputs are eliminated, with the encoded image context vector being instead associatively predicted directly from the encoded text context vector (why not also use the input text to help predict the visual context?), which is still obtained from the text input, as before.  The result is a model that can make use of the text-visual associations without needing visual stimuli.  This is a nice idea.
Actually, based on the brief discussion in Section 2.2.2, it occurs to me that the model  might not really be learning visual context vectors associatively, or, that this doesn't really have meaning in some sense.  Does it make sense to say that what it is really doing is just learning to associate other concepts/words with the input text, and that it is using the augmenting visual information in the training data to provide those associations?  Is this worth talking about?
Unfortunately, while the idea has merit, and I'd like to see it pursued, the paper suffers from a fatal lack of validation/evaluation, which is very curious, given the amount of data that was collected, the fact that the authors have both a training and a test set, and that there are several natural ways such an evaluation might be performed.  The two examples of Fig 3 and the additional four examples in the appendix are nice for demonstrating some specific successes or weaknesses of the model, but they are in no way sufficient for evaluation of the system, to demonstrate its accuracy or value in general.
Perhaps the most obvious thing that should be done is to report the model's accuracy for reproducing the news dialogue, that is, how accurately is the next sentence predicted by the baseline and ACM models over the training instances and over the test data?  How does this compare with other state-of-the-art models for dialogue generation trained on this data (perhaps trained only on the textual part of the data in some cases)?
Second, some measure of accuracy for recall of the associative image context vector should be reported; for example, on average, how close (cosine similarity or some other appropriate measure) is the associatively recalled image context vector to the target image context vector?  On average?  Best case?  Worst case?  How often is this associative vector closer to a confounding image vector than an appropriate one?
A third natural kind of validation would be some form of study employing human subjects to test it's quality as a generator of dialogue.
One thing to note, the example of learning to associate the snowy image with the text about university entrance exams demonstrates that the model is memorizing rather than generalizing.  In general, this is a false association (that is, in general, there is no reason that snow should be associated with exams on the 14th and 15th—the month is not mentioned, which might justify such an association.)
Another thought: did you try not retraining the decoder and attention mechanisms for step 3?  In theory, if step 2 is successful, the retraining should not be necessary.  To the extent that it is necessary, step 2 has failed to accurately predict visual context from text.  This seems like an interesting avenue to explore (and is obviously related to the second type of validation suggested above).  Also, in addition to the baseline model, it seems like it would be good to compare a model that uses actual visual input and the model of step 1 against the model of step 3 (possibly bot retrained and not retrained) to see the effect on the outputs generated—how well do each of these do at predicting the next sentence on both training and test sets?
Other concerns:
1. The paper is too long by almost a page in main content.
2. The paper exhibits significant English grammar and usage issues and should be carefully proofed by a native speaker.
3. There are lots of undefined variables in the Eqs. (s, W_s, W_c, b_s, e_t,i, etc.)  Given the context and associated discussion, it is almost possible to sort out what all of them mean, but brief careful definitions should be given for clarity.
4. Using news broadcasts as a substitute for true dialogue data seems kind of problematic, though I see why it was done.


In this work, we focus on the problem of grounding language by training an agent to follow a set of natural language instructions and navigate to a target object in a 2D grid environment. The agent receives visual information through raw pixels and a natural language instruction telling what task needs to be achieved. Other than these two sources of information, our model does not have any prior information of both the visual and textual modalities and is end-to-end trainable. We develop an attention mechanism for multi-modal fusion of visual and textual modalities that allows the agent to learn to complete the navigation tasks and also achieve language grounding. Our experimental results show that our attention mechanism outperforms the existing multi-modal fusion mechanisms proposed in order to solve the above mentioned navigation task. We demonstrate through the visualization of attention weights that our model learns to correlate attributes of the object referred in the instruction with visual representations and also show that the learnt textual representations are semantically meaningful as they follow vector arithmetic and are also consistent enough to induce translation between instructions in different natural languages. We also show that our model generalizes effectively to unseen scenarios and exhibit zero-shot generalization capabilities. In order to simulate the above described challenges, we introduce a new 2D environment for an agent to jointly learn visual and textual modalities.

**Paper Summary**
The paper studies the problem of navigating to a target object in a 2D grid environment by following given natural language description as well as receiving visual information as raw pixels. The proposed architecture consists of a convoutional neural network encoding visual input,  gated recurrent unit encoding natural language descriptions, an attention mechanism fusing multimodal input, and a policy learning network. To verify the effectiveness of the proposed framework, a new environment is proposed. The environment is 2-D grid based and it consists of an agent, a list of objects with different attributes, and a list of obstacles. Agents perceive the environment throught raw pixels with a limited visible region, and they can perform actions to move in the environment to reach target objects.
The problem has been studied for a while and therefore it is not novel. The proposed framework is incremental. The proposed environment is trivial and therefore it is unclear if the proposed framework is able to scale up to a more complicated environment. The experiemental results do not support several claims stated in the paper. Overall, I would vote for rejection.
    - This paper solves the problem of navigating to the target object specified by language instruction in a 2D grid environment. It requires understanding of language, language grounding for visual features, and navigating to the target object while avoiding non-target objects. An attention mechanism is used to map a language instruction into a set of 1x1 convolutional filters which are intended to distinguish visual features described in the instruction from others. The experimental results show that the proposed method performs better than other methods.
    - This paper presents an end-to-end trainable model to navigate an agent through visual sources and natural language instructions. The model utilizes a proposed attention mechanism to draw correlation between the objects mentioned in the instructions with deep visual representations, without requiring any prior knowledge about these inputs. The experimental results demonstrate the effectiveness of the learnt textual representation and the zero-shot generalization capabilities to unseen scenarios.
**Paper Strengths**
- The paper proposes an interesting task which is a navigation task with language instructions. This is important yet relatively unxplored.
- The implementation details are included, including optimizers, learning rates with weight decayed, numbers of training epochs, the discount factor, etc.
- The attention mechanism used in the paper is reasonable and the learned language embedding clearly shows meaningful relationships between instructions.
- The learnt textual representation follows vector arithmetic, which enables the agent to perceive unseen instructions as a new combination of the attributes and perform zero-shot generalization.
**Paper Weaknesses**
- The problem of following natural language descriptions together with visual representations of environments is not completely novel. For example, Both the problem and the proposed method are similar to those already introduced in the Gated Attention method (Chaplot et al., 2017). Although the proposed method performs better than the prior work, the approach is incremental.
- The proposed environment is simple. The vocabulary size is 40 and the longest instruciton only consists of 9 words. Whether the proposed framework is able to deal with more complicated environments is not clear. The experimental results shown in Figure 5 is not convincing that the proposed method only took less than 20k iterations to perform almost perfectly. The proposed environment is small and simple compared to the related work. It would be better to test the proposed method in a similar scale with the existing 3D navigation environments (Chaplot et al., 2017 and Hermann et al., 2017).
- The novelty of the proposed framework is unclear. This work is not the first  one which proposes the multimodal fusion network incorporating a CNN achitecture dealing with visual information and a GRU architecture encoding language instructions. Also, the proposed attention mechanism is an obvious choice.
- The shown visualized attention maps are not enough to support the contribution of proposing the attention mechanism. It is difficult to tell whether the model learns to attend to correct objects. Also, the effectiveness of incorporating the attention mechanism is unclear.
- The paper claims that the proposed framework is flexible and is able to handle a rich set of natural language descriptions. However, the experiemental results are not enough to support the claim.
- The presentaiton of the experiment is not space efficient at all.
- The reference of the related papers which fuse multimodal data (vision and language) are missing.
- Comapred to 8 pages was the suggested page limit, 13 pages is a bit too long.
- Stating captions of figures above figures is not recommended.
- It would be better to show where each 1x1 filter for multimodal fusion attends on the input image.  Ideally, one filter should attend on the target object and others should attend on non-target objects. However, I wonder how RNN can generate filters to detect non-target objects given an instruction. Although Figure 6 and Figure 7 try to show insights about the proposed attention model, they don’t tell which kernel is in charge of which visual feature. Blurred attention maps in Figure 6 and 7 make it hard to interpret the behavior of the model.
- The graphs shown in the Figure 5 are hard to interpret because of their large variance. It would be better to smoothing curves, so that comparing methods clearly.
- For zero-shot generalization evaluation, there is no detail about the training steps and comparisons to other methods.
- A highly related paper (Hermann et al., 2017) is missing in the references.
- Since the instructions are simple, the model does not require attention mechanism on the textual sources. If the framework can take more complex language, might be worthwhile to try visual-text co-attention mechanism. Such demonstration will be more convincing.
- The attention maps of different attribute is not as clear as the paper stated. Why do we need several “non-target” objects highlight if one can learn to consolidate all of them?
- The interpretation of n in the paper is vague, the authors should also show qualitatively why n=5 is better than that of n=1,10. If the attention maps learnt are really focusing on different attributes, given more and more objects, shouldn’t n=10 have more information for the policy learning?
- The unseen scenario generalization should also include texture change on the grid environment and/or new attribute combinations on non-target objects to be more convincing.
- The contribution in the visual part is marginal.
** Preliminary Evaluation**
- The modality fusion technique which leads to the attention maps is an effective and seem to work well approach, however, the author should present more thorough ablated analysis. The overall architecture is elegant, but the capability of it to be extended to more complex environment is in doubt. The vector arithmetic of the learnt textual embedding is the key component to enable zero-shot generalization, while the effectiveness of this method is not convincing if more complex instructions such that it contains object-object relations or interactions are perceived by the agent.


In this paper, we study the representational power of deep neural networks (DNN) that belong to the family of piecewise-linear (PWL) functions, based on PWL activation units such as rectifier or maxout. We investigate the complexity of such networks by studying the number of linear regions of the PWL function. Typically, a PWL function from a DNN can be seen as a large family of linear functions acting on millions of such regions. We directly build upon the work of Montúfar et al. (2014), Montúfar (2017), and Raghu et al. (2017) by refining the upper and lower bounds on the number of linear regions for rectified and maxout networks. In addition to achieving tighter bounds, we also develop a novel method to perform exact enumeration or counting of the number of linear regions with a mixed-integer linear formulation that maps the input space to output. We use this new capability to visualize how the number of linear regions change while training DNNs.

Paper Summary:
This paper looks at providing better bounds for the number of linear regions in the function represented by a deep neural network. It first recaps some of the setting: if a neural network has a piecewise linear activation function (e.g. relu, maxout), the final function computed by the network (before softmax) is also piecewise linear and divides up the input into polyhedral regions which are all different linear functions. These regions also have a correspondence with Activation Patterns, the active/inactive pattern of neurons over the entire network. Previous work [1], [2], has derived lower and upper bounds for the number of linear regions that a particular neural network architecture can have. This paper improves on the upper bound given by [2] and the lower bound given by [1]. They also provide a tight bound for the one dimensional input case. Finally, for small networks, they formulate finding linear regions as solving a linear program, and use this method to compute the number of linear regions on small networks during training on MNIST
Main Comments:
The paper is very well written and clearly states and explains the contributions. However, the new bounds proposed (Theorem 1, Theorem 6), seem like small improvements over the previously proposed bounds, with no other novel interpretations or insights into deep architectures. (The improvement on Zaslavsky's theorem is interesting.) The idea of counting the number of regions exactly by solving a linear program is interesting, but is not going to scale well, and as a result the experiments are on extremely small networks (width 8), which only achieve 90% accuracy on MNIST. It is therefore hard to be entirely convinced by the empirical conclusions that more linear regions is better. I would like to see the technique of counting linear regions used even approximately for larger networks, where even though the results are an approximation, the takeaways might be more insightful.
Overall, while the paper is well written and makes some interesting points, it presently isn't a significant enough contribution to warrant acceptance.
[1] On the number of linear regions of Deep Neural Networks, 2014, Montufar, Pascanu, Cho, Bengio
[2] On the expressive power of deep neural networks, 2017, Raghu, Poole, Kleinberg, Ganguli, Sohl-Dickstein


Workshop track -ICLR 2018 THE EFFECTIVENESS OF A TWO-LAYER NEURAL NET- WORK FOR RECOMMENDATIONS
We present a personalized recommender system using neural network for recommending products, such as eBooks, audio-books, Mobile Apps, Video and Music. It produces recommendations based on customer's implicit feedback history such as purchases, listens or watches. Our key contribution is to formulate recommendation problem as a model that encodes historical behavior to predict the future behavior using soft data split, combining predictor and auto-encoder models. We introduce convolutional layer for learning the importance (time decay) of the purchases depending on their purchase date and demonstrate that the shape of the time decay function can be well approximated by a parametrical function. We present offline experimental results showing that neural networks with two hidden layers can capture seasonality changes, and at the same time outperform other modeling techniques, including our recommender in production. Most importantly, we demonstrate that our model can be scaled to all digital categories, and we observe significant improvements in an online A/B test. We also discuss key enhancements to the neural network model and describe our production pipeline. Finally we open-sourced our deep learning library which supports multi-gpu model parallel training. This is an important feature in building neural network based recommenders with large dimensionality of input and output data.

The paper proposes a new neural network based method for recommendation.
The main finding of the paper is that a relatively simple method works for recommendation, compared to other methods based on neural networks that have been recently proposed.
This contribution is not bad for an empirical paper. There's certainly not that much here that's groundbreaking methodologically, though it's certainly nice to know that a simple and scalable method works.
There's not much detail about the data (it is after all an industrial paper). It would certainly be helpful to know how well the proposed method performs on a few standard recommender systems benchmark datasets (compared to the same baselines), in order to get a sense as to whether the improvement is actually due to having a better model, versus being due to some unique attributes of this particular industrial dataset under consideration. As it is, I am a little concerned that this may be a method that happens to work well for the types of data the authors are considering but may not work elsewhere.
Other than that, it's nice to see an evaluation on real production data, and it's nice that the authors have provided enough info that the method should be (more or less) reproducible. There's some slight concern that maybe this paper would be better for the industry track of some conference, given that it's focused on an empirical evaluation rather than really making much of a methodological contribution. Again, this could be somewhat alleviated by evaluating on some standard and reproducible benchmarks.


We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is to identify only whether the given example belongs to a specific class, which can be different in different applications of the classifier. For instance, this is the case in an image search engine. We consider the Single Logit Classification (SLC) task: training the network so that at test-time, it would be possible to accurately identify if the example belongs to a given class, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing losses suitable for the SLC. We show that the cross-entropy loss function is not aligned with the Principle of Logit Separation. In contrast, there are known loss functions, as well as novel batch loss functions that we propose, which are aligned with this principle. In total, we study seven loss functions. Our experiments show that indeed in almost all cases, losses that are aligned with Principle of Logit Separation obtain a 20%-35% relative performance improvement in the SLC task, compared to losses that are not aligned with it. We therefore conclude that the Principle of Logit Separation sheds light on an important property of the most common loss functions used by neural network classifiers. Tensorflow code for optimizing the new batch losses will be made publicly available upon publication; A URL will be provided in the publication version of this manuscript.

The paper addresses the problem of a mismatch between training classification loss and a loss at test time. This is motivated by use cases in which multiclass classification problems are learned during training, but where binary or reduced multi-class classifications is performed at test time. The question for me is the following: if at test time, we have to solve "some" binary classification task, possibly drawn at random from a set of binary problems (this is not made precise in the paper), then why not optimize the same classification error or a surrogate loss at training time? Instead, the authors start with a multiclass problem, which may introduce a computational burden. when the number of classes is large as one needs to compute a properly normalized softmax. The authors now seem to ask, what if one were to use a multi-classification loss at training time, but then decides at test time that a binary classification of one-vs-all is asked for.
If one buys into the relevance of the setting, then of course, one is faced with the problem that the multiclass logits (aka raw scores) may not be calibrated to be used for binary classification by applying a fixed threshold. The authors call this sententiously "Principle of logit separation". Not too surprisingly, the standard multiclass losses do not have the desired property, however approaches that reduce multi-class to binary classification at training time do, namely unnormalized models with penalized log Z (self-normalization), the NCE approach, as well as (the natural in the proposed setting) binary classification loss. I find this almost a bit circular in the line of argumentation, but ok. It remains odd that while usually one has tried to reduce multiclass to binary, the authors go the opposite direction.
The main technical contribution of the paper is the batch-nornalization that makes sure that multiclass logits across mini-batches of data are better calibrated. One can almost think of that as an additional regularization. This seems interesting and does not create much overhead, if one applies mini-batched SGD optimization anyway. However, I feel this technique would need to be investigated with regard to general improvements in a multiclass setting and as such also benchmarked relative to other methods that could be applied.


TRAINING GANS WITH OPTIMISM
We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs. Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games. WGANs is exactly a context of solving a zero-sum game with simultaneous noregret dynamics. Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs. We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle. We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum. We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences. We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants. Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam. We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

This paper proposes the use of optimistic mirror descent to train Wasserstein Generative Adversarial Networks (WGANS). The authors remark that the current training of GANs, which amounts to solving a zero-sum game between a generator and discriminator, is often unstable, and they argue that one source of instability is due to limit cycles, which can occur for FTRL-based algorithms even in convex-concave zero-sum games. Motivated by recent results that use Optimistic Mirror Descent  (OMD) to achieve faster convergence rates (than standard gradient descent) in convex-concave zero-sum games and normal form games, they suggest using these techniques for WGAN training as well. The authors prove that, using OMD, the last iterate converges to an equilibrium and use this as motivation that OMD methods should be more stable for WGAN training. They then compare OMD against GD on both toy simulations and a DNA sequence task before finally introducing an adaptive generalization of OMD, Optimistic Adam, that they test on CIFAR10.
This paper is relatively well-written and clear, and the authors do a good job of introducing the problem of GAN training instability as well as the OMD algorithm, in particular highlighting its differences with standard gradient descent as well as discussing existing work that has applied it to zero-sum games. Given the recent work on OMD for zero-sum and normal form games, it is natural to study its effectiveness in training GANs.The issue of last iterate versus average iterate for non convex-concave problems is also presented well.
The theoretical result on last-iterate convergence of OMD for bilinear games is interesting, but somewhat wanting as it does not provide an explicit convergence rate as in Rakhlin and Sridharan, 2013. Moreover, the result is only at best a motivation for using OMD in WGAN training since the WGAN optimization problem is not a bilinear game.
The experimental results seem to indicate that OMD is at least roughly competitive with GD-based methods, although they seem less compelling than the prior discussion in the paper would suggest. In particular, they are matched by SGD with momentum when evaluated by last epoch performance (albeit while being less sensitive to learning rates). OMD does seem to outperform SGD-based methods when using the lowest discriminator loss, but there doesn't seem to be even an attempt at explaining this in the paper.
I found it a bit odd that Adam was not used as a point of comparison in Section 5, that optimistic Adam was only introduced and tested for CIFAR but not for the DNA sequence problem, and that the discriminator was trained for 5 iterations in Section 5 but only once in Section 6, despite the fact that the reasoning provided in Section 6 seems like it would have also applied for Section 5. This gives the impression that the experimental results might have been at least slightly "gamed".
For the reasons above, I give the paper high marks on clarity, and slightly above average marks on originality, significance, and quality.
Specific comments:
Page 1, "no-regret dynamics in zero-sum games can very often lead to limit cycles": I don't think limit cycles are actually ever formally defined in the entire paper.
Page 3, "standard results in game theory and no-regret learning": These results should be either proven or cited.
Page 3: Don't the parameter spaces need to be bounded for these convergence results to hold?
Page 4, "it is well known that GD is equivalent to the Follow-the-Regularized-Leader algorithm": For completeness, this should probably either be (quickly) proven or a reference should be provided.
Page 5, "the unique equilibrium of the above game is...for the discriminator to choose w=0": Why is w=0 necessary here?
Page 6, "We remark that the set of equilibrium solutions of this minimax problem are pairs (x,y) such that x is in the null space of A^T and y is in the null space of A": Why is this true? This should either be proven or cited.
Page 6, Initialization and Theorem 1: It would be good to discuss the necessity of this particular choice of initialization for the theoretical result. In the Initialization section, it appears simply to be out of convenience.
Page 6, Theorem 1: It should be explicitly stated that this result doesn't provide a convergence rate, in contrast to the existing OMD results cited in the paper.
Page 7, "we considered momentum, Nesterov momentum and AdaGrad": Why isn't Adam used in this section if it is used in  later experiments?
Page 7-8, "When evaluated by....the lowest discriminator loss on the validation set, WGAN trained with Stochastic OMD (SOMD) achieved significantly lower KL divergence than the competing SGD variants.": Can you explain why SOMD outperforms the other methods when using the lowest discriminator loss on the validation set? None of the theoretical arguments presented earlier in the paper seem to even hint at this. The only result that one might expect from the earlier discussion and results is that SOMD would outperform the other methods when evaluating by the last epoch. However, this doesn't even really hold, since there exist learning rates in which SGD with momentum matches the performance of SOMD.
Page 8, "Evaluated by the last epoch, SOMD is much less sensitive to the choice of learning rate than the SGD variants": Learning rate sensitivity doesn't seem to be touched upon in the earlier discussion. Can these results be explained by theory?
Page 8, "we see that optimistic Adam achieves high numbers of inception scores after very few epochs of training": These results don't mean much without error bars.
Page 8, "we only trained the discriminator once after one iteration of generator training. The latter is inline with the intuition behind the use of optimism....": Why didn't this logic apply to the previous section on DNA sequences, where the discriminator was trained multiple times?
After reading the response of the authors (in particular their clarification of some technical results and the extra experiments they carried out during the rebuttal period), I have decided to upgrade my rating of the paper from a 6 to a 7. Just as a note, Figure 3b is now very difficult to read.


Effective training of neural networks requires much data. In the low-data regime, parameters are underdetermined, and learnt networks generalise poorly. Data Augmentation (Krizhevsky et al., 2012) alleviates this by using existing data more effectively. However standard data augmentation produces only limited plausible alternative data. Given there is potential to generate a much broader set of augmentations, we design and train a generative model to do data augmentation. The model, based on image conditional Generative Adversarial Networks, takes data from a source domain and learns to take any data item and generalise it to generate other within-class data items. As this generative process does not depend on the classes themselves, it can be applied to novel unseen classes of data. We show that a Data Augmentation Generative Adversarial Network (DAGAN) augments standard vanilla classifiers well. We also show a DAGAN can enhance few-shot learning systems such as Matching Networks. We demonstrate these approaches on Omniglot, on EMNIST having learnt the DAGAN on Omniglot, and VGG-Face data. In our experiments we can see over 13% increase in accuracy in the low-data regime experiments in Omniglot (from 69% to 82%), EMNIST (73.9% to 76%) and VGG-Face (4.5% to 12%); in Matching Networks for Omniglot we observe an increase of 0.5% (from 96.9% to 97.4%) and an increase of 1.8% in EMNIST (from 59.5% to 61.3%).

This paper proposes a conditional Generative Adversarial Networks that is used for data augmentation. In order to evaluate the performance of the proposed model, they use Omniglot, EMNIST, and VGG-Faces datasets and uses in the meta-learning task and standard classification task in the low-data regime. The paper is well-written and consistent.
Even though this paper learns to do data-augmentation (which is very interesting ) rather than just simply applies some standard data augmentation techniques and shows improvements in some tasks, I am not convinced about novelty and originality of this paper, especially on the model side. To be more specific, the paper uses the previously proposed conditional GAN as the main component of their model. And for the one-shot learning tasks, it only trains the previously proposed models with these newly augmented data.
In addition, there are some other works that used GAN as a method for some version of data augmentation:
- RenderGAN: Generating Realistic Labeled Data
  https://arxiv.org/abs/1611.01331
-Data Augmentation in Emotion Classification Using Generative Adversarial Networks
https://arxiv.org/abs/1711.00648
It is fair to say that their model shows improvement on the above tasks but this improvement comes with a cost of training of GAN network.
In summary, the idea of the paper is very interesting to learn data-augmentation but yet I am not convinced the current paper has enough novelty and contribution and see the contribution of paper as on more the application side rather than on model and problem side. That said I'd be happy to hear the argument of the author about my comments.


Artificial neural networks have opened up a world of possibilities in data science and artificial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem. We make contributions to this issue by considering a modified version of the fully connected layer we call a block diagonal inner product layer. These modified layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups. This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers. Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries. This method condenses network storage and speeds up the run time without significant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efficiency.

The paper proposes to make the inner layers in a neural network be block diagonal, mainly as an alternative to pruning. The implementation of this seems straightforward, and can be done either via initialization or via pruning on the off-diagonals. There are a few ideas the paper discusses:
(1) compared to pruning weight matrices and making them sparse, block diagonal matrices are more efficient since they utilize level 3 BLAS rather than sparse operations which have significant overhead and are not "worth it" until the matrix is extremely sparse. I think this case is well supported via their experiments, and I largely agree.
(2) that therefore, block diagonal layers lead to more efficient networks. This point is murkier, because the paper doesn't discuss possible increases in *training time* (due to increased number of iterations) in much detail. At if we only care about running the net, then reducing the time from 0.4s to 0.2s doesn't seem to be that useful (maybe it is for real-time predictions? Please cite some work in that case)
(3) to summarize points (1) and (2), block diagonal architectures are a nice alternative to pruned architectures, with similar accuracy, and more benefit to speed (mainly speed at run-time, or speed of a single iteration, not necessarily speed to train)
[as I am not primarly a neural net researcher, I had always thought pruning was done to decrease over-fitting, not to increase computation speed, so this was a surprise to me; also note that the sparse matrix format can increase runtime if implemented as a sparse object, as demonstrated in this paper, but one could always pretend it is sparse, so you never ought to be slower with a sparse matrix]
(4) there is some vague connection to random matrices, with some limited experiments that are consistent with this observation but far from establish it, and without any theoretical analysis (Martingale or Markov chain theory)
This is an experimental/methods paper that proposes a new algorithm, explained only in general details, and backs up it up with two reasonable experiments (that do a good job of convincing me of point (1) above). The authors seem to restrict themselves to convolutional networks in the first paragraph (and experiments) but don't discuss the implications or reasons of this assumption. The authors seem to understand the literature well, and not being an expert myself, I have the impression they are doing a fair job.
The paper could have gone farther experimentally (or theoretically) in my opinion. For example, with sparse and block diagonal matrices, reducing the size of the matrix to fit into the cache on the GPU must obviously make a difference, but this did not seem to be investigated. I was also wondering about when 2 or more layers are block sparse, do these blocks overlap? i.e., are they randomly permuted between layers so that the blocks mix? And even with a single block, does it matter what permutation you use? (or perhaps does it not matter due to the convolutional structure?)
The section on the variance of the weights is rather unclear mathematically, starting with the abstract and even continuing into the paper. We are talking about sample variance? What does DeltaVar mean in eq (2)? The Marchenko-Pastur theorem seemed to even be imprecise, since if y>1, then a < 0, implying that there is a nonzero chance that the positive semi-definite matrix XX' has a negative eigenvalue.
I agree this relationship with random matrices could be interesting, but it seems too vague right now. Is there some central limit theorem explanation? Are you sure that you've run enough iterations to fully converge? (Fig 4 was still trending up for b1=64). Was it due to the convolutional net structure (you could test this)? Or, perhaps train a network on two datasets, one which is not learnable (iid random labels), and one which is very easily learnable (e.g., linearly separable). Would this affect the distributions?
Furthermore, I think I misunderstood parts, because the scaling in MNIST and CIFAR was different and I didn't see why (for MNIST, it was proportional to block size, and for CIFAR it was independent of block size almost).
Minor comment: last paragraph of 4.1, comparing with Sindhwani et al., was confusing to me. Why was this mentioned? And it doesn't seem to be comparable. I have no idea what "Toeplitz (3)" is.


Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes nodes' representation recursively from their neighbors, making the receptive field size grow exponentially with the number of layers. Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive field size. Our algorithms are guaranteed to converge to GCN's local optimum regardless of the neighbor sampling size. Empirical results show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node. The time consumption of our algorithm on the Reddit dataset is only one fifth of previous neighbor sampling algorithms.

The paper proposes a method to speed up the training of graph convolutional networks, which are quite slow for large graphs. The key insight is to improve the estimates of the average neighbor activations (via neighbor sampling) so that we can either sample less neighbors or have higher accuracy for the same number of sampled neighbors. The idea is quite simple: estimate the current average neighbor activations as a delta over the minibatch running average. I was hoping the method would also include importance sampling, but it doesn’t. The assumption that activations in a graph convolution are independent Gaussians is quite odd (and unproven).
Quality: Statistically, the paper seems sound. There are some odd assumptions (independent Gaussian activations in a graph convolution embedding?!?) but otherwise the proposed methodology is rather straightforward.
Clarity: It is well written and the reader is able to follow most of the details. I wish the authors had spent more time discussing the independent Gaussian assumption, rather than just arguing that a graph convolution (where units are not interacting through a simple grid like in a CNN) is equivalent to the setting of Wang and Manning (I don’t see the equivalence). Wang and Manning are looking at MLPs, not even CNNs, which clearly have more independent activations than a CNN or a graph convolution.
Significance: Not very significant. The problem of computing better averages for a specific problem (neighbor embedding average) seems a bit too narrow. The solution is straightforward, while some of the approximations make some odd simplifying assumptions (independent activations in a convolution, infinitesimal learning rates).
Theorem 2 is not too useful, unfortunately: Showing that the estimated gradient is asymptotically unbiased with learning rates approaching zero over Lipchitz functions does not seem like an useful statement. Learning rates will never be close enough to zero (specially for large batch sizes). And if the running activation average converges to the true value, the training is probably over. The method should show it helps when the values are oscillating in the early stages of the training, not when the training is done near the local optimum.


ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS
Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a very simple, and yet counter-intuitive, postprocessing technique -eliminate the common mean vector and a few top dominating directions from the word vectors -that renders off-the-shelf representations even stronger. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classification) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

This paper proposes a simple post-processing technique for word representations designed to improve representational quality and performance on downstream tasks. The procedure involves mean subtraction followed by projecting out the first D principle directions and is motivated by improving isotropy of the partition function. Extensive empirical analysis supports the efficacy of the approach.
The idea of post-processing word embeddings to improve their performance is not new, but I believe the specific procedure and its connection to the concept of isotropy has not been investigated previously. Relative to other post-processing techniques, this method has a fair amount of theoretical justification, particularly as described in Appendix A. I think the experiments are reasonably comprehensive. All told, I think this is a good paper, but I do have some comments and questions that I think should be addressed before publication.
1) I think it is useful to analyze the distribution of singular values of the matrix of word vectors. However, I did not find the heuristic analysis based on the visual appearance of these distributions to be convincing. For example, in Fig. 1, it is not clear to me that there exists a separation between regimes of exponential decay and rough constancy. It would be ideal if a more quantitative metric is established that captures the main qualitative behavior alluded to here.
Furthermore, the vocabulary size is likely to have a strong effect on the shape of the distributions. Are the plots in Fig. 4 for the same vocabulary size? Related to this, the dimensionality of the representation will have a strong effect on the shape, and this should be controlled for in Fig. 8. One way to do this would be to instead plot the density of singular values. Finally, for the Gaussian matrix simulations, in the asymptotic limit, the density of singular values depends only on the ratio of dimensions, i.e. the vector dimension to the vocabulary size. Fig. 4/8 might be more revealing if this ratio were controlled for.
2) It would be useful to describe why isotropy of the partition function is the goal, as opposed to isotropy of the vectors themselves. This may be argued in Arora et al. (2016), but summarizing that argument in this paper would be helpful. In fact, an additional experiment that would be very valuable would be to investigate empirically which form of isotropy is more effective in governing performance. One way to do this would be to enforce approximate isotropy of the partition function without also enforcing isotropy of the vectors themselves. Practically speaking, one might imagine doing this by requiring I = 1 to second order without also requiring that the mean vanish. I think this would allow for \sigma_max > \sigma_min while still satisfying I = 1 to second order. (But this is just off the top of my head -- there may be better ways to conduct this experiment).
It is not clear to me why the experiment leading to Table 2 is a good proxy for the exact computation of I. It would be great if there were some mathematical justification for this approximation.
Why does Fig. 3 use D=10, 20 when much smaller D are considered elsewhere? Also I think a log scale on the x-axis might be more informative.
3) It would be good to mention other forms of post-processing, especially in the context of word similarity. For example, in the original paper, GloVe advocates averaging the target and context vector representations, and normalizing across the feature dimension before computing cosine similarity.
4) I think it's likely that there is a strong connection between the optimal value of D and the frequency distribution of words in the evaluation dataset. While the paper does mention that D may depend on specifics of the dataset, etc., I would expect frequency-dependence to be the main factor, and it might be worth exploring this effect explicitly.


The large memory requirements of deep neural networks limit their deployment and adoption on many devices. Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization. In this paper, we present a novel scheme for lossy weight encoding which complements conventional compression techniques. The encoding is based on the Bloomier filter, a probabilistic data structure that can save space at the cost of introducing random errors. Leveraging the ability of neural networks to tolerate these imperfections and by re-training around the errors, the proposed technique, Weightless, can compress DNN weights by up to 496× with the same model accuracy. This results in up to a 1.51× improvement over the state-of-the-art.

This paper proposes an interesting approach to compress the weights of a network for storage or transmission purposes. My understanding is, at inference, the network is 'recovered' therefore there is no difference in processing time (slight differences in accuracy due to the approximation in recovering the weights).
- The idea is nice although it's applicability is limited as it is only for distribution of the model and storing (is storage really a problem?).
Method:
- the idea of using the Bloomier filter is new to me. However, the paper is miss-leading as the filtering is a minor part of the complete process. The paper introduces a complete pipeline including quantization, and pruning to maximize the benefits of the filter and an additional (optional) step to achieve further compression.
- The method / idea seems simply and easy to reproduce (except the subsequent steps that are not clearly detailed).
Clarity
- The paper could improve its clarity. At the moment, the Bloomier is the core but needs many other components to make it effective. Those components are not detailed to the level of being reproducible.
- One interesting point is the self-implementation of the Deep compression algorithm. The paper claims this is a competitive representation as it achieves better compression than the original one. However, those numbers are not clear in tables (only in table 3 numbers seem to be equivalent to the ones in the text). This needs clarification, CSR achieves 81.8% according to Table 2 and 119 according to the text.
Results:
- Current results are interesting. However I have several concerns:
1) it is not clear to me why assuming similar performance. While Bloomier is weightless the complete process involves many retraining steps involving performance loss. Analysis on this would be nice to see (I doubt it ends exactly at the same number). Section 3 explicitly suggest there is the need of retraining to mitigate the effect of false positives which is then increased with pruning and quantization. Therefore, would be nice to see the impact in accuracy (even it is not the main focus of the work).
2) Resutls are focused on fully connected layers which carry (for the given models) the larger number of weights (and therefore it is easy to get large compression numbers). What would happen in newer models where the fully connected layer is minimal compared to conv. layers? What about the accuracy impact there? Let's say in a Resnet-34.
3) I would like to see further analysis on why Bloomier filter encoding improves accuracy (or is a typo and meant to be error?) by 2%. This is a large improvement without training from scractch.
4) It is interesting to me how the retraining process is 'hidden' all over the paper. At the beginning it is claimed that it takes about one hour for VGG-16 to compute the Bloomier filters. Howerver, that is only a minimal portion of the entire pipeline. Later in the experimental section it is mentioned that 'tens of epochs' are needed for retraining (assuming to compensate for errors) after retraining for compensating l1 pruning?.... tens of epochs is a significant portion of the entire training process assuming VGG is trained for 90epochs max.
5) Interestingly, as mentioned in the paper, this is 'static compression'. That is, the model needs to be completely 'restored' before inference. This is miss-leading as an embedded device will need the same requirements as any other at inferece time(or maybe I am missing something). That is, the benefit is mainly for storing and transmission.
6) I would like to see the sensibility analysis with respect to t and the number of clusters.
7) As mentioned before, LeNet is great but would be nice to see more complicated models (even resnet on CIFAR). These models are not only large in terms of parameters but also quite sensitive to modifications in the weight structure.
8) Results are focused on a single layer. What happens if all the layers are considered at the same time? Here I am also concerned about the retraining process (fixing one layer and retraining the deeper ones). How is this done using only fully connected layers? What is the impact of doing it all over the network (let's say VGG-16 from the first convolutional layer to the very last).
Summary:
All in all, the idea has potential but there are many missing details. I would like to see clearer and more comprehensive results in terms of modern models and in the complete model, not only in the FC layer, including accuracy impact.


Transfer learning for feature extraction can be used to exploit deep representations in contexts where there is very few training data, where there are limited computational resources, or when tuning the hyper-parameters needed for training is not an option. While previous contributions to feature extraction propose embeddings based on a single layer of the network, in this paper we propose a full-network embedding which successfully integrates convolutional and fully connected features, coming from all layers of a deep convolutional neural network. To do so, the embedding normalizes features in the context of the problem, and discretizes their values to reduce noise and regularize the embedding space. Significantly, this also reduces the computational cost of processing the resultant representations. The proposed method is shown to outperform single layer embeddings on several image classification tasks, while also being more robust to the choice of the pre-trained model used for obtaining the initial features. The performance gap in classification accuracy between thoroughly tuned solutions and the full-network embedding is also reduced, which makes of the proposed approach a competitive solution for a large set of applications.

The paper addresses the scenario when using a pretrained deep network as learnt feature representation for another (small) task where retraining is not an option or not desired. In this situation it proposes to use all layers of the network to extract feature from, instead of only one layer.
Then it proposes to standardize different dimensions of the features based on their response on the original task. Finally, it discretize each dimension into {-1, 0, 1} to compress the final concatenated feature representation.
Doing this, it shows improvements over using a single layer for 9 target image classification datasets including object, scene, texture, material, and animals.
The reviewer does not find the paper suitable for publication at ICLR due to the following reasons:
- The paper is incremental with limited novelty.
- the results are not encouraging
- the pipeline of standardization, discretization is relatively costly, the final feature vector still large.
- combining different layers, as the only contribution of the paper, has been done in the literature before,  for instance:
“The Treasure beneath Convolutional Layers: Cross-convolutional-layer Pooling
for Image Classification” CVPR 2016


In most current formulations of adversarial training, the discriminators can be expressed as single-input operators, that is, the mapping they define is separable over observations. In this work, we argue that this property might help explain the infamous mode collapse phenomenon in adversarially-trained generative models. Inspired by discrepancy measures and two-sample tests between probability distributions, we propose distributional adversaries that operate on samples, i.e., on sets of multiple points drawn from a distribution, rather than on single observations. We show how they can be easily implemented on top of existing models. Various experimental results show that generators trained in combination with our distributional adversaries are much more stable and are remarkably less prone to mode collapse than traditional models trained with observation-wise prediction discriminators. In addition, the application of our framework to domain adaptation results in strong improvement over recent state-of-the-art.

The paper proposes to replace single-sample discriminators in adversarial training with discriminators that explicitly operate on distributions of examples, so as to incentivize the generator to cover the full distribution of the training data and not collapse to isolated modes.
The idea of avoiding mode collapse by providing multiple samples to the discriminator is not new; the paper acknowledges prior work on minibatch discrimination but does not really describe the differences with previous work in any technical detail. Not being highly familiar with this literature, my reading is that the scheme in this paper grounds out into a somewhat different architecture than previous minibatch discriminators, with a nice interpretation in terms of a sample-based approximation to a neural mean embedding. However the paper does not provide any empirical evidence that their approach actually works better than previous approaches to minibatch discrimination. By comparing only to one-sample discriminators it leaves open the (a priori quite plausible) possibility that minibatch discrimination is generally a good idea but that other architectures might work equally well or better, i.e., the experiments do not demonstrate that the MMD machinery that forms the core of the paper has any real purchase.
The paper also proposes a two-sample objective DAN-2S, in which the discriminator is asked to classify two sets of samples as coming from the same or different distributions. This is an interesting approach, although empirically it does not appear to have any advantage over the simpler DAN-S -- do the authors agree with this interpretation? If so it is still a worthwhile negative result, but the paper should make this conclusion explicit. Alternately if there are cases when the two-sample test is actually recommended, that should be made explicit as well.
Overall this paper seems borderline -- a nice theoretical story, grounding out into a simple architecture that does seem to work in practice (the domain adaptation results are promising), but with somewhat sloppy writing and experimentation that doesn't clearly demonstrate the value of the proposed approach. I hope the authors continue to improve the paper by comparing to other minibatch discrimination techniques. It would also be helpful to see value on a real-world task where mode collapse is explicitly seen as a problem (and/or to provide some intuition for why this would be the case in the Amazon reviews dataset).
Specific comments:
- Eqn (2.2) is described as representing the limit of a converged discriminator, but it looks like this is just the general gradient of the objective --- where does D* enter into the picture?
- Fig 1: the label R is never explained; why not just use P_x?
- Section 5.1 "we use the pure distributional objective for DAN (i.e., setting λ != 0 in (3.5))" should this be λ = 0?
- "Results" in the domain adaptation experiments are not clearly explained -- what do the reported numbers represent? (presumably accuracy(stddev) but the figure caption should say this). It is also silly to report accuracy to 2 decimal places when they are clearly not significant at that level.


Published as a conference paper at ICLR 2018 LEARNING SPARSE LATENT REPRESENTATIONS WITH THE DEEP COPULA INFORMATION BOTTLENECK
Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information bottleneck method, leads to disentanglement of the features in the latent space. Building on that, we show how this transformation translates to sparsity of the latent space in the new model. We evaluate our method on artificial and real data.

This paper identifies and proposes a fix for a shortcoming of the Deep Information Bottleneck approach, namely that the induced representation is not invariant to monotonic transform of the marginal distributions (as opposed to the mutual information on which it is based). The authors address this shortcoming by applying the DIB to a transformation of the data, obtained by a copula transform. This explicit approach is shown on synthetic experiments to preserve more information about the target, yield better reconstruction and converge faster than the baseline. The authors further develop a sparse extension to this Deep Copula Information Bottleneck (DCIB), which yields improved representations (in terms of disentangling and sparsity) on a UCI dataset.
(significance) This is a promising idea. This paper builds on the information theoretic perspective of representation learning, and makes progress towards characterizing what makes for a good representation. Invariance to transforms of the marginal distributions is clearly a useful property, and the proposed method seems effective in this regard.
Unfortunately, I do not believe the paper is ready for publication as it stands, as it suffers from lack of clarity and the experimentation is limited in scope.
(clarity) While Section 3.3 clearly defines the explicit form of the algorithm (where data and labels are essentially pre-processed via a copula transform), details regarding the “implicit form” are very scarce. From Section 3.4, it seems as though the authors are optimizing the form of the gaussian information bottleneck I(x,t), in the hopes of recovering an encoder $f_\beta(x)$ which gaussianizes the input (thus emulating the explicit transform) ? Could the authors clarify whether this interpretation is correct, or alternatively provide additional clarifying details ? There are also many missing details in the experimental section: how were the number of “active” components selected ? Which versions of the algorithm (explicit/implicit) were used for which experiments ? I believe explicit was used for Section 4.1, and implicit for 4.2 but again this needs to be spelled out more clearly. I would also like to see a discussion (and perhaps experimental comparison) to standard preprocessing techniques, such as PCA-whitening.
(quality) The experiments are interesting and seem well executed. Unfortunately, I do not think their scope (single synthetic, plus a single UCI dataset) is sufficient. While the gap in performance is significant on the synthetic task, this gap appears to shrink significantly when moving to the UCI dataset. How does this method perform for more realistic data, even e.g. MNIST ? I think it is crucial to highlight that the deficiencies of DIB matter in practice, and are not simply a theoretical consideration. Similarly, the representation analyzed in Figure 7 is promising, but again the authors could have targeted other common datasets for disentangling, e.g. the simple sprites dataset used in the beta-VAE paper. I would have also liked to see a more direct and systemic validation of the claims made in the paper. For example, the shortcomings of DIB identified in Section 3.1, 3.2 could have been verified more directly by plotting I(y,t) for various monotonic transformations of x. A direct comparison of the explicit and implicit forms of the algorithms would also also make for a stronger paper in my opinion.
Pros:
* Theoretically well motivated
* Promising results on synthetic task
* Potential for impact
Cons:
* Paper suffers from lack of clarity (method and experimental section)
* Lack of ablative / introspective experiments
* Weak empirical results (small or toy datasets only).


We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilistic forecast. A Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function. We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classification task. Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution. A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented. The evaluation of the proposed algorithm on three synthetic and two real data sets shows advantage over the compared baselines.

Interesting ideas that extend LSTM to produce probabilistic forecasts for univariate time series, experiments are okay. Unclear if this would work at all in higher-dimensional time series. It is also unclear to me what are the sources of the uncertainties captured.
The author proposed to incorporate 2 different discretisation techniques into LSTM, in order to produce probabilistic forecasts of univariate time series. The proposed approach deviates from the Bayesian framework where there are well-defined priors on the model, and the parameter uncertainties are subsequently updated to incorporate information from the observed data, and propagated to the forecasts. Instead, the conditional density p(y_t|y_{1:t-1|, \theta}) was discretised by 1 of the 2 proposed schemes and parameterised by a LSTM. The LSTM was trained using discretised data and cross-entropy loss with regularisations to account for ordering of the discretised labels. Therefore, the uncertainties produced by the model appear to be a black-box. It is probably unlikely that the discretisation method can be generalised to high-dimensional setting?
Quality: The experiments with synthetic data sufficiently showed that the model can produce good forecasts and predictive standard deviations that agree with the ground truth. In the experiments with real data, it's unclear how good the uncertainties produced by the model are. It may be useful to compare to the uncertainty produced by a GP with suitable kernels. In Fig 6c, the 95pct CI looks more or less constant over time. Is there an explanation for that?
Clarity: The paper is well-written. The presentations of the ideas are pretty clear.
Originality: Above average. I think the regularisation techniques proposed to preserve the ordering of the discretised class label are quite clever.
Significance: Average. It would be excellent if the authors can extend this to higher dimensional time series.
I'm unsure about the correctness of Algorithm 1 as I don't have knowledge in SMC.


Published as a conference paper at ICLR 2018 ROUTING NETWORKS: ADAPTIVE SELECTION OF NON-LINEAR FUNCTIONS FOR MULTI-TASK LEARN- ING
Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer. To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm. A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks. A function block may be any neural network -for example a fully-connected or a convolutional layer. Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached. In this way the routing network dynamically composes different function blocks for each input. We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks. We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a significant improvement in accuracy, with sharper convergence. In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks. On CIFAR-100 (20 tasks) we obtain cross-stitch performance levels with an 85% reduction in training time.

Summary:
The paper suggests to use a modular network with a controller which makes decisions, at each time step, regarding the next nodule to apply. This network is suggested a tool for solving multi-task scenarios, where certain modules may be shared and others may be trained independently for each task. It is proposed to learn the modules with standard back propagation and the controller with reinforcement learning techniques, mostly tabular.
-	page 4:
In algorithm 2, line 6, I do not understand the reward computation. It seems that either a _{k+1} subscript index is missing for the right hand side R, or an exponent of n-k is missing on \gamma. In the current formula, the final reward affects all decisions without a decay based on the distance between action and reward gain. This issue should be corrected or explicitly stated.
The ‘collaboration reward’ is not clearly justified: If I understand correctly, It is stated that actions which were chosen often in the past get higher reward when chosen again. This may create a ‘winner takes all’ effect, but it is not clear why this is beneficial for good routing. Specifically, this term is optimized when a single action is always chosen with high probability – but such a single winner does not seem to be the behavior we want to encourage.
-	Page 5: It is not described clearly (and better: defined formally) what exactly is the state representation. It is said to include the current network output (which is a vector in R^d), the task label and the depth, but it is not stated how this information is condensed into a single integer index for the tabular methods. If I understand correctly, the state representation used in the tabular algorithms includes only the current depth. If this is true, this constitutes a highly restricted controller, making decisions only based on depth without considering the current output.
-	The functional approximation versions are even less clear: Again it is not clear what information is contained in the state and how it is represented. In addition it is not clear in this case what network architecture is used for computation of the policy (PG) or valkue (Q-learning), and how exactly they are optimized.
-	The WPL algorithm is not clear to me
o	 In algorithm box 3, what is R_k? I do not see it defined anywhere. Is it related to \hat{R}? how?
o	Is it assumed that the actions are binary?
o	I do not understand why positive gradients are multiplied with the action probability and negative gradients with 1 minus this probability. What is the source of a-symmetry between positive and negative gradients?
-	Page 6:
o	It is not clear why MNist is tested over 200 examples, where there is a much larger test set available
o	In MIN-MTL I do not understand the motivation from creating superclasses composed of 5 random classes each: why do we need such arbitrary and un-natural class definitions?
-	Page 7:
The results on Cifar-100 are compared to several baselines, but not to the standard non-MTL solution: Solve the multi-class classification problem using a softmax loss and a unified, non routing architecture in which all the layers are shared by all classes, with the only distinction in the last classification layer. If the routing solution does not beat this standard baseline, there is no justification for its more complex structure and optimization.
-	Page 8: The author report that when training the controller with single agent methods the policy collapses into choosing a single module for most tasks. However, this is not-surprising, given that the action-based reward (whos strength is unclear) seems to promote such winner-takes-all behavior.
Overall:
-	The paper is highly unclear in its method representation
o	There is no unified clear notation. The essential symbols (states, actions, rewards) are not formally defined, and often it is not clear even if they are integers, scalars, or vectors. In notation existing, there are occasional notation errors.
o	The reward is a) not clear, and b) not well motivated when it is explained, and c) not explicitly stated anywhere: it is said that the action-specific reward may be up to 10 times larger than the final reward, but the actual tradeoff parameter between them is not stated. Note that this parameter is important, as using a 10-times larger action-related reward means that the classification-related reward becomes insignificant.
o	The state representation used is not clear, and if I understand correctly, it includes only the current depth. This is a severely limited state representation, which does not enable to learn actions based on intermediate results
o	The continuous versions of the RL algorithms are not explained at all: no state representation, nor optimization is described.
o	The presentation suffers from severe over-generalization and lack of clarity, which disabled my ability to understand the network and algorithms for a specific case. Instead, I would recommend that in future versions of this document a single network, with a specific router  and set of decisions, and  with a single algorithm, will be explained with clear notation end-to-end
Beyond the clarity issues, I suspect also that the novelty is minor (if the state does not include any information about the current output) and that the empirical baseline is lacking. However, it is hard to judge these due to lack of clarity.
After revision:
- Most of the clarity issues were handled well, and the paper now read nicely
- It is now clear that routing is not done based on the current input (an example is not dynamically routed based on its current representation). Instead routing depends on the task and depth only.  This is still interesting, but is far from reaching context-dependent routing.
- The results presented are nice and show that task-dependent routing may be better than plain baseline or the stiching alternative.  However, since this is a task transfer issue, I believe several data size points should be tested. For example, as data size rises, the task-specific-all-fc alternative is expected to get stronger (as with more data, related task are less required for good performance).

-


This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive models for raw audio waveform generation. As an example, we propose a hybrid model that combines an autoregressive network named WaveNet and a conventional LSTM model to address speech synthesis. Instead of generating one sample per time-step, the proposed HybridNet generates multiple samples per time-step by exploiting the long-term memory utilization property of LSTMs. In the evaluation, when applied to text-to-speech, HybridNet yields state-of-art performance. HybridNet achieves a 3.83 subjective 5-scale mean opinion score on US English, largely outperforming the same size WaveNet in terms of naturalness and provide 2x speed up at inference.

This paper presents HybridNet, a neural speech (and other audio) synthesis system (vocoder) that combines the popular and effective WaveNet model with an LSTM with the goal of offering a model with faster inference-time audio generation.
Summary: The proposed model, HybridNet is a fairly straightforward variation of WaveNet and thus the paper offers a relatively low novelty. There is also a lack of detail regarding the human judgement experiments that make the significance of the results difficult to interpret.
Low novelty of approach / impact assessment:
The proposed model is based closely on WaveNet, an existing state-of-the-art vocoder model. The proposal here is to extend WaveNet to include an LSTM that will generate samples between WaveNet samples -- thus allowing WaveNet to sample at a lower sample frequency. WaveNet is known for being relatively slow at test-time generation time, thus allowing it to run at a lower sample frequency should decrease generation time. The introduction of a local LSTM is perhaps not a sufficiently significant innovation.
Another issue that lowers the assessment of the likely impact of this paper is that there are already a number of alternative mechanism to deal with the sampling speed of WaveNet. In particular, the cited method of Ramachandran et al (2017) uses caching and other tricks to achieve a speed up of 21 times over WaveNet (compared to the 2-4 times speed up of the proposed method). The authors suggest that these are orthogonal strategies that can be combined, but the combination is not attempted in this paper. There are also other methods such as sampleRNN (Mehri et al. 2017) that are faster than WaveNet at inference time. The authors do not compare to this model.
Inappropriate evaluation:
While the model is motivated by the need to reduce the generation of WaveNet sampling, the evaluation is largely based on the quality of the sampling rather than the speed of sampling. The results are roughly calibrated to demonstrate that HybridNet produces higher quality samples when (roughly) adjusted for sampling time. The more appropriate basis of comparison is to compare sample time as a function of sample quality.
Experiments:
Few details are provided regarding the human judgment experiments with Mechanical Turkers. As a result it is difficulty to assess the appropriateness of the evaluation and therefore the significance of the findings. I would also be much more comfortable with this quality assessment if I was able to hear the samples for myself and compare the quality of the WaveNet samples with HybridNet samples. I will also like to compare the WaveNet samples generated by the authors' implementation with the WaveNet samples posted by van den Oord et al (2017).
Minor comments / questions:
How, specifically, is validation error defined in the experiments?
There are a few language glitches distributed throughout the paper.


Most deep latent factor models choose simple priors for simplicity, tractability or not knowing what prior to use. Recent studies show that the choice of the prior may have a profound effect on the expressiveness of the model, especially when its generative network has limited capacity. In this paper, we propose to learn a proper prior from data for adversarial autoencoders (AAEs). We introduce the notion of code generators to transform manually selected simple priors into ones that can better characterize the data distribution. Experimental results show that the proposed model can generate better image quality and learn better disentangled representations than AAEs in both supervised and unsupervised settings. Lastly, we present its ability to do cross-domain translation in a text-to-image synthesis task.

This paper propose a simple extension of the adversarial auto-encoders for (conditional) image generation. The general idea is that instead of using Gaussian prior, the propose algorithm uses a "code generator" network  to warp the gaussian distribution, such that the internal prior of the latent encoding space is more expressive and complicated.
Pros:
- The proposed idea is simple and easy to implement
- The results show improvement in terms of visual quality
Cons:
- I agree that the proposed prior should better capture the data distribution. However, incorporating a generic prior over the latent space plays a vital role as regularisation, this helps avoid model collapse. Adding a complicated code generation network brings too much flexibility for the prior part. This makes the prior and posterior learnable, which makes it easier to fool the regularisation discriminator (think about the latent code and prior code collapsed to two different points). As a result, this weakens the regularisation over the latent encoder space.
- The above mentioned could be verified through qualitative results. As shown in Fig. 5. I believe this is a result due to the fact that the adversarial loss in the regularisation phase does not a significant influence there.
- I have some doubts over why AAE works so poorly when the latent dimension is 2000. How to make sure it's not a problem of implementation or the model wasn't trapped into a bad local optima / saddle points. Could you justify this?
- Contributions; this paper propose an improvement over a existing model. However, neither the idea/insights it brought can be applied onto other generative models, nor the improvement bring a significant improvement over the-state-of-the-arts. I am wondering what the community will learn from this paper, or what the author would like to claim as significant contributions.


Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from inthe-loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia that is an order of magnitude larger than comparable datasets. By applying policybased reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

This paper presents a new approach to support the conversion from natural language to database queries.
One of the major contributions of the work is the introduction of a new real-world benchmark dataset based on questions over Wikipedia. The scale of the data set is significantly larger than any existing ones. However, from the technical perspective, the reviewer feels this work has limited novelty and does not advance the research frontier by much. The detailed comments are listed below.
1) Limitation of the dataset: While the authors claim this is a general approach to support seq2sql, their dataset only covers simple queries in form of aggregate-where-select structure. Therefore, their proposed approach is actually an advanced version of template filling, which considers the expression/predicate for one of the three operators at a time, e.g., (Giordani and Moschitti, 2012).
2) Limitation of generalization: Since the design of the algorithms is purely based on their own WikiSQL dataset, the reviewer doubts if their approach could be generalized to handle more complicated SQL queries, e.g., (Li and Jagadish, 2014). The high complexity of real-world SQL stems from the challenges on the appropriate connections between tables with primary/foreign keys and recursive/nested queries.
3) Comparisons to existing approaches: Since it is a template-based approach in nature, the author should shrink the problem scope in their abstract/introduction and compare against existing template approaches. While there are tons of semantic parsing works, which grow exponentially fast in last two years, these works are actually handling more general problems than this submission does. It thus makes sense when the performance of semantic parsing approaches on a constrained domain, such as WikiSQL, is not comparable to the proposal in this submission. However, that only proves their method is fully optimized for their own template.
As a conclusion, the reviewer believes the problem scope they solve is much smaller than their claim, which makes the submission slightly below the bar of ICLR. The authors must carefully consider how their proposed approach could be generalized to handle wider workload beyond their own WikiSQL dataset.
PS, After reading the comments on OpenReview, the reviewer feels recent studies, e.g., (Guu et al., ACL 2017), (Mou et al, ICML 2017) and (Yin et al., IJCAI 2016), deserve more discussions in the submission because they are strongly relevant and published on peer-reviewed conferences.


EIGENOPTION DISCOVERY THROUGH THE DEEP SUCCESSOR REPRESENTATION
Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning. However, autonomously learning effective sets of options is still a major challenge in the field. In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process. Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment. We extend the existing algorithms for eigenoption discovery to settings with stochastic transitions and in which handcrafted features are not available. We propose an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels. It exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation. We use traditional tabular domains to provide intuition about our approach and Atari 2600 games to demonstrate its potential.

Eigenoption Discovery Through the Deep Successor Representation
The paper is a follow up on previous work by Machado et al. (2017) showing how proto-value functions (PVFs) can be used to define options called “eigenoptions”. In essence, Machado et al. (2017) showed that, in the tabular case, if you interpret the difference between PVFs as pseudo-rewards you end up with useful options. They also showed how to extend this idea to the linear case: one replaces the Laplacian normally used to build PVFs with a matrix formed by sampling differences phi(s') - phi(s), where phi are features. The authors of the current submission extend the approach above in two ways: they show how to deal with stochastic dynamics and how to replace a linear model with a nonlinear one. Interestingly, the way they do so is through the successor representation (SR). Stachenfeld et al. (2014) have showed that PVFs can be obtained as a linear transformation of the eigenvectors of the matrix formed by stacking all SRs of an MDP. Thus, if we have the SR matrix we can replace the Laplacian mentioned above. This provides benefits already in the tabular case, since SRs naturally extend to domains with stochastic dynamics. On top of that, one can apply a trick similar to the one used in the linear case --that is,  construct the matrix representing the diffusion model by simply stacking samples of the SRs. Thus, if we can learn the SRs, we can extend the proposed approach to the nonlinear case. The authors propose to do so by having a deep neural network similar to Kulkarni et al. (2016)'s Deep Successor Representation. The main difference is that, instead of using an auto-encoder, they learn features phi(s) such that the next state s' can be recovered from it (they argue that this way psi(s) will retain information about aspects of the environment the agent has control over).
This is a well-written paper with interesting (and potentially useful) insights. I only have a few comments regarding some aspects of the paper that could perhaps be improved, such as the way eigenoptions are evaluated.
One question left open by the paper is the strategy used to collect data in order to compute the diffusion model (and thus the options). In order to populate the matrix that will eventually give rise to the PVFs the agent must collect transitions. The way the authors propose to do it is to have the agent follow a random policy. So, in order to have options that lead to more direct, "purposeful" behaviour, the agent must first wander around in a random, purposeless, way, and hope that this will lead to a reasonable exploration of the state space.
This problem is not specific to the proposed approach, though: in fact, any method to build options will have to resolve the same issue. One related point that is perhaps more specific to this particular work is the strategy used to evaluate the options built: the diffusion time, or the expected number of steps between any two states of an MDP when following a random walk. First, although this metric makes intuitive sense, it is unclear to me how much it reflects control performance, which is what we ultimately care about. Perhaps more important, measuring performance using the same policy used to build the options (the random policy) seems somewhat unsatisfactory to me. To see why, suppose that the options were constructed based on data collected by a non-random policy that only visits a subspace of the state space. In this case it seems likely that the decrease in the diffusion time would not be as apparent as in the experiments of the paper. Conversely, if the diffusion time were measured under another policy, it also seems likely that options built with a random policy would not perform so well (assuming that the state space is reasonably large to make an exhaustive exploration infeasible). More generally, we want options built under a given policy to reduce the diffusion time of other policies (preferably ones that lead to good control performance).
Another point associated with the evaluation of the proposed approach is the method used to qualitatively assess options in the Atari experiments described in Section 4.2. In the last paragraph of page 7 the authors mention that eigenoptions are more effective in reducing the diffusion time than “random options” built based on randomly selected sub-goals. However, looking at Figure 4, the terminal states of the eigenoptions look a bit like randomly-selected  sub-goals. This is especially true when we note that only a subset of the options are shown: given enough random options, it should be possible to select a subset of them that are reasonably spread across the state space as well.
Interestingly, one aspect of the proposed approach that seems to indeed be an improvement over random options is made visible by a strategy used by the authors to circumvent computational constraints. As explained in the second paragraph of page 8, instead of learning policies to maximize the pseudo-rewards associated with eigenoptions the authors used a myopic policy that only looks one step ahead (which is the same as having a policy learned with a discount factor of zero). The fact that these myopic policies are able to navigate to specific locations and stay there suggests that the proposed approach gives rise to dense pseudo-rewards that are very informative. As a comparison, when we define a random sub-goal the resulting reward is a very sparse signal that would almost certainly not give rise to useful myopic policies. Therefore, one could argue that the proposed approach not only generate useful options, it also gives rise to dense pseudo-rewards that make it easier to build the policies associated with them.