STOCHASTIC VARIATIONAL VIDEO PREDICTION
Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images require the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world videos. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication.

1) Summary
This paper proposed a new method for predicting multiple future frames in videos. A new formulation is proposed where the frames’ inherent noise is modeled separate from the uncertainty of the future. This separation allows for directly modeling the stochasticity in the sequence through a random variable z ~ p(z)  where the posterior  q(z | past and future frames) is approximated by a neural network, and as a result, sampling of a random future is possible through sampling from the prior p(z) during testing. The random variable z can be modeled in a time-variant and time-invariant way. Additionally, this paper proposes a training procedure to prevent their method from ignoring the stochastic phenomena modeled by z. In the experimental section, the authors highlight the advantages of their method in 1) a synthetic dataset of shapes meant to clearly show the stochasticity in the prediction, 2) two robotic arm datasets for video prediction given and not given actions, and 3) A challenging human action dataset in which they perform future prediction only given previous frames.
2) Pros:
+ Novel/Sound future frame prediction formulation and training for modeling the stochasticity of future prediction.
+ Experiments on the synthetic shapes and robotic arm datasets highlight the proposed method’s power of multiple future frame prediction possible.
+ Good analysis on the number of samples improving the chance of outputting the correct future, the modeling power of the posterior for reconstructing the future, and a wide variety of qualitative examples.
+ Work is significant for the problem of modeling the stochastic nature of future frame prediction in videos.

3) Cons:
Approximate posterior in non-synthetic datasets:
The variable z seems to not be modeling the future very well. In the robot arm qualitative experiments, the robot motion is well modeled, however, the background is not. Given that for the approximate posterior computation the entire sequence is given (e.g. reconstruction is performed), I would expect the background motion to also be modeled well. This issue is more evident in the Human 3.6M experiments, as it seems to output blurriness regardless of the true future being observed. This problem may mean the method is failing to model a large variety of objects and clearly works for the robotic arm because a very similar large shape (e.g. robot arm) is seen in the training data. Do you have any comments on this?
Finn et al 2016 PNSR performance on Human 3.6M:
Is the same exact data, pre-processing, training, and architecture being utilized? In her paper, the PSNR for the first timestep on Human 3.6M is about 41 (maybe 42?)  while in this paper it is 38.
Additional evaluation on Human 3.6M:
PSNR is not a good evaluation metric for frame prediction as it is biased towards blurriness, and also SSIM does not give us an objective evaluation in the sense of semantic quality of predicted frames. It would be good if the authors present additional quantitative evaluation to show that the predicted frames contain useful semantic information [1, 2, 3, 4]. For example, evaluating the predicted frames for the Human 3.6M dataset to see if the human is still detectable in the image or if the expected action is being predicted could be useful to verify that the predicted frames contain the expected meaningful information compared to the baselines.
Additional comments:
Are all 15 actions being used for the Human 3.6M experiments? If so, the fact of the time-invariant model performs better than the time-variant one may not be the consistent action being performed (last sentence of 5.2). The motion performed by the actors in each action highly overlaps (talking on the phone action may go from sitting to walking a little to sitting again, and so on). Unless actions such as walking and discussion were only used, it is unlikely the time-invariant z is performing better because of consistent action. Do you have any comments on this?
4) Conclusion
This paper proposes an interesting novel approach for predicting multiple futures in videos, however, the results are not fully convincing in all datasets. If the authors can provide additional quantitative evaluation besides PSNR and SSIM (e.g. evaluation on semantic quality), and also address the comments above, the current score will improve.
References:
[1] Emily Denton and Vighnesh Birodkar. Unsupervised Learning of Disentangled Representations from Video. In NIPS, 2017.
[2] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICML, 2017.
[3] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv preprint arXiv:1710.10196, 2017.
[4] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NIPS, 2017.
Revised review:
Given the authors' thorough answers to my concerns, I have decided to change my score. I would like to thank the authors for a very nice paper that will definitely help the community towards developing better video prediction algorithms that can now predict multiple futures.


AUTO-ENCODING SEQUENTIAL MONTE CARLO
We build on auto-encoding sequential Monte Carlo (AESMC):
1 a method for model and proposal learning based on maximizing the lower bound to the log marginal likelihood in a broad family of structured probabilistic models. Our approach relies on the efficiency of sequential Monte Carlo (SMC) for performing inference in structured probabilistic models and the flexibility of deep neural networks to model complex conditional probability distributions. We develop additional theoretical insights and experiment with a new training procedure which can improve both model and proposal learning. We demonstrate that our approach provides a fast, easy-to-implement and scalable means for simultaneous model learning and proposal adaptation in deep generative models.

Update:
On further consideration (and reading the other reviews), I'm bumping my rating up to a 7. I think there are still some issues, but this work is both valuable and interesting, and it deserves to be published (alongside the Naesseth et al. and Maddison et al. work).
-----------
This paper proposes a version of IWAE-style training that uses SMC instead of classical importance sampling. Going beyond the several papers that proposed this simultaneously, the authors observe a key issue: the variance of the gradient of these IWAE-style bounds (w.r.t. the inference parameters) grows with their accuracy. They therefore propose using a more-biased but lower-variance bound to train the inference parameters, and the more-accurate bound to train the generative model.
Overall, I found this paper quite interesting. There are a few things I think could be cleared up, but this seems like good work (although I'm not totally up to date on the very recent literature in this area).
Some comments:
* Section 4: I found this argument extremely interesting. However, it’s worth noting that your argument implies that you could get an O(1) SNR by averaging K noisy estimates of I_K. Rainforth et al. suggest this approach, as well as the approach of averaging K^2 noisy estimates, which the theory suggests may be more appropriate if the functions involved are sufficiently smooth, which even for ReLU networks that are non-differentiable at a finite number of points I think they should be.
This paper would be stronger if it compared with Rainforth et al.’s proposed approaches. This would demonstrate the real tradeoffs between bias, variance, and computation. Of course, that involves O(K^2) or O(K^3) computation, which is a weakness. But one could use a small value of K (say, K=5).
That said, I could also imagine a scenario where there is no benefit to generating multiple noisy samples for a single example versus a single noisy sample for multiple examples. Basically, these all seem like interesting and important empirical questions that would be nice to explore in a bit more detail.
* Section 3.3: Claim 1 is an interesting observation. But Propositions 1 and 2 seem to just say that the only way to get a perfectly tight SMC ELBO is to perfectly sample from the joint posterior. I think there’s an easier way to make this argument:
Given an unbiased estimator \hat{Z} of Z, by Jensen’s inequality E[log \hat{Z}] ≤ log Z, with equality iff the variance of \hat{Z} = 0. The only way to get an SMC estimator’s variance to 0 is to drive the variance of the weights to 0. That only happens if you perfectly sample each particle from the true posterior, conditioned on all future information.
All of which is true as far as it goes, but I think it’s a bit of a distraction. The question is not “what’s it take to get to 0 variance” but “how quickly can we approach 0 variance”. In principle IS and SMC can achieve arbitrarily high accuracy by making K astronomically large. (Although [particle] MCMC is probably a better choice if one wants extremely low bias.)
* Section 3.2: The choice of how to get low-variance gradients through the ancestor-sampling choice seems seems like an important technical challenge in getting this approach to work, but there’s only a very cursory discussion in the main text. I would recommend at least summarizing the main findings of Appendix A in the main text.
* A relevant missing citation: Turner and Sahani’s “Two problems with variational expectation maximisation for time-series models” (http://www.gatsby.ucl.ac.uk/~maneesh/papers/turner-sahani-2010-ildn.pdf). They discuss in detail some examples where tighter variational bounds in state-space models lead to worse parameter estimates (though in a quite different context and with a quite different analysis).
* Figure 1: What is the x-axis here? Presumably phi is not actually 1-dimensional?
Typos etc.:
* “learn a particular series intermediate” missing “of”.
* “To do so, we generate on sequence y1:T” s/on/a/, I think?
* Equation 3: Should there be a (1/K) in Z?


ADVERSARIAL DROPOUT REGULARIZATION
We present a domain adaptation method for transferring neural representations from label-rich source domains to unlabeled target domains. Recent adversarial methods proposed for this task learn to align features across domains by "fooling" a special domain classifier network. However, a drawback of this approach is that the domain classifier simply labels the generated features as in-domain or not, without considering the boundaries between classes. This means that ambiguous target features can be generated near class boundaries, reducing target classification accuracy. We propose a novel approach, Adversarial Dropout Regularization (ADR), which encourages the generator to output more discriminative features for the target domain. Our key idea is to replace the traditional domain critic with a critic that detects non-discriminative features by using dropout on the classifier network. The generator then learns to avoid these areas of the feature space and thus creates better features. We apply our ADR approach to the problem of unsupervised domain adaptation for image classification and semantic segmentation tasks, and demonstrate significant improvements over the state of the art.

Unsupervised Domain adaptation is the problem of training a classifier without labels in some target domain if we have labeled data from a (hopefully) similar dataset with labels. For example, training a classifier using simulated rendered images with labels, to work on real images.
Learning discriminative features for the target domain is a fundamental problem for unsupervised domain adaptation. The problem is challenging (and potentially ill-posed) when no labeled examples are given in the target domain. This paper proposes a new training technique called ADR, which tries to learn discriminative features for the target domain. The key idea of this technique is to move the target-domain features away from the source-domain decision boundary. ADR achieves this goal by encouraging the learned features to be robust to the dropout noise applied to the classifier.
My main concern about this paper is that the idea of "placing the target-domain features far away from the source-domain decision boundary" does not necessarily lead to *discriminative features* for the target domain. In fact, it is easy to come up with a counter-example: the target-domain features are far from the *source-domain* decision boundary, but they are all (both the positive and negative examples) on the same side of the boundary, which leads to poor target classification accuracy. The loss function (Equations 2-5) proposed in the paper does not prevent the occurrence of this counter-example.
Another concern comes from using the proposed idea in training a GAN (Section 4.3). Generating fake images that are far away from the boundary (as forced by the first term of Equation 9) is somewhat opposite to the objective of GAN training, which aims at aligning distributions of real and fake images. Although the second term of Equation 9 tries to make the generated and the real images similar, the paper does not explain how to properly balance the two terms of Equation 9. As a result, I am worried that the proposed method may lead to more mode-collapsing for GAN.
The experimental evaluation seems solid for domain adaptation. The semi-supervised GANs part seemed significantly less developed and might be weakening rather than strengthening the paper.
Overall the performance of the proposed method is quite well done and the results are encouraging, despite the lack of theoretical foundations for this method.


Multi-view recognition is the task of classifying an object from multi-view image sequences. Instead of using a single-view for classification, humans generally navigate around a target object to learn its multi-view representation. Motivated by this human behavior, the next best view can be learned by combining object recognition with navigation in complex environments. Since deep reinforcement learning has proven successful in navigation tasks, we propose a novel multi-task reinforcement learning framework for joint multi-view recognition and navigation. Our method uses a hierarchical action space for multi-task reinforcement learning. The framework was evaluated with an environment created from the ModelNet40 dataset. Our results show improvements on object recognition and demonstrate human-like behavior on navigation.

The ambition of this paper is to address multi-view object recognition and the associated navigation as a unified reinforcement learning problem using a deep CNN to represent the policy.
Multi-view recognition and active viewpoint selection have been studied for more than 30 years, but this paper ignores most of this history.  The discussion of related work as well as the empirical evaluation are limited to very recent methods using neural networks.  I encourage the authors to look e.g. at Paletta and Pinz [1] (who solve a very similar and arguably harder problem in related ways) and at Bowyer & Dyer [2] as well as the references contained in these papers for history and context.  Active vision goes back to Bajcsy, Aloimonos, and Ballard; these should be cited instead of Ammirato et al.  Conversely, the related work cites a handful of papers (e.g. in the context of Atari 2600 games) that are unrelated to this work.
The navigation aspect is limited to fixed-size left or right displacements (at least for ModelNet40 task which is the only one to be evaluated and discussed).  This is strictly weaker than active viewpoint selection.  Adding this to the disregard of prior work, it is (at best) misleading to claim that this is "the first framework to combine learning of navigation and object recognition".
Calling this "multi-task" learning is also misleading.  There is only one ultimate objective (object recognition), while the agent has two types of actions available (moving or terminating with a classification decision).
There are other misleading, vague, or inaccurate statements in the paper, for example:
- "With the introduction of deep learning to reinforcement learning, there has been ... advancements in understanding ... how humans navigate": I don't think such a link exists; if it does, a citation needs to be provided.
- "inductive bias like image pairs": Image pairs do not constitute inductive bias.  Either the term is misused or the wording must be clarified; likewise for other occurrences of "inductive bias".
- "a single softmax layer is biased towards tasks with larger number of actions": I think I understand what this is intended to say, but a "softmax layer" cannot be "biased towards tasks" as there is only one, given, task.
- I do not understand what the stated contribution of "extrapolation of the action space to a higher dimension for multi-task learning" is meant to be.
- "Our method performs better ... than state-of-the-art in training for navigation to the object": The method does not involve "navigation to the object", at least not for the ModelNet40 dataset, the only for which results are given.
It is not clear what objective function the system is intended to optimize.  Since the stated task is object recognition and from Table 2 I was expecting it to be the misclassification rate, but this is clearly not the case, as the system is not set up to minimize it.  What "biases" the system towards classification actions (p. 5)?  Why is it bad if the agent shows "minimal movement actions" as long as the misclassification rate is minimized? No results are given to show whether this is the case or not.  The text then claims that the "hierarchical method gives superior results", but this is not shown either.
Table 3 reveals that the system fails to learn much of interest at all.  Much of the time the agent chooses not to move and performs relatively poorly; taking more steps improves the results; often all 12 views are collected before a classification decision is made.  Two of the most important questions remain open: (1) What would be the misclassification rate if all views are always used? (2) What would be the misclassification rate under a random baseline policy not involving navigation learning (e.g., taking a random number of steps in the same direction)?
Experiments using the THOR dataset are announced but are left underspecified (e.g., the movement actions), but no results or discussion are given.
SUMMARY
Quality: lacking in may ways; see above.
Clarity: Most of the paper is clear enough, but there are confusions and missing information about THOR and problems with phrasing and terminology.  Moreover, there are many grammatical and typographical glitches.
Originality: Harder tasks have been looked at before (using methods other than CNN).  Solving a simpler version using CNN I do not consider original unless there is a compelling pay-off, which this paper does not provide.
Significance: Low.
Pros: The problem would be very interesting and relevant if it was formulated in a more ambitious way (e.g., a more elaborate action space than that used for ModelNet40) with a clear objective function,
Cons: See above.
[1] Lucas Paletta and Axel Pinz, Active object recognition by view integration and reinforcement learning, Robotics and Autonomous Systems 31, 71-86, 2000
[2] Bowyer, K. W. and Dyer, C. R. (1990), Aspect graphs: An introduction and survey of recent results. Int. J. Imaging Syst. Technol., 2: 315–328. doi:10.1002/ima.1850020407


Automated metrics such as BLEU are widely used in the machine translation literature. They have also been used recently in the dialogue community for evaluating dialogue response generation. However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting. Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity. It is thus reasonable to think that these automated metrics would correlate well with human judgment in the task-oriented setting where the generation task consists of translating dialogue acts into a sentence. We conduct an empirical study to confirm whether this is the case. Our findings indicate that these automated metrics have stronger correlation with human judgments in the task-oriented setting compared to what has been observed in the non task-oriented setting. We also observe that these metrics correlate even better for datasets which provide multiple ground truth reference sentences. In addition, we show that some of the currently available corpora for task-oriented language generation can be solved with simple models and advocate for more challenging datasets.

The authors present a solid overview of unsupervised metrics for NLG, and perform a correlation analysis between these metrics and human evaluation scores on two task-oriented dialog generation datasets using three LSTM-based models. They find weak but statistically significant correlations for a subset of the evaluated metrics, an improvement over the situation that has been observed in open-domain dialog generation.
Other than the necessarily condensed model section (describing a model explained at greater length in a different work) the paper is quite clear and well-written throughout, and the authors' explication of metrics like BLEU and greedy matching is straightforward and readable. But the novel work in the paper is limited to the human evaluations collected and the correlation studies run, and the authors' efforts to analyze and extend these results fall short of what I'd like to see in a conference paper.
Some other points:
1. Where does the paper's framework for response generation (i.e., dialog act vectors and delexicalized/lexicalized slot-value pairs) fit into the landscape of task-oriented dialog agent research? Is it the dominant or state-of-the-art approach?
2. The sentence "This model is a variant of the “ld-sc-LSTM” model proposed by Sharma et al. (2017) which is based on an encoder-decoder framework" is ambiguous; what is apparently meant is that Sharma et al. (2017) introduced the hld-scLSTM, not simply the ld-scLSTM.
3. What happens to the correlation coefficients when exact reference matches (a significant component of the highly-rated upper right clusters) are removed?
4. The paper's conclusion naturally suggests the question of whether these results extend to more difficult dialog generation datasets. Can the authors explain why the datasets used here were chosen over e.g. El Asri et al. (2017) and Novikova et al. (2016)?


Published as a conference paper at ICLR 2018 DEEP BAYESIAN BANDITS SHOWDOWN AN EMPIRICAL COMPARISON OF BAYESIAN DEEP NETWORKS FOR THOMPSON SAMPLING
Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting.

If two major questions below are answered affirmatively, I believe this article could be very good contribution to the field and deserve publication in ICLR.
In this article the authors provide a service to the community by comparing the current most used algorithms for Thompson Sampling-based contextual (parametric) bandits on clear empirical benchmark. They reimplement the key algorithms, investing time to make up for the lack of published source code for some.
After a clear exposure of the reasons why Thompson Sampling is attractive, they overview concisely the key ideas behind 7 different families of algorithms, with proper literature review. They highlight some of the subtleties of benchmarking bandit problems (or any active learning algorithms for that matter): the lack of counterfactual and hence the difference in observed datasets. They explain their benchmark framework and datasets, then briefly summarise the results for each class of algorithms. Most of the actual measures from the benchmark are provided in a lengthy appendix 12 pages appendix choke-full of graphs and tables.
It is refreshing to see an article that does not boast to offer the new "bestest-ever" algorithm in town, overcrowding a landscape, but instead tries to prune the tree of possibilities and wading through other people's inflated claims. To the authors: thank you! It is too easy to dismiss these articles as "pedestrian non-innovative groundwork": if there were more like it, our field would certainly be more readable and less novelty-prone.
Of course, there is no perfect benchmark, and like every benchmark, the choices made by the authors could be debated to no end. At least, the authors try to explain them, and the tradeoffs they faced, as clearly as possible (except for two points mentioned below), which again is too rare in our field.
Major clarifications needed:
My two key questions are:
* Is the code of good quality, with exact  reproducibility and good potential extension in a standard language (e.g. Python)? This benchmark only gets its full interest if the code is publicised and well engineered. The open-sourcing is planned, according to footnote 1, is planned -- but this should be made clearer in the main text. There is no discussion of the engineering quality, not even of the language used, and this is quite important if the authors want the community to build upon this work. The code was not submitted for review, and as such its accessibility to new contributors is unknown to this reviewer. That could be a make or break feature of this work.
* Is the hyper parameter tuning reproducible? Hyperparameter tuning should be discussed much more clearly (in the Appendix): while I appreciate the discussion page 8 of how they were frozen across datasets, "they were chosen through careful tuning" is way too short. What kind of tuning? Was it  manual, and hence not reproducible? Or was it a clear, reproducible grid search or optimiser? I thoroughly hope for the later, otherwise an unreproducible benchmark would be very
If the answers to the two questions above is "YES", then brilliant article, I am ready to increase my score. However, if either is a "NO", I am afraid that would limit to how much this benchmark will serve as a reference (as opposed to "just one interesting datapoint").
Minor improvements:
* Please proofread some obvious typos:
  - page 4 "suggesed" -> "suggested",
  - page 8 runaway math environment wreaking the end of the sentence.
  - reference "Meire Fortunato (2017)" should be  "Fortunato et al. (2017)", throughout.
* Improve readability of figures' legends, e.g. Figure 2.(b) key is un-readable.
* A simple table mapping the name of the algorithm to the corresponding article is missing. Not everyone knows what BBB and BBBN stands for.
* A measure of wall time would be needed: while computational cost is often mentioned (especially as a drawback to getting proper performance out of variational inference), it is nowhere plotted. Of course that would partly depend on the quality of the implementation, but this is somewhat mitigated if all the algorithms have been reimplemented by the authors (is that the case? please clarify).


Workshop track -ICLR 2018 LSH-SAMPLING BREAKS THE COMPUTA- TIONAL CHICKEN-AND-EGG LOOP IN ADAP- TIVE STOCHASTIC GRADIENT ESTIMATION
Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems. SGD estimates the gradient by uniform sampling with sample size one. There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates. Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient. As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop. In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling. Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently. As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms. We demonstrate the benefits of our proposal on both SGD and AdaGrad.

The main contribution of this work is just a combination of LSH schemes and SGD updates. Since hashing schemes essentially reduce the dimension, LSH brings computational benefits to the SGD operation. The targeted issue is fundamentally important, and the proposed approach (exploiting LSH schemes) seems to be sound.  Specifically, LSH schemes fit into the SGD schemes since they hash two vectors to the same bucket with probability in proportional to their distance (here, inner product or Cosine similarity).
Strengths:  a sound approach; a simple and straightforward idea that is shown to work well in evaluations.
Weaknesses:
1. The phrase of "computational chicken-and-egg loop" in the title and also in the main body is misleading and not accurate. The so-called "chicken-and-egg” issue concerns the causality dilemma: two causally related things, which comes the first. In the paper, the authors concerned "more accurate gradients" and "faster convergence"; their causality is very clear (the first leads to the second), and there is no causality dilemma. Even from a computational perspective, "SDG schemes aim for computational efficiency" and "stochastic makes the convergence slow down" are not a causality dilemma.  The reason behind is that the latter is the cost of the first one, just the old saying that "there is no such thing as a free lunch". Therefore, this disordered logic makes the title very misleading, and all the corresponding descriptions in the main body are obscured by "twisted" and unnatural logics.

2. The depth is so limited. Besides a good observation that LSH fits well into SDG, there are no more in-depth results provided. The theorems (Theorems 1~3) are trivial, with loose relations with LSH.

3. The LSH schemes are not correctly referred to. Since the similarity metric is inner-product, the authors are expected to refer to Cosine similarity and inner-product based LSHs, which were published recently in NIPS. It is not in depth to assume "any known LSH scheme" in Alg. 2. Accordingly again, Theorems 1~3 are unrelated with this specific kind of similarity metric (Cosine similarity).
4. As the authors tried hard to stick to the unnecessary (a bit bragging) phrase "computational chicken-and-egg loop", the organization and presentation of the whole manuscript are poor.
5. Occasionally, there are typos, and it is not good to use words in formulas. Please proof-read carefully.


We examine how learning from unaligned data can improve both the data efficiency of supervised tasks as well as enable alignments without any supervision. For example, consider unsupervised machine translation: the input is two corpora of English and French, and the task is to translate from one language to the other but without any pairs of English and French sentences. To address this, we develop feature matching auto-encoders (FMAEs). FMAEs ensure that the marginal distribution of feature layers are preserved across forward and inverse mappings between domains. We show that FMAEs achieve state of the art for data efficiency and alignment across three tasks: text decipherment, sentiment transfer, and neural machine translation for English-to-German and English-to-French. Most compellingly, FMAEs achieve state of the art for semi-supervised neural machine translation with significant BLEU score differences of up to 5.7 and 6.3 over traditional supervised models. Furthermore, on English-to-German, FMAEs outperform last year's best models such as ByteNet (Kalchbrenner et al., 2016) while using only half as many supervised examples.

This paper proposes a generative model called matching auto-encoder to carry out the learning from unaligned data.
However, it is very disappointed to read the contents after the introduction, since most of the contributions are overclaimed.
Detailed comments:
- Figure 1 is incorrect because the pairs (x, z) and (y, z) should be put into two different plates if  x and y are unaligned.
- Lots of contents in Sec. 3 are confusing to me. What is the difference between g_l(x) and g_l(y) if g_l : H_{l−1} → H_l and f_l: H_{l−1} → H_l are the same? What are e_x and e_y? Why is there a λ if it is a generative model?
- If the title is called 'text decipherment', there should be no parallel data at all, otherwise it is a huge overclaim on the decipherment tasks. Please add citations of Kevin Knight's recent papers on deciperment.
- Reading the experiment results of 'Sentiment Transfer' is a disaster to me. I couldn't get much information on 'sentiment transfer' from a bunch of ungrammatical unnatural language sentences. I would prefer to see some results of baseline models for comparison instead of a pure qualitative analysis.
- The claim on "FMAEs are state of the art for neural machine translation with limited supervision on EN-DE and EN-FR" is not exciting to me. Semi-supervised learning is interesting, but in the scenario of MT we do have enough parallel data for many language pairs. Unless the model is able to exceed the 'real' state-of-the-art that uses the full set of parallel data, otherwise we couldn't identify whether the models are able to benefit NMT.  Interestingly, the authors didn't provide any of the results that are experimented with full parallel data set. Possibly it is because the introduction of stochastic variables that prevent the models from overfitting on small datasets.


A DIRT-T APPROACH TO UNSUPERVISED DOMAIN ADAPTATION
Domain adaptation refers to the problem of leveraging labeled data in a source domain to learn an accurate model in a target domain where labels are scarce or unavailable. A recent approach for finding a common representation of the two domains is via domain adversarial training (Ganin & Lempitsky, 2015), which attempts to induce a feature extractor that matches the source and target feature distributions in some feature space. However, domain adversarial training faces two critical limitations: 1) if the feature extraction function has high-capacity, then feature distribution matching is a weak constraint, 2) in non-conservative domain adaptation (where no single classifier can perform well in both the source and target domains), training the model to do well on the source domain hurts performance on the target domain. In this paper, we address these issues through the lens of the cluster assumption, i.e., decision boundaries should not cross high-density data regions. We propose two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes violation of the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) 1 model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation. Extensive empirical results demonstrate that the combination of these two models significantly improve the state-of-the-art performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation benchmarks.

As there are many kinds of domain adaptation problems, the need to mix several learning strategies to improve the existing approaches is obvious. However, this task is not necessarily easy to succeed. The authors proposed a sound approach to learn a proper representation (in an adversarial way) and comply the cluster assumption.
The experiments show that this Virtual Adversarial Domain Adaptation network (VADA) achieves great results when compared to existing learning algorithms. Moreover, we also see the learned model is consistently improved using the proposed "Decision-boundary Iterative Refinement Training with a Teacher" (DIRT-T) approach.
The proposed methodology relies on multiple choices that could sometimes be better studied and/or explained. Namely, I would like to empirically see which role of the locally-Lipschitz regularization term (Equation 7). Also, I wonder why this term is tuned by an hyperparameter (lamda_s) for the source, while a single hyperparamer (lambda_t) is used for the sum of the two target quantity.

On the theoretical side, the discussion could be improved. Namely, Section 3 about "limitation of domain adversarial training" correctly explained that "domain adversarial training may not be sufficient for domain adaptation if the feature extraction function has high-capacity". It would be interesting to explain whether this observation is consistent with Theorem 1 of the paper (due to Ben-David et al., 2010), on which several domain adversarial approaches are based. The need to consider supplementary assumptions (such as ) to achieve good adaptation can also be studied through the lens of more recent Ben-David's work, e.g. Ben-David and Urner (2014). In the latter, the notion of "Probabilistic Lipschitzness", which is a relaxation of the "cluster assumption" seems very related to the actual work.
Reference:
Ben-David and Urner. Domain adaptation-can quantity compensate for quality?, Ann. Math. Artif. Intell., 2014
Pros:
- Propose a sound approach to mix two complementary strategies for domain adaptation.
- Great empirical results.
Cons:
- Some choices leading to the optimization problem are not sufficiently explained.
- The theoretical discussion could be improved.
Typos:
- Equation 14: In the first term (target loss), theta should have an index t (I think).
- Bottom of page 6: "... and that as our validation set" (missing word).


While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging. In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space. In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations. ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar. This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation. Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art results in unaligned text style transfer task using only a shared continuous-space representation.

I was asked to contribute this review rather late in the process, and in order
to remain unbiased I avoided reading other reviews. I apologize if some of
these comments have already been addressed in replies to other reviewers.
This paper proposes a regularization strategy for autoencoders that is very
similar to the adversarial autoencoder of Makhzani et al. The main difference
appears to be that rather than using the classic GAN loss to shape the
aggregate posterior of an autoencoder to match a chosen, fixed distribution,
they instead employ a Wasserstein GAN loss (and associated weight magnitude
constraint, presumably enforced with projected gradient descent) on a system
where the matched distribution is instead learned via a parameterized sampler
("generator" in the GAN lingo). Gradient steps that optimize the encoder,
decoder and generator are interleaved. The authors apply an extension of this
method to topic and sentiment transfer and show moderately good latent space
interpolations between generated sentences.
The difference from the original AAE is rather small and straightforward, making the
novelty mainly in the choice of task, focusing on discrete vectors and sequences.
The exposition leaves ample room for improvement. For one thing, there is the
irksome and repeated use of "discrete structure" when discrete *sequences* are
considered almost exclusively (with the exception of discretized MNIST digits).
The paper is also light on discussion of related work other than Makhzani et al
-- the wealth of literature on combining autoencoders (or autoencoder-like
structures such as ALI/BiGAN) and GANs merits at least passing mention.
The empirical work is somewhat compelling, though I am not an expert in this
task domain. The annealed importance sampling technique of Wu et al (2017) for
estimating bounds on a generator's log likelihood could be easily applied in
this setting and would give (for example, on binarized MNIST) a quantitative
measurement of the degree of overfitting, and this would have been preferable
than inventing new heuristic measures. The "Reverse PPL" metric requires more
justification, and it looks an awful lot like the long-since-discredited Parzen
window density estimation technique used in the original GAN paper.
High-level comments:
- It's not clear why the optimization is done in 3 separate steps. Aside
from the WGAN critic needing to be optimized for more steps, couldn't the
remaining components be trained jointly, with a weighted sum of terms for the
encoder?
- In section 2, "This [pre-training or co-training with maximum likelihood]
  precludes there being a latent encoding of the sentence." It is not at all
  clear to me why this would be the case.
- "One benefit of the ARAE framework is that it compresses the input to a
  single code vector." This is true of any autoencoder.
- It would be worth explaining, in a sentence, the approach in Shen et al for
  those who are not familiar with it, seeing as it is used as a baseline.
- We are told that the encoder's output is l2-normalized but the generator's
  is not, instead output units of the generator are squashed with the tanh
  activation. The motivation for this choice would be helpful. Shortly
  thereafter we are told that the generator quickly learns to produce norm 1
  outputs as evidence that it is matching the encoder's distribution, but this
  is something that could have just as easily have been built-in, and is a
  trivial sort of "distribution matching"
- In general, tables that report averages would do well to report error bars as
  well. In general some more nuanced statistical analysis of these results
  would be worthwhile, especially where they concern human ratings.
- The dataaset fractions chosen for the semi-supervised experience seem
  completely arbitrary. Is this protocol derived from some other source?
  Putting these in a table along with the results would improve readability.
- Linear interpolation in latent space may not be the best choice here
  seeing as e.g. for a Gaussian code the region near the origin has rather low
  probability. Spherical interpolation as recommended by White (2016) may
  improve qualitative results.
- For the interpolation results you say "we output the argmax", what is meant?
  Is beam search performed in the case of sequences?
- Finally, a minor point: I will challenge the authors to justify their claim
  that the learned generative model is "useful" (their word). Interpolating
  between two sentences sampled from the prior is a neat parlour trick, but the
  model as-is has little utility. Even some speculation on how this aspect
  could be applied would be appreciated (admittedly, many GAN papers could use
  some reflection of this sort).


We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures. Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures. Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l 1 norm bounded weights.

Summary:
This paper considers a learning method for the ResNet using the boosting framework. More precisely, the authors view the structure of the ResNet as a (weighted) sum of base networks (weak hypotheses) and apply the boosting framework. The merit of this approach is to decompose the learning of complex networks to that of small to large networks in a moderate way and it uses less computational costs. The experimental results are good. The authors also show training and generalization error bounds for the proposed approach.
Comments:
The idea of the paper is natural and interesting. Experimental results are somewhat impressive. However, I am afraid that theoretical results in the paper contain several mistakes and does not hold. The details are below.
I think the proof of Theorem 4.2 is wrong. More precisely, there are several possibly wrong arguments as follows:
- In the proof, \alpha_t+1 is chosen so as to minimize an upper bound of Z_t, while the actual algorithm is chosen to minimize Z_t. The minimizer of Z_t and that of an upper bound are different in general. So, the obtained upper bound does not hold for the training error of the actual algorithm.
- It is not a mistake, but, there is no explanation why the equality between (27) and (28) holds. Please add an explanation. Indeed, equation (21) matters.
Also, the statement of Theorem 4.2 looks somewhat cheating: The statement seems to say that it holds for any iteration T and the training error decays exponentially w.r.t. T. However, the parameter T is determined by the parameter gamma, so it is some particular iteration, which might be small and the bound could be large.
The generalization error bound Corollary 4.3 seems to be wrong, too. More precisely, Lemma 2 of Cortes et al. is OK, but the application of Lemma 2 is not. In particular, the proof does not take into account of the function \sigma. In other words, the proof considers the Rademacher complexity R_m(\calF_t), of the class \calF_t, but, acutually, I think it should consider R_m(\sigma(\calF_t)), where the class \sigma(\calF_t) consists of the composition of functions \sigma and f_t in \calF_t. Talagrand’s lemma (see, e.g., Mohri et al.’ s book: Foundation of Machine Learning) can be used to analyze the complexity of the composite class. But, the resulting bound would depend on the Lipschizness of \sigma in an exponential way.
The explanation of the generalization ability is not sufficient. While the latter weak hypotheses are complex enough and would have large edges, the complexity of the function class of weak hypotheses grows exponentially w.r.t. the iteration T, which should be mentioned.
As a summary, the paper contains nice ideas and experimental results are promising, but has non-negligible mistakes in theoretical parts which degrade the contribution of the paper.
Minor Comments:
-In Algorithm 1, \gamma_t is not defined when a while-loop starts. So, the condition of the while-loop cannot be checked.


Inspired by previous work on emergent communication in referential games, we propose a novel multi-modal, multi-step referential game, where the sender and receiver have access to distinct modalities of an object, and their information exchange is bidirectional and of arbitrary duration. The multi-modal multi-step setting allows agents to develop an internal communication significantly closer to natural language, in that they share a single set of messages, and that the length of the conversation may vary according to the difficulty of the task. We examine these properties empirically using a dataset consisting of images and textual descriptions of mammals, where the agents are tasked with identifying the correct object. Our experiments indicate that a robust and efficient communication protocol emerges, where gradual information exchange informs better predictions and higher communication bandwidth improves generalization.

--------------
Summary and Evaluation:
--------------
The paper presents a nice set of experiments on language emergence in a mutli-modal, multi-step setting. The multi-modal reference game provides an interesting setting for communication, with agents learning to map descriptions to images. The receiving agent's direct control over dialog length is also novel and allows for the interesting analysis presented in later sections.
Overall I think this is an interesting and well-designed work; however, some details are missing that I think would make for a stronger submission (see weaknesses).
--------------
Strengths:
--------------
- Generally well-written with the Results and Analysis section appearing especially thought-out and nicely presented.
- The proposed reference game provides a number of novel contributions -- giving the agents control over dialog length, providing both agents with the same vocabulary without constraints on how each uses it (implicit through pretraining or explicit in the structure/loss), and introducing an asymmetric multi-modal context for the dialog.
- The analysis is extensive and well-grounded in the three key hypothesis presented at the beginning of Section 6.
--------------
Weaknesses:
--------------
- There is room to improve the clarity of Sections 3 and 4 and I encourage the authors to revisit these sections. Some specific suggestions that might help:
		- numbering all display style equations
		- when describing the recurrent receiver, explain the case where it terminates (s^t=1) first such that P(o_r=1) is defined prior to being used in the message generation equation.
- I did not see an argument in support of the accuracy@K metric. Why is putting the ground truth in the top 10% the appropriate metric in this setting? Is it to enable comparison between the in-domain, out-domain, and transfer settings?
- Unless I missed something, the transfer test set results only comes up once in the context of attention methods and are not mentioned elsewhere. Why is this? It seems appropriate to include in Figure 5 if no where else in the analysis.
- Do the authors have a sense for how sensitive these results are to different runs of the training process?
- I did not understand this line from Section 5.1: "and discarding any image with a category beyond the 398-th most frequent one, as classified by a pretrained ImageNet classifier'"
- It is not specified (or I missed it) whether the F1 scores from the separate classifier are from training or test set evaluations.
- I would have liked to see analysis on the training process such as a plot of reward (or baseline adjusted reward) over training iterations.
- I encourage authors to see the EMNLP 2017 paper "Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog" which also perform multi-round dialogs between two agents. Like this work, the authors also proposed removing memory from one of the agents as a means to avoid learning degenerate 'non-dialog' protocols.
- Very minor point: the use of fixed-length, non-sequence style utterances is somewhat disappointing given the other steps made in the paper to make the reference game more 'human like' such as early termination, shared vocabularies, and unconstrained utterance types. I understand however that this is left as future work.
--------------
Curiosities:
--------------
- I think the analysis is Figure 3 b,c is interesting and wonder if something similar can be computed over all examples. One option would be to plot accuracy@k for different utterance indexes -- essentially forcing the model to make a prediction after each round of dialog (or simply repeating its prediction if the model has chosen to stop).


We propose Episodic Backward Update -a new algorithm to boost the performance of a deep reinforcement learning agent by fast reward propagation. In contrast to the conventional use of replay memory with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state into its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate effectively throughout the sampled episode. We evaluate our algorithm on 2D MNIST Maze Environment and 49 games of the Atari 2600 Environment and show that our agent improves sample efficiency with a competitive computational cost.

This paper proposes a new variant of DQN where the DQN targets are computed on a full episode by a « backward » update (i.e. from end to start of episode). The targets’ update rule is similar to a regular tabular Q-learning update with high learning rate beta: this allows faster propagation of rewards obtained at the end of the episode (while beta=0 corresponds to regular DQN with no such reward propagation). This mechanism is shown to improve on Q-learning in a toy 2D maze environment (with MNIST-based pixel states providing cell coordinates) with beta=1, and on DQN and its optimality tightening variant on Atari games with beta=0.5.
The intuition behind the algorithm (that one should try to speed up the propagation of rewards across multiple steps) is not new, in fact it has inspired other approaches like n-step Q-learning, eligibility traces or more recently Retrace(lambda) in deep RL. Actually the idea of replaying experiences in backward order can be traced back to the origins of experience replay («  Programming Robots Using Reinforcement Learning and Teaching », Lin, 1991), something that is not mentioned here. That being said, to the best of my knowledge the specific algorithm proposed in this submission (Alg. 2) is novel, even if Alg. 1 is not (Alg. 1 can be seen as a specific instance of Lin’s algorithm with a very high learning rate, and clearly only makes sense in toy deterministic environments).
In the absence of any theoretical analysis of the proposed approach, I would have expected an in-depth empirical validation. Unfortunately this is not the case here. In the toy environment (4.1) I am surprised by the really poor quality of the results (paths 5-10 times longer than the shortest path on average): have algorithms been run for a long enough time? Or maybe the average is a bad performance measure due to outliers? I would have also appreciated a comparison to Retrace(lambda), which is a more principled way to use multi-step rewards than n-step Q-learning (which is technically an on-policy method). Similar remarks can be made on the Atari experiments (4.2), where 10M frames is really low (the original DQN paper had results on 50M frames, and Rainbow reports 200M frames in only ~2x the training time reported here). The comparison also should have included prioritized experience replay, which has been shown to provide a significant boost in DQN, but may be tricky to combine with the proposed algorithm. Overall comparing only to vanilla DQN and its optimality tightening variant is too limited when there have been so many other meaningful improvements over DQN. This makes it really hard to tell whether the proposed algorithm would actually help when combined with a state-of-the-art method like Rainbow for instance.
A few additional small remarks and questions:
- « Second, there is no point in updating a one-step transition unless the future transitions have not been updated yet. »: should « unless » be replaced by « if »?
- In 4.1 is there a maximum number of steps per episode and can you please confirm that training is done independently for each maze?
- Typo in eq. 3: the - in the max should be a comma
- There is a good amount of typos and grammar errors, though they do not harm the readability of the paper
- Citations for « Deep Reinforcement Learning with Double Q-learning » and « Dueling Network Architectures for Deep Reinforcement Learning » could refer to their conference versions
- « epsilon starts from 1 and is annealed to 0 at 200,000 steps in a quadratic manner »: please specify the exact formula
- Fig. 7 is really confusing, there seem to be typos and it is not clear why the beta updates appear in these specific cells, please revise it if you want to keep it


Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoderdecoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required. We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.

-- updates to review: --
Thanks for trying to respond to my comments. I find the new results very interesting and fill in some empirical gaps that I think were worth investigating. I'm now more confident that this paper is worth publishing and I increased my rating from 6 to 7.
I admit that this is a pretty NLP-specific paper, but to the extent that ICLR has core NLP papers (I think it does have some), I think the paper is a reasonable fit for ICLR. It might feel more at home at a *ACL conference though.
-- original review is below: --
This paper is about modifications to the skip-thought framework for learning sentence embeddings. The results show performance comparable to or better than skip-thought while decreasing training time.
I think the overall approach makes sense: use an RNN encoder because we know it works well, but improve training efficiency by changing the decoder to a combination of feed-forward and convolutional layers.
I think it may be the case that this works well because the decoder is not auto-regressive but merely predicts each word independently. This is possible because the decoder will not be used after training. So all the words can be predicted all at once with a fixed maximum sentence length. In typical encoder-decoder applications, the decoder is used at test time to get predictions, so it is natural to make it auto-regressive. But in this case, the decoder is thrown away after training, so it makes more sense to make the decoder non-auto-regressive.  I think this point should be made in the paper.
Also, I think it's worth noting that an RNN decoder could be used in a non-auto-regressive architecture as well. That is, the sentence encoding could be mapped to a sequence of length 30 as is done with the CNN decoder currently; then a (multi-layer) BiLSTM could be run over that sequence, and then a softmax classifier can be attached to each hidden vector to predict the word at that position. It would be interesting to compare that BiLSTM decoder with the proposed CNN decoder and also to compare it to a skip-thought-style auto-regressive RNN decoder. This would let us understand whether the benefit is coming more from the non-auto-regressive nature of the decoder or from the CNN vs RNN differences.
That is, it would make sense to factor the decision of decoder design along multiple axes. One axis could be auto-regressive vs predict-all-words. Another axis could be using a CNN over the sequence of word positions or an RNN over the sequence of word positions.  For auto-regressive models, another axis could be train using previous ground-truth word vs train using previous predicted word.  Skip-thought corresponds to an auto-regressive RNN (using the previous ground-truth word IIRC).  The proposed decoder is a predict-all-words CNN.  It would be natural to also experiment with an auto-regressive CNN and a predict-all-words RNN (like what I described in the paragraph above). The paper is choosing a single point in the space and referring to it as a "CNN decoder" whereas there are many possible architectures that can be described this way and I think it would strengthen the paper to increase the precision in discussing the architecture and possible alternatives.
Overall, I think the architectural choices and results are strong enough to merit publication. Adding any of the above empirical comparisons would further strengthen the paper.
However, I did have quibbles with some of the exposition and some of the claims made throughout the paper. They are detailed below:
Sec. 2:
In the "Decoder" paragraph: please add more details about how the words are predicted. Are there final softmax layers that provide distributions over output words? I couldn't find this detail in the paper. What loss is minimized during training? Is it the sum of log losses over all words being predicted?
Sec. 3:
Section 3 does not add much to the paper. The motivations there are mostly suggestive rather than evidence-based. Section 3 could be condensed by about 80% or so without losing much information. Overall, the paper has more than 10 pages of content, and the use of 2 extra pages beyond the desired submission length of 8 should be better justified. I would recommend adding a few more details to Section 2 and removing most of Section 3. I'll mention below some problematic passages in Section 3 that should be removed.
Sec. 3.2:
"...this same constraint (if using RNN as the decoder) could be an inappropriate constraint in the decoding process."  What is the justification or evidence for this claim?  I think the claim should be supported by an argument or some evidence or else should be removed. If the authors intend the subsequent paragraphs to justify the claim, then see my next comments.
Sec. 3.2:
"The existence of the ground-truth current word embedding potentially decreases the tendency for the decoder to exploit other information from the sentence representation."
But this is not necessarily an inherent limitation of RNN decoders since it could be addressed by using the embedding of the previously-predicted word rather than the ground-truth word. This is a standard technique in sequence-to-sequence learning; cf. scheduled sampling (Bengio et al., 2015).
Sec. 3.2:
"Although the word order information is implicitly encoded in the CNN decoder, it is not emphasized as it is explicitly in the RNN decoder. The CNN decoder cares about the quality of generated sequences globally instead of the quality of the next generated word. Relaxing the emphasis on the next word, may help the CNN decoder model to explore the contribution of context in a larger space."
Again, I don't see any evidence or justification for these arguments. Also see my discussion above about decoder variations; these are not properties of RNNs vs CNNs but rather properties of auto-regressive vs predict-all-words decoders.
Sec. 5.2-5.3:
There are a few high-level decisions being tuned on the test sets for some of the tasks, e.g., the length of target sequences in Section 5.2 and the number of layers and channel size in Section 5.3.
Sec. 5.4:
When trying to explain why an RNN encoder works better than a CNN encoder, the paper includes the following: "We stated above that, in our belief, explicit usage of the word order information will augment the transferability of the encoder, and constrain the search space of the parameters in the encoder. The results match our belief."
I don't think these beliefs are concrete enough to be upheld or contradicted. Both encoders explicitly use word order information. Can you provide some formal or theoretical statement about how the two encoders treat word order differently? I fear that it's only impressions and suppositions that lead to this difference, rather than necessarily something formal.
Sec. 5.4:
In Table 1, it is unclear why the "future predictor" model is the one selected to be reported from Gan et al (2017). Gan et al has many settings and the "future predictor" setting is the worst. An explanation is needed for this choice.
Sec. 6.1:
In the "BYTE m-LSTM" paragraph:
"Our large RNN-CNN model trained on Amazon Book Review (the largest subset of Amazon Review) performs on par with BYTE m-LSTM model, and ours works better than theirs on semantic relatedness and entailment tasks."  I'm not sure this "on par" assessment is warranted by the results in Table 2.  BYTE m-LSTM is better on MR by 1.6 points and better on CR by 4.7 points. The authors' method is better on SUBJ by 0.7 and better on MPQA by 0.5.  So on sentiment tasks, BYTE m-LSTM is clearly better, and on the other tasks the RNN-CNN is typically better, especially on SICK-r.
More minor things are below:
Sec. 1:
The paper contains this: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b)"
I don't think this is accurate. When restricting attention to neural network methods, it would be more correct to give credit to Collobert et al. (2011). But moving beyond neural methods, there were decades of previous work in using context information (counts of context words) to produce vector representations of words.
typo: "which d reduces" --> "which reduces"
Sec. 2:
The notation in the text doesn't match that in Figure 1: w_i^1 vs. w_1 and h_i^1 vs h_1.
Instead of writing "non-parametric composition function", describe it as "parameter-free". "Non-parametric" means that the number of parameters grows with the data, not that there are no parameters.
In the "Representation" paragraph: how do you compute a max over vectors? Is it a separate max for each dimension? This is not clear from the notation used.
Sec. 3.1:
inappropriate word choice: the use of "great" in "a great and efficient encoding model"
Sec. 3.2:
inappropriate word choice: the use of "unveiled" in "is still to be unveiled"
Sec. 3.4:
Tying input and output embeddings can be justified with a single sentence and the relevant citations (which are present here). There is no need for speculation about what may be going on, e.g.: "the model learns to explore the non-linear compositionality of the input words and the uncertain contribution of the target words in the same space".
Sec. 4:
I think STS14 should be defined and cited where the other tasks are described.
Sec. 5.3:
typo in Figure 2 caption: "and and"
Sec. 6.1:
In the "Skip-thought" paragraph:
inappropriate word choice: "kindly"
The description that says "we cut off a branch for decoding" is not clear to me. What is a "branch for decoding" in this context? Please modify it to make it more clear.
References:
Bengio S, Vinyals O, Jaitly N, Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS 2015.
Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P. Natural language processing (almost) from scratch. Journal of Machine Learning Research 2011.


Published as a conference paper at ICLR 2018 PROGRESSIVE REINFORCEMENT LEARNING WITH DISTILLATION FOR MULTI-SKILLED MOTION CONTROL
Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a specific skill and its associated state distribution. We extend policy distillation methods to the continuous action setting and leverage this technique to combine expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain. We also introduce an input injection method for augmenting an existing policy network to exploit new input features. Lastly, our method uses transfer learning to assist in the efficient acquisition of new skills. The combination of these methods allows a policy to be incrementally augmented with new skills. We compare our progressive learning and integration via distillation (PLAID) method against three alternative baselines.

Hi,
This was a nice read. I think overall it is a good idea. But I find the paper lacking a lot of details and to some extend confusing.
Here are a few comments that I have:
Figure 2 is very confusing for me. Please first of all make the figures much larger. ICLR does not have a strict page limit, and the figures you have are hard to impossible to read. So you train in (a) on the steps task until 350k steps? Is (b), (d),(c) in a sequence or is testing moving from plain to different things? The plot does not explicitly account for the distillation phase. Or at least not in an intuitive way. But if the goal is transfer, then actually PLAID is slower than the MultiTasker because it has an additional cost to pay (in frames and times) for the distillation phase right? Or is this counted.
Going then to Figure 3, I almost fill that the MultiTasker might be used to simulate two separate baselines. Indeed, because the retention of tasks is done by distilling all of them jointly, one baseline is to keep finetuning a model through the 5 stages, and then at the end after collecting the 5 policies you can do a single consolidation step that compresses all. So it will be quite important to know if the frequent integration steps of PLAID are helpful (do knowing 1,2 and 3 helps you learn 4 better? Or knowing 3 is enough).
Where exactly is input injection used? Is it experiments from figure 3. What input is injecting? What do you do when you go back to the task that doesn't have the input, feed 0? What happens if 0 has semantics ?
Please say in the main text that details in terms of architecture and so on are given in the appendix. And do try to copy a bit more of them in the main text where reasonable.
What is the role of PLAID? Is it to learn a continual learning solution? So if I have 100 tasks, do I need to do 100-way distillation at the end to consolidate all skills? Will this be feasible? Wouldn't the fact of having data from all the 100 tasks at the end contradict the traditional formulation of continual learning?

Or is it to obtain a multitask solution while maximizing transfer (where you always have access to all tasks, but you chose to sequentilize them to improve transfer)?  And even then maximize transfer with respect to what? Frames required from the environment? If that are you reusing the frames you used during training to distill? Can we afford to keep all of those frames around? If not we have to count the distillation frames as well. Also more baselines are needed. A simple baseline is just finetunning as going from one task to another, and just at the end distill all the policies found through out the way.  Or at least have a good argument of why this is suboptimal compared to PLAID.
I think the idea of the paper is interesting and I'm willing to increase (and indeed decrease) my score. But I want to make sure the authors put a bit more effort into cleaning up the paper, making it more clear and easy to read. Providing at least one more baseline (if not more considering the other things cited by them).


Published as a conference paper at ICLR 2018 BOOSTING THE ACTOR WITH DUAL CRITIC
This paper proposes a new actor-critic-style algorithm called Dual Actor-Critic or Dual-AC. It is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation, which can be viewed as a two-player game between the actor and a critic-like function, which is named as dual critic. Compared to its actor-critic relatives, Dual-AC has the desired property that the actor and dual critic are updated cooperatively to optimize the same objective function, providing a more transparent way for learning the critic that is directly related to the objective function of the actor. We then provide a concrete algorithm that can effectively solve the minimax optimization problem, using techniques of multistep bootstrapping, path regularization, and stochastic dual ascent algorithm. We demonstrate that the proposed algorithm achieves state-of-the-art performance across several benchmarks.

This paper proposes a method, Dual-AC, for optimizing the actor(policy) and critic(value function) simultaneously which takes the form of a zero-sum game resulting in a principled method for using the critic to optimize the actor. In order to achieve that, they take the linear programming approach of solving the bellman optimality equations, outline the deficiencies of this approach, and propose solutions to mitigate those problems. The discussion on the deficiencies of the naive LP approach is mostly well done. Their main contribution is extending the single step LP formulation to a multi-step dual form that reduces the bias and makes the connection between policy and value function optimization much clearer without loosing convexity by applying a regularization. They perform an empirical study in the Inverted Double Pendulum domain to conclude that their extended algorithm outperforms the naive linear programming approach without the improvements. Lastly, there are empirical experiments done to conclude the superior performance of Dual-AC in contrast to other actor-critic algorithms.
Overall, this paper could be a significant algorithmic contribution, with the caveat for some clarifications on the theory and experiments. Given these clarifications in an author response, I would be willing to increase the score.
For the theory, there are a few steps that need clarification and further clarification on novelty. For novelty, it is unclear if Theorem 2 and Theorem 3 are both being stated as novel results. It looks like Theorem 2 has already been shown in "Randomized Linear Programming Solves the Discounted Markov Decision Problem in Nearly-Linear Running Time”. There is a statement that “Chen & Wang (2016); Wang (2017) apply stochastic first-order algorithms (Nemirovski et al., 2009) for the one-step Lagrangian of the LP problem in reinforcement learning setting. However, as we discussed in Section 3, their algorithm is restricted to tabular parametrization”. Is you Theorem 2 somehow an extension? Is Theorem 3 completely new?
This is particularly called into question due to the lack of assumptions about the function class for value functions. It seems like the value function is required to be able to represent the true value function, which can be almost as restrictive as requiring tabular parameterizations (which can represent the true value function). This assumption seems to be used right at the bottom of Page 17, where U^{pi*} = V^*. Further, eta_v must be chosen to ensure that it does not affect (constrain) the optimal solution, which implies it might need to be very small. More about conditions on eta_v would be illuminating.
There is also one step in the theorem that I cannot verify. On Page 18, how is the squared removed for difference between U and Upi? The transition from the second line of the proof to the third line is not clear. It would also be good to more clearly state on page 14 how you get the first inequality, for || V^* ||_{2,mu}^2.
For the experiments, the following should be addressed.
1. It would have been better to also show the performance graphs with and without the improvements for multiple domains.
2. The central contribution is extending the single step LP to a multi-step formulation. It would be beneficial to empirically demonstrate how increasing k (the multi-step parameter) affects the performance gains.
3. Increasing k also comes at a computational cost. I would like to see some discussions on this and how long dual-AC takes to converge in comparison to the other algorithms tested (PPO and TRPO).
4. The authors concluded the presence of local convexity based on hessian inspection due to the use of path regularization. It was also mentioned that increasing the regularization parameter size increases the convergence rate. Empirically, how does changing the regularization parameter affect the performance in terms of reward maximization? In the experimental section of the appendix, it is mentioned that multiple regularization settings were tried but their performance is not mentioned. Also, for the regularization parameters that were tried, based on hessian inspection, did they all result in local convexity? A bit more discussion on these choices would be helpful.
Minor comments:
1. Page 2: In equation 5, there should not be a 'ds' in the dual variable constraint


We study how, in generative adversarial networks, variance in the discriminator's output affects the generator's ability to learn the data distribution. In particular, we contrast the results from various well-known techniques for training GANs when the discriminator is near-optimal and updated multiple times per update to the generator. As an alternative, we propose an additional method to train GANs by explicitly modeling the discriminator's output as a bi-modal Gaussian distribution over the real/fake indicator variables. In order to do this, we train the Gaussian classifier to match the target bi-modal distribution implicitly through metaadversarial training. We observe that our new method, when trained together with a strong discriminator, provides meaningful, non-vanishing gradients.

The authors provided empirical analysis of different variants of GANs and proposed a regularization scheme to combat the vanishing gradient when the discriminator is well trained.
More specifically, the authors demonstrated the importance of intra-class variance in the discriminator’s output. Methods whose discriminators tend to map inputs of a class to single real values are unable to provide a reliable learning signal for the generator, such as the standard GAN and Least Squares GAN. Variance in the discriminator’s output is essential to allow the generator to learn in the presence of a well-trained discriminator. To ensure the discriminator’s output follows the mixture of two univariate Gaussians, the authors proposed to add two additional discriminators which are trained in a similar was as the original GAN formulation. The technique is related to Linear Discriminant Analysis. From a broader perspective, the new meta-adversarial learning can be applied to ensure various desirable properties in GANs.
The performance of variance regularization scheme was evaluated on the CIFAR-10 and CelebA data.
Summary:
——
I think the paper discusses a very interesting topic and presents an interesting direction for training the GANs. A few points are missing which would provide significantly more value to readers. See comments below for details and other points.
Comments:
——
1.	Why would a bi-modal distribution be meaningful? Deep nets implicitly transform the data which is probably much more effective than using complex bi-modal Gaussian distribution; the bi-modal concept can likely be captured using classical techniques.
2.	On page 4, in Eq. (8) and (9), it remains unclear what $\mathcal{R}$ and $\mathcal{F}$ really are beyond two-layer MLPs; are the results of those two-layer MLPs used as the mean of a Gaussian distribution, i.e., $\mu_r$ and $\mu_f$?
3.	Regarding the description above Eq. (12), what is really used in practice, i.e., in the experiments? The paper omits many details that seem important for understanding. Could the authors provide more details on choosing the generator loss function and why Eq. (12) provides satisfying results in practice?
Minor Comments:
——
1.	In Sec 2.1, the sentence needs to be corrected: “As shown in Arjovsky & Bottou (2017), the JS divergence will be flat everywhere important if P and Q both lie on low-dimensional manifolds (as is likely the case with real data) and do not prefectly align.”
2.	Last sentence in Conclusion: “which can be applied to ensure enforce various desirable properties in GANs.” Please remove either “ensure” or “enforce.”


We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes with some probability, for each instance. Specifically, we overlay binary masking variables over class output probabilities, which are learned based on the input via regularized variational inference. This stochastic regularization has an effect of building an ensemble classifier out of exponential number of classifiers with different decision boundaries. Moreover, the learning of dropout probabilities for non-target classes on each instance allows the classifier to focus more on classification against the most confusing classes. We validate our model on multiple public datasets for classification, on which it obtains improved accuracy over regular softmax classifier and other baselines. Further analysis of the learned dropout masks shows that our model indeed selects confusing classes more often when it performs classification.

Pros
- The proposed model is a nice way of multiplicatively combining two features :
  one which determines which classes to pay attention to, and other that
provides useful features for discrimination.
- The adaptive component seems to provide improvements for small dataset sizes
  and large number of classes.
Cons
- "One can easily see that if o_t(x; w) = 0, then class t becomes neutral in the
  classification and the gradients are not back-propagated from it." : This does
not seem to be true. Even if the logits are zero, the class would have a
non-zero probability and would receive gradients. Do the authors mean
exp(o_t(x;w)) = 0 ?
- Related to the above, it should be clarified what is meant by dropping a
  class. Is its logit set to zero or -\infty ? Excluding a class from the
softmax is equivalent to having a logit of -\infty, not zero. However, from the
equations in the paper it seems that the logit is set to zero. This would not
result in excluding the unit. The overall effect would just be to raise the
magnitude of logits across the entire softmax.
- It seems that the model benefits from at least two separate effects - one is
  the attention mechanism provided by the sigmoids, and the other is the
stochasticity during training. Presently, it is not clear if only one of the
components is providing most of the benefits, or if both things are useful. It
would be great to compare this model to a non-stochastic one which just has the
multiplicative effects applied in a deterministic way (during both training and
testing).
- The objective of the attention mechanism that sets the dropout mask seems to
  be the same as the primary objective of classifying the input, and the
attention mechanism is prevented from solving the task by adding an extra
entropy regularization. It would be useful to explain more why this is needed.
Would it not be fine if the attention mechanism did a perfect job of selecting
the class ?
Quality
The paper makes relevant comparisons and is overall well-motivated. However,
some aspects of the paper can be improved by adding more explanations.
Clarity
Some crucial aspects of the paper are unclear as mentioned above.
Originality
The main contribution of the paper is similar to multiplicative gating. The
added stochasticity and the model ensembling interpretation is probably novel.
However, experiments are insufficient to determine whether it is this novelty
that contributes to improved performance or just the gating.
Significance
This paper makes incremental improvements and would be of moderate interest to
the machine learning community.
Typos :
- In Eq 3, the numerator has z_t. Should that be z_y ?
- In Eq 5, the denominator has z_y. Should that be z_t ?


FEW-SHOT AUTOREGRESSIVE DENSITY ESTIMATION: TOWARDS LEARNING TO LEARN DISTRIBUTIONS
Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet. However, such models require many thousands of gradient-based weight updates and unique image examples for training. Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks. In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation. Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset. Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision. Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

This paper focuses on the density estimation when the amount of data available for training is low. The main idea is that a meta-learning model must be learnt, which learns to generate novel density distributions by learn to adapt a basic model on few new samples. The paper presents two independent method.
The first method is effectively a PixelCNN combined with an attention module. Specifically, the support set is convolved to generate two sets of feature maps, the so called "key" and the "value" feature maps. The key feature map is used from the model to compute the attention in particular regions in the support images to generate the pixels for the new "target" image. The value feature maps are used to copmpute the local encoding, which is used to generate the respective pixels for the new target image, taking into account also the attention values. The second method is simpler, and very similar to fine-tuning the basis network on the few new samples provided during training. Despite some interesting elements, the paper has problems.
First, the novelty is rather limited. The first method seems to be slightly more novel, although it is unclear whether the contribution by combining different models is significant. The second method is too similar to fine-tuning: although the authors claim that \mathcal{L}_inner can be any function that minimizes the total loss \mathcal{L}, in the end it is clear that the log-likelihood is used. How is this approach (much) different from standard fine-tuning, since the quantity P(x; \theta') is anyways unknown and cannot be "trained" to be maximized.
Besides the limited novelty, the submission leaves several parts unclear. First, why are the convolutional features of the support set in the first methods divided into "key" and "value" feature maps as in p_key=p[:, 0:P], p_value=p[:, P:2*P]? Is this division arbitrary, or is there a more basic reason? Also, is there any different between key and value? Why not use the same feature map for computing the attention and computing eq (7)?
Also, in the first model it is suggested that an additional feature can be having a 1-of-K channel for the supporting image label: the reason is that you might have multiple views of objects, and knowing which view contributes to the attention can help learning the density. However, this assumes that the views are ordered, namely that the recording stage has a very particular format. Isn't this a bit unrealistic, given the proposed setup anyways?
Regarding the second method, it is not clear why leaving this room for flexibility (by allowing L_inner to be any function) to the model is a good idea. Isn't this effectively opening the doors to massive overfitting? Besides, isn't the statement that the function \mathcal{L}_inner void? At the end of the day one can also claim the same for gradient descent: you don't need to have the true gradients of the true loss, as long as the objective function obtains gradually lower and lower values?
Last, it is unclear what is the connection between the first and the second model. Are these two independent models that solve the same problem? Or are they connected?
Regarding the evaluation of the models, the nature of the task makes the evaluation hard: for real data like images one cannot know the true distribution of particular support examples. Surrogate tasks are explored, first image flipping, then likelihood estimation of Omniglot characters, then image generation. Image flipping does not sound a very relevant task  to density estimation, given that the task is deterministic. Perhaps, what would make more sense would be to generate a new image given that the support set has images of a particular orientation, meaning that the model must learn how to learn densities from arbitrary rotations. Regarding Omniglot character generation, the surrogate task of computing likelihood of known samples gives a bit better, however, this is to be expected when combining a model without attention, with an attention module.
All in all, the paper has some interesting ideas. I encourage the authors to work more on their submission and think of a better evaluation and resubmit.


Methods that calculate dense vector representations for features in unstructured data-such as words in a document-have proven to be very successful for knowledge representation. We study how to estimate dense representations when multiple feature types exist within a dataset for supervised learning where explicit labels are available, as well as for unsupervised learning where there are no labels. Feat2Vec calculates embeddings for data with multiple feature types enforcing that all different feature types exist in a common space. In the supervised case, we show that our method has advantages over recently proposed methods; such as enabling higher prediction accuracy, and providing a way to avoid the cold-start problem. In the unsupervised case, our experiments suggest that Feat2Vec significantly outperforms existing algorithms that do not leverage the structure of the data. We believe that we are the first to propose a method for learning unsupervised embeddings that leverage the structure of multiple feature types.

SUMMARY.
The paper presents an extension of word2vec for structured features.
The authors introduced a new compatibility function between features and, as in the skipgram approach, they propose a variation of negative sampling to deal with structured features.
The learned representation of features is tested on a recommendation-like task.
----------
OVERALL JUDGMENT
The paper is not clear and thus I am not sure what I can learn from it.
From what is written on the paper I have trouble to understand the definition of the model the authors propose and also an actual NLP task where the representation induced by the model can be useful.
For this reason, I would suggest the authors make clear with a more formal notation, and the use of examples, what the model is supposed to achieve.
----------
DETAILED COMMENTS
When the authors refer to word2vec is not clear if they are referring to skipgram or cbow algorithm, please make it clear.
Bottom of page one: "a positive example is 'semantic'", please, use another expression to describe observable examples, 'semantic' does not make sense in this context.
Levi and Goldberg (2014)  do not say anything about factorization machines, could the authors clarify this point?
Equation (4), what do i and j stand for? what does \beta represent? is it the embedding vector? How is this formula related to skipgram or cbow?
The introduction of structured deep-in factorization machine should be more clear with examples that give the intuition on the rationale of the model.
The experimental section is rather poor, first, the authors only compare themselves with word2ve (cbow), it is not clear what the reader should learn from the results the authors got.
Finally, the most striking flaw of this paper is the lack of references to previous works on word embeddings and feature representation, I would suggest the author check and compare themselves with previous work on this topic.