Sentence encoders are typically trained on generative language modeling tasks with large unlabeled datasets. While these encoders achieve strong results on many sentence-level tasks, they are difficult to train with long training cycles. We introduce fake sentence detection as a new discriminative training task for learning sentence encoders. We automatically generate fake sentences by corrupting original sentences from a source collection and train the encoders to produce representations that are effective at detecting fake sentences. This binary classification task turns to be quite efficient for training sentence encoders. We compare a basic BiLSTM encoder trained on this task with strong sentence encoding models (Skip-thought and FastSent) trained on a language modeling task. We find that the BiLSTM trains much faster on fake sentence detection (20 hours instead of weeks) using smaller amounts of data (1M instead of 64M sentences). Further analysis shows the learned representations also capture many syntactic and semantic properties expected from good sentence representations.

Summary:
=======
The paper proposes a discriminative training formulation for learning sentence representations, where a classifier is required to distinguish between real and fake sentences. The sentences are encoded with a Bi-LSTM and the resulting sentence representations are then used in a number of sentence-level tasks (classification, entailment, and retrieval). The experiments show benefits on most tasks compared to Skip-Thought and FastSent baselines, and the information captured by the representations is analyzed with probing tasks, showing that they are better at capturing certain kinds of information like the presence or order of words.
The paper proposes a simple and fairly effective approach for learning sentence encoders. The basic idea is appealing and the experimental results are fairly good. However, at present it seems like more work is required for delivering a comprehensive evaluation and analysis. My main concerns with the paper are the insufficient comparison with prior work, its lack of clarity and organization in certain places, and the limited amount of work. Please see below detailed comments on these and other points, as well as suggestions for how to improve some of these issues.
Major comments:
==============
1. Better baselines and comparisons:
- The results are compared only with SKip-Thought and (the weaker) FastSent. However, there are far better models by now. First, already in the Skip-Thought paper there is a version combining Naive Bayes bi-gram features which performs much better on some benchmarks, for example that version would be better than the paper's results on MR (80.4).
- Moreover, there have been many newer papers with better results on many of the tasks [1, 2, 4, and references therein]. At the very least, mention should be made that there are better published results, and ideally there should be some comparison to the more relevant papers [1, and maybe others].
2. Paper organization and clarity:
- I found Section 2 to be unnecessarily lengthy and disorganized. It mixes motivation with modeling, introduces excessive notation, sometimes without clearly defining it (what is L_{aux}? Why is U in eq. 2 not defined on first usage?), and digresses to weakly related discussions (the link to GANs seems vague and the relation to Introspective Neural Networks is not made clear). The last paragraph is largely redundant with the introduction.
- There is also a statement that seems just wrong: "maximizing the data likelihood P(enc(x,\theta_1)|\theta_1,\thera_3)" -- the data likelihood is P(X | ...). Maximizing the encoding of x can be trivially achieved by simply having a constant encoding whose probability is 1.
- The entire Section 2 can be condensed to one or two paragraph, essentially deriving the discriminative training task in equations (1) and (2).
- On the paper organization level, this lengthy section is followed by the related work and then section 4 on "training tasks for encoders". There is again redundancy between section 4 and 2. Consider merging sections 2 and 4 into one Methodology section, where the general task is formulated, the sentence encoding (Bi-LSTM with max-pooling) and binary classifier (the MLP) are defined, and the fake sentence generation is described. This would make a better flow and remove excessive text.
3. Motivation and advantages of the approach:
- The approach is motivated by shortcomings of sentence encodings based on language modeling, such as Skip-Thought, which are computationally intensive due to the large output space and the complicated decoding process. This is an appealing motivation, although there have also been simpler methods for sentence representations that work as well as or better than Skip-Thought [1, 2].
- The second motivation is not clear to me, and the claim that "the training text collection should include many instances of sentences that have only minor lexical differences but found in completely different contexts" needs more support, either theoretical or empirical. Why wouldn't a language model be able to distinguish such differences?
- The advantages of the binary classification task make sense. The point about forcing the encoder to track both syntax and semantics is interesting. Have you tried to analyze whether this indeed happens? The probing tasks are a good way to evaluate this, but most of them are syntactic, except SOMO and perhaps CoordInv and BShift. Still, more analysis of this point would be good.
- One concern with generating fake sentences by swapping words is that it would not apply to languages with free word order. Have you considered how well your approach would work on other languages?
4. Relevant related work:
- The fake data generation resembles noise used in denoising auto-encoders. A recent application is in unsupervised neural machine translation [3], but there is relevant prior work (see references in [3]).
- The binary classification task resembles that in [1], where they train a classifier to distinguish between the representation of a correct neighbor sentences and incorrect sentences.
5. Ideas for more experiments and analysis:
- The results are fairly good by using only 1M sentences. How good would they be with the full corpus? What's the effect of training data size on the method?
- Table 4 is providing nice examples showing how the fake sentence task generates better sentences representations. Can this be measured on a larger set of examples in aggregate? Why is t-SNE needed for calculating the neighbor rank?
- Proving tasks are very interesting, but the discussion is limited. A more detailed discussion and analysis would be useful.
- Consider other techniques for generating fake sentences.
Minor comments:
==============
- Related work: the Skip-Thought decoder is a unidirectional LSTM and not a bidirectional one as mentioned, right?
- Related work: more details on supervised approaches would be useful.
- Section 4.1: how many fake exampels are generated from every real example? Have you experimented with this?
- Section 4.2 mentions 2 hidden layers in the MLP but figure 3 indicates 3 layers.
- Is there a reason to use multiple layers without a non-linearity in the MLP? This seems unusual. In terms of expressivity, this is equivalent to using one larger linear layer, although there might be some benefit in optimization.
- Table 1 seems unnecessary as there is no discussion of how dataset statistics refer to the results. It's enough to refer to previous work.
- What are some results missing in table 2, specifically SKipthought (1M) on COCO datasets?
- The paragraph on sentence encoder implementation mentions a "validation set accuracy of 89 for word shuffle". Which validation set is that? How is convergence determined for word drop?
- In analyzing sentence lengths, figure 2 shows the fake sentence to be similar to SKip-Thought on short sentences in SST. Do you have any idea why? Also, fake sentence is better than Skip-Thought on all lengths in MR, not just longer sentences, so I'm not sure there's any signal there.
- Figure 3: what is the test set for WordShuffle?
- The idea to create negative samples focused towards specific phenomena sounds like a good way to go
Writing, grammar, etc.:
======================
- Introduction, paragraph 3, last sentence: start with "The".
- Introduction, paragraph 4, first sentences: discriminative training task fake sentence detection -> discriminative training task *of* fake sentence detection
- Motivation: an useful -> a useful; we assumes -> we assume; then number -> the number; this much -> this is much
- Motivation: do not differ -> do not differ much?
- Related work: skip-gram -> skip-gram model; Training Skipthought model -> Training a Skipthought model
- Section 4: Prior work use -> Prior work uses/used
- Section 4.2: space between "Multi-layer Perceptron" and "(MLP)". This also happens with other acronyms.
- Page 6: Our models, however, train -> are trained
- Table 3 caption: is bigram in -> is bigram; is co-ordination is -> is-coordination
- Page 7: The analysis ... also indicate*s* ... but do*es* not ...
- Figure 3 caption: classification/proving task -> tasks
- References: fix capitalization in paper titles
References
==========
[1] Logeswaran and Lee, An efficient framework for learning sentence representations
[2] Khodak et al., A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
[3] Artetxe et al., Unsupervised Neural Machine Translation
[4] Arora et al., A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs


Humans are experts at high-fidelity imitation -closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a task instance, and to bootstrap learning of new tasks. Achieving these abilities in autonomous agents is an open problem. In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap. MetaMimic can learn both (i) policies for high-fidelity one-shot imitation of diverse novel skills, and (ii) policies that enable the agent to solve tasks more efficiently than the demonstrators. MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL. This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task. The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions.

Summary:
This paper proposes MetaMimic, an algorithm that does the following:
(i) Learn to imitate with high-fidelity with one-shot. The setting is that we have access to several demonstrations (only states, no actions) of the same task. During training, we have pixel observations plus proprioceptive measurements). At test time, the learned policy can imitate a single new demonstration (consisting of only pixel observations) of the same task.
(ii) When given access to rewards, the policy can exceed the human demonstrator by augmenting its experience replay buffer with the experience gained while learning (i). Therefore, even in a setting with sparse rewards and no access to expert actions (only states), the policy can learn to solve the task.
Overall Evaluation:
This is a good paper. In my opinion however, it does not pass the bar for ICLR.
Pros:
- The paper is well written. The contributions are clearly listed, the methods section is easy to follow and the authors explain the choices they make. The illustrations are clear and intuitive.
- The overview of hyperparameter choice and tuning / importance factor in the Appendix is useful.
- Interesting pipeline of learning policies that can use demonstrations without actions.
- The results on the simulated robot arm (block stacking task with two blocks) are good.
Cons:
- The abstracts oversells the contribution a bit when saying that MetaMimic can learn "policies for high-fidelity one-shot imitation of diverse novel skills". The setting that's considered in the paper is that of a single task, but different demonstrations (different humans from different starting points). This seems restrictive, and could have been motivated better.
- Experimental results are shown only for one task; block stacking with a robot arm in simulation.
- Might not be a good topical fit for ICLR, but more suited for a conference like CoRL or a workshop. The paper is very specific to imitation learning for a manipulation / control tasks, where we can (1) reset the environment to the exact starting position of the demonstrations, (2) the eucledian distance between states in the demonstration and visited by the policy is meaningful (3) we have access to both pixel observations and proprioceptive measurements. The proposed method is an elegant way to solve this, but it's unclear how well it would perform on different types of control problems, or when we want to transfer policies between different (but related) tasks.
Questions:
- Where does the "task stochasticity" come from? Only from the starting state, and from having different demonstrations? Or is the transition function also stochastic?
- The learned policy is able to do one-shot imitation, i.e., given a new demonstration (of the same task) the policy can follow this demonstration. Do I understand correct that this mean that there is *no* additional learning required at test time?
- It is not immediately clear to me why the setting of a single task but new demonstrations is interesting. Could the authors comment on this? One setting I could imagine is that the policy is trained in simulation, but then executed in the real-world, given a new demonstration. (If that's the main motivation though, then the experiments might have to support that this is possible - if no real-world robot is available, maybe the same simulator with a slightly different camera angle / light conditons or so.)
- The x-axis in the figures says "time (hours)" - is that computation time or simulated time?
Other Comments:
- In 3.2, I would be interested in seeing the following baseline comparison: Learn the test task from scratch using the one available demonstration, with the RL procedure (Equation 2, but possibly without the second term to make it fair). In Figure 5, we can see that the performance on the training tasks is much better when training on only 10 tasks, compared to 500. Then why not overfit to a single task, if that's what we're interested in?
- An interesting baseline for 3.3 might be an RL algorithm with shaped rewards: using an additional reward term that is the eucledian distance to the *closest* datapoint from the demonstration. Compared to the baselines shown in the results section, this would be a fairer comparison because (1) unlike D4PG we also have access to information from the demonstrations and (2) no additional information is needed like the action information in D4PGfD and (3) we don't have the need for a curriculum.
Nitpick (no influence on score):
[1. Introduction]
- I find the first sentence, "One-shot imitation is a powerful way to show agents how to solve a task" a bit confusing. I'd say one-shot imitation is a method, not a way to show how to solve a task. Maybe an introductory sentence like "Expert demonstrations are a powerful way to show agents how to solve a task." works better?
- Second sentence, the chosen example is "manufacturing" tasks - do you mean manipulation? When reading this, I had to think of car manufacturing - a task I could certainly not imitate with just a few demonstrations.
- Add note that with "unconditional policy" you mean not conditioned on a demonstration.
[2. MetaMimic]
- [2.1] Third paragraph: write "Figure 2, Algorithm 1" or split the algorithm and figure up so you can refer to them separately.
- [2.1] Last paragraph, second line: remove second "to"


Natural language processing has made significant inroads into learning the semantics of words through distributional approaches, however representations learnt via these methods fail to capture certain kinds of information implicit in the real world. In particular, spatial relations are encoded in a way that is inconsistent with human spatial reasoning and lacking invariance to viewpoint changes. We present a system capable of capturing the semantics of spatial relations such as behind, left of, etc from natural language. Our key contributions are a novel multi-modal objective based on generating images of scenes from their textual descriptions, and a new dataset on which to train it. We demonstrate that internal representations are robust to meaning preserving transformations of descriptions (paraphrase invariance), while viewpoint invariance is an emergent property of the system.

This paper presents a system to map natural language descriptions of scenes containing spatial relations to 3D visualizations of the corresponsing scene. The authors collect a dataset of different scenes containing objects of varying shapes and colors, along with several descriptions from different viewpoints. They train a model based on the Generative Query Network, to generate scenes conditioned on multiple text descriptions as input, along with associated camera angles. Empirical results using human evaluators demonstrate better performance compared to baselines and the authors perform a good analysis of the model, showing that it learns to ground the meaning of spatial words robustly.
Pros:
1. Well-executed paper, with convincing empirical results on the newly collected dataset.
2. Nice analysis to demonstrate that the model indeed learns good semantic representations for spatial language.
Cons:
1. The positioning of this paper with respect to recent work is disappointing. In both the Introduction and Related Work sections, the authors talk about dated models for spatial reasoning in language (pre-2012). There have been several pieces of work that have looked at learning multimodal representations for spatial reasoning. These are a few examples:
   a) Misra, Dipendra, John Langford, and Yoav Artzi. "Mapping Instructions and Visual Observations to Actions with Reinforcement Learning." Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.
   b) Michael Janner, Karthik Narasimhan, and Regina Barzilay. "Representation Learning for Grounded Spatial Reasoning." Transactions of the Association of Computational Linguistics 6 (2018): 49-61.
   c) Paul, Rohan, et al. "Grounding abstract spatial concepts for language interaction with robots." Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 2017.
   d) Ankit Goyal, Jian Wang, Jia Deng. Think Visually: Question Answering through Virtual Imagery. Annual Meeting of the Association for Computational Linguistics (ACL), 2018
Even though Gershman & Tenenbaum (2015) demonstrate weaknesses of a specific model, some of the above papers demonstrate models that can understand things like "A is in front of B" = "B is behind A". A discussion of how this paper relates to some of this prior work, and an empirical comparison (if possible) would be good to have.
2. The introduction reads a bit vague. It would help to clearly state the task considered in this paper upfront i.e. generating 3D scenes from text descriptions at various viewpoints. In the current form, it is hard to understand the task till one arrives at Section 3.
Other comments:
1. What is the difference between the bar graphs of Figures 5 and 6? Is the one on Figure 5 generated using the sentences (and their transforms) from Gershman & Tenenbaum? If so, how do you handle unseen words that are not present in your training data?
2. Would be helpful to clearly explain what the red and black arrows represent in Figure 7.
3. How does the model handle noisy input text i.e. if the object descriptions (shape/color) are off or if some of the input text is incorrect (say a small fraction of the different viewpoints)?
------
Edit:
Thank you for the author response. Even if you consider the story to be the same across literature (which in this case is not, since the more recent models handle spatial relations that the previous ones failed on), it's still worth doing due diligence to the recent work, especially so that the reader gets a better sense of how to position your work amongst these.


DOUBLY REPARAMETERIZED GRADIENT ESTIMATORS FOR MONTE CARLO OBJECTIVES
Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling, 2013;Rezende et al., 2014). These approaches maximize a variational lower bound on the intractable log likelihood of the observed data. Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases. Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al., 2018;. Roeder et al. (2017) propose an improved gradient estimator, however, are unable to show it is unbiased. We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick. The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues. The same idea can be used to improve many recently introduced training techniques for latent variable models. In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio, 2014), and the jackknife variational inference (JVI) gradient (Nowozin, 2018). Finally, we show that this computationally efficient, unbiased drop-in gradient estimator translates to improved performance for all three objectives on several modeling tasks.

Overall:
This paper works on improving the gradient estimator of the ELBO. Author experimentally found that the estimator of the existing work(STL) is biased and proposed to reduce the bias by using the technique like  REINFORCE.
The problem author focused on is unique and the solution is simple, experiments show that proposed method seems promising.
Clarity:
The paper is clearly written in the sense that the motivation of research is clear, the derivation of the proposed method is easy to understand.
Significance:
I think this kind of research makes the variational inference more useful, so this work is significant. But I cannot tell the proposed method is really useful, so I gave this score.
The reason I doubt the reason is that as I written in the below, the original STL can handle the mixture of Gaussians as the latent variable but the proposed method cannot. So I do not know which is better and whether I should use this method or use the original STL with flexible posterior distribution to tighten the evidence lower bound. I think additional experiments are needed. I know that motivation is a bit different for STL and proposed method but some comparisons are needed.
Question and minor comments:
In the original paper of STL, the author pointed out that by freezing the gradient of variational parameters to drop the score function term, we can utilize the flexible variational families like the mixture of Gaussians.
In this work, since we do not freeze the variational parameters, we cannot utilize the mixture of Gaussians as in the STL. IWAE improves the lower bound by increasing the samples, but we can also improve the bound by specifying the flexible posteriors like the mixture of Gaussians in STL.
Faced on this, I wonder which strategy is better to tighten the lower bound, should we use the STL with the mixture of Gaussians or use the proposed method?
To clarify the usefulness of this method, I think the additional experimental comparisons are needed.
About the motivation of the paper, I think it might be better to move the Fig.1 about the Bias to the introduction and clearly state that the author found that the STL is biased "experimentally".
Followings are minor comments.
In experiment 6.1, I'm not sure why the author present the result of K ELBO estimator in the plot of Bias and Variance.
I think author want to point that when K=1, STL is unbiased with respect to the 1 ELBO, but when k>1, it is biased with respect to IWAE estimator.
However, the objective of K ELBO and IWAE are different, it may be misleading. So this should be noted in the paper.
In Figure 3, the left figure, what each color means? Is the color assignment is the same with the middle figure?
(Same for Figure 4)


Motivated by the need for higher order gradients in multi-agent reinforcement learning and meta-learning, this paper studies the construction of baselines for second order Monte Carlo gradient estimators in order to reduce the sample variance. Following the construction of a stochastic computation graph (SCG), the Infinitely Differentiable Monte-Carlo Estimator (DiCE) can generate correct estimates of arbitrary order gradients through differentiation. However, a baseline term that serves as a control variate for reducing variance is currently provided only for first order gradient estimation, limiting the utility of higher-order gradient estimates. To improve the sample efficiency of DiCE, we propose a new baseline term for higher order gradient estimation. This term may be easily included in the objective, and produces unbiased variance-reduced estimators under (automatic) differentiation, without affecting the estimate of the objective itself or of the first order gradient. Importantly it reuses the same baseline function (e.g. the state-value function in reinforcement learning) already used for the first order baseline. We provide theoretical analysis and numerical evaluations of our baseline term, which demonstrate that it can dramatically reduce the variance of second order gradient estimators produced by DiCE. This computational tool can be easily used to estimate second order gradients with unprecedented efficiency wherever automatic differentiation is utilised, and has the potential to unlock applications of higher order gradients in reinforcement learning and meta-learning.

Overview:
This nicely written paper contributes a useful variance reduction baseline to make the recent formalism of the DiCE estimator more practical in application. I assess the novelty and scale of the current contribution as too low for publication at ICLR. Also, the paper includes a few incorrect assertions regarding the control variate framework as well as action-dependent baselines in reinforcement learning. Such issues reduce the value of the contribution in its current form and may contribute to ongoing misunderstandings of the control variate framework and action-dependent baselines in RL, to the detriment of variance reduction techniques in machine learning. I do not recommend publication at this time.
Pros:
The paper is well written modulo the issues discussed below. It strikes me as a valuable workshop contribution once the errors are addressed, but it lacks enough novelty for the main conference track.
Issues:
* (p.5) "R_w and b_w are positively correlated by design, as they should be for variance reduction of the first order gradients."
This statement is not true in general. Intuitively, a control variate reduces variance because when a single estimate of an expectation of a function diverges from its true value according to some delta, then, with high probability, some function strongly correlated with that function will also diverge with a similar delta. Such a delta might be positive or negative, so long as the error may be appropriately modeled as drawn from some symmetric distribution (i.e. is Gaussian).
Control variates are often estimated with an optimal scaling constant that depends on the covariance of the original function and its control variate. Due to the dependence on the covariance, the scaling constant flips sign as appropriate in order reduce variance for any delta. For more information, see the chapter on variance reduction and subsection on control variates in Sheldon Ross's textbook "Simulation."
The fact that a control variate appears to work despite this is not surprising. Biased and suboptimal unbiased gradient estimators have been shown to work well for reasons not fully explored in the literature yet. See, for example, Tucker et al.'s "Mirage of Action-Dependent Baselines", https://arxiv.org/abs/1802.10031.
Since the authors claim on page 6 that the baseline is positively correlated by design, this misunderstanding of the control variate framework appears to be baked into the baseline itself. I recommend the authors look into adaptively estimating an optimal scale for the baseline using a rolling estimator of the covariance and variance to fix this issue. See the Ross book cited above for full derivation of this optimal scale.
* The second error is a mischaracterization of the use and utility of action-dependent baselines for RL problems, on page 6: "We choose the baseline ... to be a function of state ... it must be independent of the action ...." and "it is essential to exclude the current action ... because the baselines ... must be independent of the action ... to remain unbiased." In the past year, a slew of papers have presented techniques for the use of action-dependent baselines, with mixed results (see the Mirage paper just cited), including two of the papers the authors cited.
Cons
* Much of paper revises the DiCE estimator results, arguing for and explaining again those results rather than referring to them as a citation.
* I assess the novelty of proposed contribution as too low for publication. The baseline is an extension of the same method used in the original paper, and does not generalize past the second order gradient, making the promising formalism of the DiCE estimator as infinitely differentiable still unrealizable in practice.
* The experiments are practically identical to the DiCE estimator paper, also reducing the novelty and contribution of the paper.
*EDIT:
I thank the authors for a careful point-by-point comparison of our disagreements on this paper so that we may continue the discussion. However, none of the points I identified were addressed, and so I maintain my original score and urge against publication. In their rebuttal, the authors have defended errors and misrepresentations in the original submission, and so I provide a detailed response to each of the numbered issues below:
(1) I acknowledge that it is common to set c=1 in experiments. This is not the same as the misstatements I cited, verbatim, in the paper that suggest this is required for variance reduction. My aim in identifying these mistakes is not to shame the authors (they appear to simply be typos) but simply to ensure that future work in this area begins with a correct understanding of the theory. I request again that the authors revise the cited lines that incorrectly state the reliance of a control variate on positive correlation. It is not enough to state that "everyone knows" what is meant when the actual claim is misleading.
(2) Without more empirical investigation, the authors' new claim that a strictly state-value-function baseline is a strength rather than a weakness cannot be evaluated. This may be the case, and I would welcome some set of experiments that establish this empirical claim by comparing against state-action-dependent baselines. The authors appear to believe that state-action-dependent baselines are never effective in reducing variance, and this is perhaps the central error in the paper that should be addressed. See response (3). Were the authors to fix this, they would necessarily compare against state-action-dependent baselines, which would be of great value for the community at large in settling this open issue.
(3) Action-dependent baselines have not been shown to be ineffective. I wish to strongly emphasize that this is not the conclusion of the Mirage paper, and the claim repeated in the authors' response (3) has not been validated empirically or analytically, and does not represent the state of variance reduction in reinforcement learning as of this note. I repeat a few key arguments from the Mirage paper in an attempt to dispel the authors' repeated misinterpretation of the paper.
The variance of the policy gradient estimator, subject to a baseline "phi," is decomposed using the Law of Total Variance in Eq (3) of the Mirage paper. This decomposition identifies a non-zero contribution from "phi(a,s)", the (adaptive or non-adaptive) baseline. The Mirage paper analyzes under what conditions such a contribution is expected to be non-negligible. Quoting from the paper:
"We expect this to be the case when single actions have a large effect on the overall discounted
return (e.g., in a Cliffworld domain, where a single action could cause the agent to fall of the cliff and suffer a large negative reward)."
Please see Sec. 3, "Policy Gradient Variance Decomposition" of the Mirage paper for further details.
The Mirage paper does indeed cast reasonable doubt on subsets of a few papers' experiments, and shows that the strong claim, mistakenly made by these papers, that state-action-dependence is always required for an adaptive control variate to reduce variance over state dependence, is not true.
It should be clear from the discussion of the paper to this point that this does _not_ imply the even stronger claim in "A Better Second Order Baseline" that action dependence is never effective and should no longer be considered as a means to reduce variance from a practitioner's point of view. Such a misinterpretation should not be legitimized through publication, as it will muddy the waters in future research. I again urge the authors to remove this mistake from the paper.
(4) I acknowledge the efforts of the authors to ensure that adequate background is provided for readers. This is a thorny issue, and it is difficult to balance in any work. Since this material represents a sizeable chunk of the paper and is nearly identical to existing published work, it leads me to lower the score for novelty of contribution simply by that fact. Perhaps the authors could have considered placing the extensive background materials in the appendix and instead summarizing them briefly in the body of the paper, leaving more room for discussion and experimental validation beyond the synthetic cases already studied in the DiCE paper.
(5), (6) In my review I provided specific, objective criteria by which I have assessed the novelty of this paper: the lack of original written material, and the nearly identical experiments to the DiCE paper. As I noted in response (4) above, this reduces space for further analysis and experimentation.


In federated learning problems, data is scattered across different servers and exchanging or pooling it is often impractical or prohibited. We develop a Bayesian nonparametric framework for federated learning with neural networks. Each data server is assumed to train local neural network weights, which are modeled through our framework. We then develop an inference approach that allows us to synthesize a more expressive global network without additional supervision or data pooling. We then demonstrate the efficacy of our approach on federated learning problems simulated from two popular image classification datasets.

Summary: The paper considers federate learning of neural networks, i.e. data are distributed on multiple machines and the allocation of data points is potentially inhomogenous and unbalanced. The paper proposes a method to combine multiple networks trained locally on different data shards and form a global neural network. The key idea is to identify and reuse neurons that can be used for all shards and add new neurons to the global network if necessary. The matching and combination process is done via MAP inference of a model using a Beta-Bernoulli process. Some experiments on federated learning demonstrate the performance of the proposed method.
General evaluation (+ pro/ - con, more specific comments/questions below):
+ the paper is very well-written -- the BBP presentation is light but very accessible. The experimental set up seems sound.
+ the matching procedure is novel for federated training of neural networks, as far as I know, but might not be if you are a Bayesian nonparametric person, as the paper pointed out similar techniques have been used for topic models.
- the results seem to back up the claim that the proposed is a good candidate for combining networks at the end of training, but the performance is very similar or inferior to naive combination methods and that the global network is way larger than individual local network and nearly as large as simply aggregating all neurons together.
- the comparison to recent federated learning methods is lacking (e.g. McMahan et al, 2017) (perhaps less communication efficient than the proposed method, but more accurate).
Specific comments/questions/suggestions:
- the MAP update for the weights given the assignment matrix is interesting and resembles exactly how the Bayesian committee machine algorithm of Tresp (2000) works, except that the variances are not learnt for each parameter but fixed for each neuron. On this, there are several hyperparameters for the model, e.g. variance sigma_j -- how are these tuned/selected?
- the local neural networks are very mall (only 50 neurons per layer). How do they perform on the test set on the homogeneous case? Is there a performance loss by combining these networks together?
- the compression rate is not that fantastic, i.e. the global network tends to add new neuron for each local neuron considered. Is this because it is in general very hard to identify similar neuron and group them together? In the homogeneous case, surely there are some neurons that might be similar. Or is it because of the MAP inference procedure/local optima?


In this paper, we address quantitative sample-based evaluation for Generative Adversarial Networks (GANs) on generating domain-specific images, where we improve conventional sample-based evaluation methods on two levels: the feature representation and the evaluation metric. Unlike most existing evaluation frameworks which transfer the representation of ImageNet inception model for mapping images onto the feature space, our method uses a specialized encoder to acquire fine-grained domain-specific representation. Moreover, for datasets with multiple classes, we propose Class-Aware Frechet Distance (CAFD), which employs a Gaussian mixture model on the feature space to better fit the feature distribution with multiple clusters. Experiments and analysis on both the feature level and the image level were conducted to demonstrate improvements of our proposed framework over the recently proposed state-of-the-art sample-based method FID. To our best knowledge, we are the first to provide counter examples where FID gives inconsistent results with human judgments. It is shown in the experiments that our framework is able to overcome the shortness of FID and improves robustness. Code will be made available.

This paper proposes a variant of the popular FID score for evaluating GAN-type generative models. The paper makes two major complaints about the FID as it is currently used:
1. The standard Inception network features trained on ImageNet might not be a good representation for whatever different dataset is being modeled, e.g. CelebA or CIFAR-10.
2. The globally-Gaussian assumption made by the FID doesn't hold, which can cause some problems with the metric.
To address issue 1, the paper proposes choosing features based on a dataset-specific VAE, which can additionally incorporate labels when they're available. For 2, the authors propose to compute something like the FID between each component of a Gaussian mixture, based on soft assignments of points to a class with the VAE's inference network to estimate p(y|x), when labels y are available.
In terms of the definition of the CAFD: it is worth emphasizing that (9) is *not* the Frechet = Wasserstein-2 distance between Gaussian mixtures (which is fine). Rather, it's essentially the mean FID of the class-conditional distributions. This has previously been considered in the conditional GAN case, e.g. by Miyato and Koyama (ICLR 2018, https://openreview.net/forum?id=ByS1VpgRZ ). The difference here is that soft-assignments are supported, through the VAE's inference network, though using any classifier would be essentially equivalent. As long as you have a classifier, you can compute the CAFD, regardless of using a VAE representation or not; the VAE just conveniently gives you a classifier out too. Thus the two components of your proposal are essentially orthogonal.
On the choice of dataset-dependent features:
You say several times through the paper that ImageNet-based features are "ineffective" because the class labels do not match with the target, e.g. "fine-grained features distinguishing 'African hunting dog' from 'Cape hunting dog' (which all belong to the category 'dog' in CIFAR-10) are not needed." This is, I think, somewhat misguided: imagine I took ImageNet and assigned higher-level labels to it, such that each image is only assigned a label at the level "dog," and then trained a GAN on it. Then a classifier wouldn't need to distinguish "African hunting dog" from "Cape hunting dog." But a GAN, which doesn't see the labels at all, is being given *exactly the same problem*, and so the GAN still needs to be able to produce both African hunting dogs and Cape hunting dogs (though it doesn't need to be able to tell the two apart).
Moreover, some people believe that CNNs trained on general-purpose approximate the human visual system reasonably well (for an overview of the arguments, see https://neurdiness.wordpress.com/2018/05/17/deep-convolutional-neural-networks-as-models-of-the-visual-system-qa/ ), and although the overall goals of GANs are somewhat fuzzy, "the distribution appears the same to the human visual system" seems pretty good as a goal.
- I think it's obvious that ImageNet-trained Inception features do not model the human visual system very well on, say, MNIST.
- They're probably also not amazing on CelebA, because it hasn't been fine-tuned for faces the way the human visual system has. (Incidentally, you say that "the ImageNet models can hardly distinguish different faces" -- this needs either a citation or some experimental support, in the appendix, because this is not a well-known fact and seems quite relevant to the common practice of applying ImageNet-trained features to CelebA evaluation.)
- But it's not clear to me that they don't model the human visual system reasonably well on CIFAR-10, or at least a theoretical higher-resolution version of it. It's true that ImageNet models will contain some features specific to distinguishing different types of guitars, and there are no guitars in CIFAR-10. But as long as those features aren't strongly activated by actual images from your model, they shouldn't mess up the distributions you're comparing too much.
So if you're going to argue that ImageNet representations are insufficient on vaguely ImageNet-like tasks such as CIFAR-10, I don't think the arguments you have here are quite convincing. Probably, you need some evidence that the scores are made noisier by the irrelevant features and thus harder to estimate, or else maybe strong empirical evidence that using comparable features specific to the dataset distribution performs better.
Anyway, for datasets that are not very much like ImageNet, using dataset-specific features is clearly sensible and perhaps necessary. But:
- You only provide pretty limited evidence that the VAE is better than a plain autoencoder, namely Table 2 which shows that the VAE puts less information in the top few principal components. But you only show that up to the top 5 components, and in any case it's not obvious that a more-spread distribution would be better.
- An important question that's not really considered here: how much does the FID/CAFD then just measure how well the generative model matches the VAE you get features from? Is it the case that this VAE would give a (nearly-)perfect score under the CAFD, or not?
- The results of Figure 1/Table 3 are very interesting. But I wonder how much of this difference in behavior is due to training on CelebA vs ImageNet and how much is due to the architecture or objective of the autoencoder. It might be interesting to compare to features from an ImageNet VAE and/or a CelebA classifier and see what those say. (The discriminator features are something like a CelebA classifier, but there's other things going on there too.)
On the CAFD versus FID:
Your main argument for the CAFD over the FID is that it is based on a richer model of the distribution, which you claim to be closer to true: the FID is based on a multivariate Gaussian assumption with a total of n + n (n-1)/2 parameters, while you use K times as many parameters. Your Table 9 also gives some slight evidence that the Gaussian mixture gives a better fit to the data than a single Gaussian.
I'm not entirely convinced by Table 9; comparing p-values is in general not necessarily very meaningful, and in particular it seems quite possible that the Anderson-Darling test simply prefers the mixture because the samples are more closely "clumped together" by the VAE than a random subset of inputs. Moreover, in either case the Gaussian assumption is clearly false a priori: in the Inception case, features are the output of a ReLU activation function and hence zero-inflated, and this or something like it may also be the case in your VAE. So comparing the p-value of tests for hypotheses known a priori to be false is probably a misguided endeavor.
But in any case the FID doesn't *really* assume Gaussianity. It coincides with the Frechet / Wasserstein-2 distance between Gaussians, but it's a perfectly plausible semimetric between any pair of distributions that have means and variances. The claim for superiority of CAFD over FID would then need to be something like "the class-conditional means and variances are more representative of the distribution than the global means and variances."
Re: your claim that "As both FID and CAFD aim to model how well domain-specific images are generated, they are not designed to deal with mode dropping" -- this is something of a strange claim, as dropping an entire mode will hopefully affect both the feature mean and especially the variance unless it is done extremely carefully. A related problem, though, is that the CAFD is essentially insensitive to drastically *reweighting* modes, e.g. producing twice as many 1s as 2s on MNIST: if each mode is modeled correctly, the CAFD will not be changed, while the FID would be strongly affected with reasonable features. The Mode Score KL(p(y*) || p(y)) would be sensitive to this, as you suggest, but it feels somewhat hacky.
The type of analysis in Table 1 is interesting, but one issue is that it is sensitive to the scale of each mode in feature space: if your encoder happens to place 1s close together and 2s relatively more spread apart, you'll see a higher conditional FID for 2s than for 1s even if the visual "sample quality" is the same.
One important piece of related work that's missing is Binkowski et al. (ICLR 2018, https://openreview.net/forum?id=r1lUOzWCW ), who demonstrate that the FID estimator is strongly biased in a misleading way. The same problems are inherited by the CAFD, which you should at least mention. Binkowski et al., and independently Xu et al. (https://arxiv.org/abs/1806.07755 ), also proposed using MMD variants on top of Inception features. This has better statistical properties as shown by Binkowski et al., and also explicitly does not make any parametric assumptions about the distribution of features. It would be worth thinking about the relationship of that approach to the FID/CAFD.
Another metric you could compare to is the "Adversarial Divergence" of Yang et al. (ICLR 2017, https://openreview.net/forum?id=HJ1kmv9xx ) which compares the distribution of classifier output, p(y|x), for x from the model to that from test data. It's a pretty different metric from CAFD with different properties, but since you both require a classifier, it would be good to know how the two compare.
Minor points:
In the related work, your discussion of the MMD is misleading: Dziugaite et al. and Li et al. proposed using the MMD for *training* generative models, not for evaluating. Evaluating with two-sample tests based on the MMD using simple kernels was done e.g. by Sutherland et al. (ICLR 2017, https://openreview.net/forum?id=HJWHIKqgl ) and Olmos et al. (https://openreview.net/forum?id=HJWHIKqgl ), and used on top of Inception-like representations e.g. by Lopez-Paz and Oquab (2017), as well as Xu et al. and Binkowski et al. mentioned above.
The derivation (6) of the CAFD is that the derivation (6) is somewhat sloppy about exactly what p() means -- in particular, it's somewhat confusing to use a lowercase p when every distribution you deal with here is actually discrete (for discrete y or for the empirical distribution S, since you're dealing with that and not actually the true distribution of the model, where p(x_i) would not be constant across samples x). It would probably be clearer to distinguish your notation for the true model distribution from the empirical distribution of the S samples.
I don't understand your claim on page 6 that "Unlike Inception Score, because CAFD measures distance on the feature space as FID does, it is able to report overfitting." CAFD, like FID, probably doesn't allow for distributions to appear better than the target in the way that Inception score does. But I don't see how this corresponds to "reporting overfitting"; a model that simply reproduces exactly the empirical distribution of the training set would get an excellent CAFD/FID score, but that's the usual sense of "overfitting."
Overall thoughts:
Using dataset-specific features for evaluation metrics makes a lot of sense, but I don't feel totally satisfied by this paper's investigation of the specific proposal of a VAE, and am particularly worried about whether the metric just ends up preferring models similar to that VAE. I'd really like to see some theoretical and empirical investigation into that.
The CAFD as opposed to FID doesn't feel as nice to me; it's both something of an obvious extension of the previously-used "intra-class FID," and I am also unconvinced by the paper's arguments for its preferability over the FID or other metrics based on image representations like those of Xu et al.


GRADIENT DESCENT ALIGNS THE LAYERS OF DEEP LINEAR NETWORKS
This paper establishes risk convergence and asymptotic weight matrix alignment -a form of implicit regularization -of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar results for gradient descent with particular decreasing step sizes): (i) the risk converges to 0; (ii) the normalized i th weight matrix asymptotically equals its rank-1 approximation u i v i ; (iii) these rank-1 matrices are aligned across layers, meaning |v i+1 u i | → 1. In the case of the logistic loss (binary cross entropy), more can be said: the linear function induced by the network -the product of its weight matrices -converges to the same direction as the maximum margin solution. This last property was identified in prior work, but only under assumptions on gradient descent which here are implied by the alignment phenomenon.

In this work the authors prove several claims regarding the inductive bias of gradient descent and gradient flow trained on deep linear networks with linearly separable data. They show that asymptotically gradient descent minimizes the risk, each weight matrix converges to its rank one approximation and the top singular vectors of two adjacent weight matrices align. Furthermore, for the logistic and exponential loss the induced linear predictor converges to the max margin solution.
This work is very interesting and novel. It provides a comprehensive and exact characterization of the dynamics of gradient descent for linear networks. Such strong guarantees are essential for understanding neural networks and extremely rare in the realm of non-convex optimization results. The work is a major contribution over the paper of Gunasekar et al. (2018) which assume that the risk is minimized. The proof techniques are interesting and I believe that they will be useful in analyzing neural networks in other settings.
Regarding Lemma 3, the proof is not clear. Lemma 8 does not exist in the paper of Soudry et al. (2017). It is also claimed that with probability 1 there are at most d support vectors. How does this relate with assumption 3, which implies that there are at least d support vectors?
-------Revision---------
Thank you for the response. I have not changed the original review.


Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations.

This paper demonstrates that CNNs are more robust to class-relevant label noise. They argue that real-world noise should be class-relevant.
Pros:
1. The authors find a new angle to exploit robust learning with noisy labels.
2. The authors perform numerical experiments to demonstrate the effectiveness of their proposal. And their experimental result support their previous claims.
Cons:
We have two questions in the following.
1. Basic definition: in learning with noisy labels, there are two basic models. First, most research focuses on class-conditional noise (CCN) model [1]. Second, recent research explore a bit on instance-dependent noise (IDN) model [2, 3]. As far as I know, there is no class-irrelevant label noise and class-relevevant label noise. In CCN mode, people would like to use symmetric noise and asymmetric noise as a basic benchmark to conduct experiments.
2. Motivation: The authors want to claim CNNs are more robust to such realistic label noise than class-irrelevant label noise. However, they make one mistake. They do not have a clear definition about realistic label noise. In my mind, I believe Clothing1M [4] should be realistic label noise dataset.
By the way, in learning with noisy labels, there are two kinds of research. First, people propose new robust methods for CCN model. Second, people propose new robust methods for IDN models. Proposing new setting should be encouraged. However, the setting and conclusion should be reasonable.
References:
[1] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 1988.
[2] A. Menon, B. Rooyen, and N. Natarajan. Learning from binary labels with instance-dependent corruption. Machine Learning, 2018.
[3] J. Cheng, T. Liu, K. Ramamohanarao, D. Tao. Learning with bounded instance-and label-dependent label noise. arxiv 1709.03768, 2017.
[4] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.


ANYTIME MINIBATCH: EXPLOITING STRAGGLERS IN ONLINE DISTRIBUTED OPTIMIZATION
Distributed optimization is vital in solving large-scale machine learning problems. A widely-shared feature of distributed optimization techniques is the requirement that all nodes complete their assigned tasks in each computational epoch before the system can proceed to the next epoch. In such settings, slow nodes, called stragglers, can greatly slow progress. To mitigate the impact of stragglers, we propose an online distributed optimization method called Anytime Minibatch. In this approach, all nodes are given a fixed time to compute the gradients of as many data samples as possible. The result is a variable per-node minibatch size. Workers then get a fixed communication time to average their minibatch gradients via several rounds of consensus, which are then used to update primal variables via dual averaging. Anytime Minibatch prevents stragglers from holding up the system without wasting the work that stragglers can complete. We present a convergence analysis and analyze the wall time performance. Our numerical results show that our approach is up to 1.5 times faster in Amazon EC2 and it is up to five times faster when there is greater variability in compute node performance.

Summary:
The paper considers the problem of online stochastic convex optimization in a fully distributed topology. In particular, the authors focus on the synchronous setting and to avoid the slow progress that can be obtained by slow nodes, called stragglers, they propose an online distributed optimization method called Anytime Minibatch (AMB). In the update of AMB rather than fixing the minibatch size, they fix the computation time in each epoch. This characteristic prevents the stragglers from holding up the entire network, while allowing nodes to benefit from the partial work carried out by the slower nodes.
A convergence analysis of AMB is provided showing that the online regret achieves the optimum performance. Numerical evaluations where a comparison of AMB and the "Fixed MiniBatch" method (FMB)are also presented.
Comments:
I believe that the idea of the paper is interesting and the convergence analysis seems correct, however i have some concerns regarding  the presentation and the numerical evaluation.
1) In the title the word "online" is mentioned but never explained  in the main text. What is this mean? What are the differences compare to the "static" setting? See for example the work of [Tsianos, Rabbat (2016)] for more details on that. What are the related literature on this setting?
2) In the last paragraph of Introduction is highlighted that the algorithm AMB has the optimum performance?  The authors should add an appropriate reference there and explain why this is optimum for their setting. I believe that for the convenience of the reader current Section 5 called "previous work" can move immediately after introduction and more details of AMB with the existing literature should be provided. Probably rename the section "Closely relate work".
3) Section 2 is devoted mostly on the formal presentation of algorithm AMB. I strongly suggest the addition of a pseudocode of the algorithm in the appendix (or even in the main text if there is a space) where the reader can easily understand how the algorithm works.
4) On the Algorithm:  if some nodes are very slow and they do not make any update during the given time T what will happen? How this will affect the performance of the method? In this case does it make sense to increase the value of T.
5) On numerical evaluation:  A comparison of AMB and FMB  is presented both in synthetic and real data showing that AMB can be faster than FMB in terms of wall clock time.
I am not sure if the performance of the AMB is as good as one should expect especially for the case of synthetic data. Will it be possible to construct a synthetic example with extremely slow nodes where the improvement of the performance is much better than 50%?
In general i find the paper interesting, with nice ideas and I believe that will be appreciated from researchers that are interested on control theory/signal processing and information theory.  Since the paper is focused on convex optimization I am not sure if it will be particularly interesting for a substantial fraction of the ICLR attendees.


Human world knowledge is both structured and flexible. When people see an object, they represent it not as a pixel array but as a meaningful arrangement of semantic parts. Moreover, when people refer to an object, they provide descriptions that are not merely true but also relevant in the current context. Here, we combine these two observations in order to learn fine-grained correspondences between language and contextually relevant geometric properties of 3D objects. To do this, we employed an interactive communication task with human participants to construct a large dataset containing natural utterances referring to 3D objects from ShapeNet in a wide variety of contexts. Using this dataset, we developed neural listener and speaker models with strong capacity for generalization. By performing targeted lesions of visual and linguistic input, we discovered that the neural listener depends heavily on part-related words and associates these words correctly with the corresponding geometric properties of objects, suggesting that it has learned task-relevant structure linking the two input modalities. We further show that a neural speaker that is 'listener-aware' -that plans its utterances according to how an imagined listener would interpret its words in context -produces more discriminative referring expressions than an 'listener-unaware' speaker, as measured by human performance in identifying the correct object.

#update: I've read the authors comments but unfortunately my main concerns about the contributions and novelty of this work are not answered. As such, I cannot increase my score.
------------------
The authors provide a study on learning to refer to 3D objects. The authors collect a dataset of referential expressions and train several models by experimenting with a number of architectural choices.
This is an interesting study reporting results on the effect that several architectural choices have generating referential expressions. Overall, while I appreciate all the experiments and results, I don't really feel I've learned something from this paper.
First and foremost, the paper, from the title already starts to build up expectations about the 3d nature of the study, however this is pretty much ignored at the rest of the paper. I would expect the paper to provide  some results and insights regarding the 3D nature of the dataset and how this affects referential expressions, however, there is no experiment that has used this 3d-ness in any way. Even the representations of the objects are stripped down to essentially 2D (a single-view of a 3D object used to derived VGG features is as 3D as any image dataset used for similar studies, right?).
My major question is then: why should all this research take place in a 3D dataset? Is it to validate that research like this is at all possible with 3D objects?
Moreover, all interesting aspects of referential expressions are stripped out since the authors experiment only with this geometric visual property (which has again nothing to do with 3d-ness, you could totally get that out of images). An interesting study would be to have all objects in the same image and have referential expressions that have to do with spatial expressions, something that the depth or a different view of the of the object could play a role.
Given the fact that there are no technical innovations, I can't vouch for accepting this paper, since there has been quite a lot of research on generating  referential expressions on image datasets (e.g., Kazemzadeh., 2014 and related papers). However, learning to refer to 3D objects is a very interesting topic, and of great importance given the growing interest of training agents in 3D virtual environments, and I would really encourage the authors to embrace the 3d-ness of objects and design studies that highlight the challenges and opportunities that the third dimension brings.
Kazemzadeh et al.: ReferIt Game: Referring to Objects in Photographs of Natural Scenes


Massively multi-label prediction/classification problems arise in environments like health-care or biology where it is useful to make very precise predictions. One challenge with massively multi-label problems is that there is often a longtailed frequency distribution for the labels, resulting in few positive examples for the rare labels. We propose a solution to this problem by modifying the output layer of a neural network to create a Bayesian network of sigmoids which takes advantage of ontology relationships between the labels to help share information between the rare and the more common labels. We apply this method to the two massively multi-label tasks of disease prediction (ICD-9 codes) and protein function prediction (Gene Ontology terms) and obtain significant improvements in per-label AUROC and average precision.

The authors propose a new training scheme for training neural networks for multi-label prediction tasks by introducing ontology relationships between labels.
The paper motivates very well by the observations that some labels include very small amount of data points.
However, the authors don’t really investigate why such labels are rarely observed and the experiments don’t include any significance.
Thus overall I don’t think the paper is ready for publishing for ICLR yet.
Below are some more detailed comments:
1) The authors discuss nicely about the intuition to introduce the Bayesian networks in the tasks of disease prediction. Essentially, the probability of assigning the label (leaf node) should be account for the probability of it being observed, namely the prior. Thus it is not surprising that for the rare labels, the proposed method would yield higher precision. However, the experiments don’t really include any significance measurement; especially for such tasks where the number of testing examples with rare labels is small (5~10 positive examples), significance measurement or some forms of hypothesis testing is a must-have in order to draw conclusion about the performance comparison. Answering such significance issue with tests for overfitting would be nice.
2) My other major concern is for the protein function prediction task, the reason of why for certain labels, the number of instances is small, could be due to that a) there don’t exist much biological web-lab evidence, or b) among the population, there indeed only exist small number of proteins associated with such labels. The proposed method can address b) but not necessarily address a).
3) The paper discusses the other results very briefly in Section 5 but doesn’t include any experiment comparison. Thus it is not convincing that the proposed method is making contribution to the field of disease prediction, protein function prediction or even general multi-label prediction. I would suggest to include the comparison with the state of the art methods for each application.
4) Particularly for protein function prediction, another line of studies is to use protein protein interaction networks or other sources such as functional pathways rather than using sequence information alone (ref below). Some discussion would be nice.
Schwikowski, Benno, Peter Uetz, and Stanley Fields. "A network of protein–protein interactions in yeast." Nature biotechnology 18.12 (2000): 1257.
Cao, Mengfei, et al. "New directions for diffusion-based network prediction of protein function: incorporating pathways with confidence." Bioinformatics 30.12 (2014): i219-i227.


Abstractive summarization has been studied using neural sequence transduction methods with datasets of large, paired document-summary examples. However, such datasets are rare and the models trained from them do not generalize to other domains. Recently, some progress has been made in learning sequenceto-sequence mappings with only unpaired examples. In our work, we consider the setting where there are only documents (product or business reviews) with no summaries provided, and propose an end-to-end, neural model architecture to perform unsupervised abstractive summarization. Our proposed model consists of an auto-encoder trained so that the mean of the representations of the input reviews decodes to a reasonable summary-review. We consider variants of the proposed architecture and perform an ablation study to show the importance of specific components. We show through metrics and human evaluation that the generated summaries are highly abstractive, fluent, relevant, and representative of the average sentiment of the input reviews.

Overall and positives:
The paper investigates the problem of multidocument summarization
without paired documents to summary data, thus using an unsupervised
approach. The main model is constructed using a pair of locked
autoencoders and decoders. The model is trained to optimize the
combination of 1. Loss between reconstructions of the original reviews
(from the encoded reviews) and original the reviews, 2. And the
average similarity of the encoded version of the docs with the encoded
representation of the summary, generated from the mean representation
of the given documents.
By comparing with a few simple baseline models, the authors were able
to demonstrate the potential of the design against several naive
approaches (on real datasets, YELP and AMAZON reviews).
The necessity of several model components is demonstrated
through ablation studies. The paper is relatively well structured and
complete. The topic of the paper fits well with ICLR. The paper
provides decent technical contributions with some novel ideas about
multi-doc summary learning models without a (supervised) paired
data set.
Comments / Issues
[ issue 6 is most important ]
1.  Problem presentation. The problem was not properly introduced and
elaborated. In fact, there is not a formal and mathematical
introduction of the problem, input, output, dataset and model
parameters. The notations used are not very clearly defined and are
quite handwavy, (e.g. what is V, dimensions of inputs x_i was not
mentioned until much later in the paper). The authors should make
these more precise. Similar problem with presentations of the models,
parameters, and hyperparameters.
3.  How does non-equal weighted linear combinations of l_rec and l_sim
change the results? Other variation of the overall loss function? How
do we see the loss function interaction in the training, validation
and test data? With the proposed model, these could be interesting to
observe.
4.  In equation two, the decoder seems to be very directly affecting
the quality of the output summary. Teacher forcing was used to train
the decoder in part (1) of the model, but without ground truth, I
would expect more discussions and experiments on how the Gumbel
softmax trick affect or help the performance of the output.
5.  Baseline models and metrics
(1) There should be more details on how the language model is trained,
some examples, and how the reviews are generated from the language
model as a base model (in supplement?).
(2). It is difficult to get a sense of how these metrics corresponds
to the actual perceived quality of the summary from the
presentation. (see next)
(3). It will be more relevant to evaluate the proposed design
vs. other neural models, and/or more tested and proved methods.
6. The rating classifier (CLF) is intriguing, but it's not clearly
explained and its effect on the evaluation of the performance is not
clear: One of the key metrics used in the evaluation relies on the
output rating of a classifier, CLF, that predicts reader ratings on
reviews (eg on YELP).  The classifier is said to have 72%
accuracy. First, the accuracy is not clearly defined, and the details
of the classifier and its training is not explained (what features are
its input, is the output ordinal regression).  Equation 4 is not
explained clearly: what does 'comparing' in 'by comparing the
predicted rating given the summary rating..' mean?  The classifier may
have good performance, but it's unclear how this accuracy should
affect the results of the model comparisons.
The CLF is used to evaluate the rating of output
reviews from various models. There is no justification these outputs
are in the same space or generally the same type of document with the
training sample (assuming real Yelp reviews).  That is probably
particularly true for concatenation of the reviews, and the CLF classifier
scores the concatenation very high (or  eq 4 somehow leads to highest value
for the concatenation of reviews )... It's not clear whether such a classifier is
beneficial in this context.
7. Summary vs Reviews. It seems that the model is built on an implicit
assumption that the output summary of the multi-doc should be
sufficiently similar with the individual input docs.  This may be not
true in many cases, which affects whether the approach generalizes.
Doc inputs could be covering different aspects of the review subject
(heterogeneity among the input docs, including topics, sentiment etc),
or they could have very different writing styles or length compared to
a summary.  The evaluation metrics may not work well in such
scenarios.  Maybe some pre-classification or clustering of the inputs,
and then doing summarization for each, would  help?  In the conclusions section, the
authors do mention summarizing negative and positive reviews
separately.


Variational autoencoders (VAEs) have received much attention recently as an endto-end architecture for text generation. Existing methods primarily focus on synthesizing relatively short sentences (with less than twenty words). In this paper, we propose a novel framework, hierarchically-structured variational autoencoders (hier-VAE), for generating long and coherent units of text. To enhance the model's plan-ahead ability, intermediate sentence representations are introduced into the generative networks to guide the word-level predictions. To alleviate the typical optimization challenges associated with textual VAEs, we further employ a hierarchy of stochastic layers between the encoder and decoder networks. Extensive experiments are conducted to evaluate the proposed method, where hier-VAE is shown to make effective use of the latent codes and achieve lower perplexity relative to language models. Moreover, the generated samples from hier-VAE also exhibit superior quality according to both automatic and human evaluations.

This paper proposes a hierarchical variational autoencoder for modeling paragraphs. The model creates two levels of latent variables; one level of sentence-level latent variables and another single global latent. This avoids posterior collapse issues and the authors show convincing results on a few different applications to two datasets.
Overall, it is an impressive result to be able to convincingly model paragraphs with a useful global latent variable. Apart from some issues with confusing/incomplete notation (see below), my main criticism is that the authors fail to compare their approach to "A Hierarchical Latent Structure for Variational Conversation Modeling" by Park et al. As far as I can tell, the approaches are extremely similar, except that Park et al. may not learn the prior parameters and also use a hierarchical RNN encoder rather than a CNN (which may be irrelevant). They also are primarily interested in dialog generation, so the lower-level of their hierarchy models utterances in a conversation rather than sentences in general, but I don't see this as a major difference. I'd encourage the authors to compare to this and potentially use it as a baseline. More generally, it would have been nice to see more ablation experiments (e.g. convolutional vs. LSTM encoder). Finally, I know that space is tight, but other papers on global-latent-variable models tend to include more demonstrations that teh global variable is capturing meaningful information, e.g. with attribute vector arithmetic. The authors could include results of manipulating review sentiment via attribute vector arithmetic, for example.
Specific comments:
- "The Kullback-Leibler (KL) divergence term ... which can be written in closed-form (Kingma & Welling, 2013), encourages the approximate posterior distribution qφ(z|x) to be close to the multivariate Gaussian prior p(z)." The prior is not always taken to be a multivariate Gaussian. You should add a sentence stating that the VAE prior is often taken to be a diagonal-covariance Gaussian for convenience.
- 3.2 has a few things which are unclear. In the second paragraph, you define z as the sampled latent code which is fed through an MLP "to obtain the starting state of the sentence-level LSTM decoder". But then LSTM^{sent} appears to be fed z at every timestep. LSTM^{sent} is also not defined - am I to assume that its arguments are the previous state and current input, so that z is the input at every timestep? Also, you write "where h^s_0 is a vector of zeros" which makes it sound like the starting state of the sentence-level LSTM decoder is a vector of zeros, not the output of the MLP which takes z as input. In contrast, LSTM^{word} takes three arguments as input. Which are the "state" and which are the "input" to the LSTM?
- I don't see any description of your CNN encoder (only the LSTM decoder in section 3.2, 3.3 only covers the hierarchy of latent variables, not the CNN architecture). What is its structure? Figure 1 shows a CNN encoder generating lower-level sentence embeddings and a high-level global embedding. How are those computed? It is briefly mentioned in 4.1 under "Datasets" but this seems insufficient.
- p_\theta(x | z) is defined as the generating distribution, but also as a joint distribution of z_1 and z_2. Unless I am missing something I think you are overloading the notation for p_\theta.
- I don't think enough information is given about the AAE and ARAE baselines. Are they the same as the flat-VAE, except with the KL term replaced by the an adversarial divergence between the prior and approximate posterior?


LEARNING TO UNDERSTAND GOAL SPECIFICATIONS BY MODELLING REWARD
Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations-and for instructions-not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

==========
Update
==========
Upon reviewing the paper revision and the author comments to my and the other reviewers' comments, I will revise my suggestion to that of acceptance. As I said in my summary, my primary concern was novelty with respect to prior work which the authors have clarified. They have also increased the rigor of their experimental results by providing variances in the plots.
I think this work will be of interest to the community.
==========
Strengths:
==========
- The problem of learning to predict state rewards given language in interesting and useful.
- The proposed AGILE framework is intuitively simple and works with any existing RL framework.
- With the models and tasks explored in this paper, the approach does seem to learn to evaluate whether a state matches the instructions quite well.
- The writing is very clear and direct.
==========
Concerns:
==========
[A] The discussion of differences to the closely related GAIL methodology is left until the related work after experiments. Given the similarities between GAIL and AGILE, this seems too late. The authors list three major differences between AGILE and GAIL:

1) AGILE is conditioned on a goal specification, language in this case. GAIL is unconditioned and trained for one task.
2) AGILE takes only the final/goal state rather than a trajectory like in GAIL.
3) AGILE discretizes the discriminator probability when assigning reward, GAIL does not.
Some concerns about each:

1) This is an interesting and fair difference but also a necessary and somewhat obvious modification to GAIL in tasks with explicit goal-specification.
2) This does not seem like an improvement, but rather a loss of generality. The authors justify this change saying "in AGILE the reward model observes only states s_i (either goal states from an expert, or states from the agent acting on the environment) rather than traces (s1, a1),(s2, a2), . . ., learning to reward the agent based on “what” needs to be done rather than according to “how” it must be done."
In many real applications, the how is deeply important. For instance, navigation in the world is both a "what" (arrive at location X) and a "how" (in fastest time without hitting anything or in such a way that humans aren't frightened). Further, the trace includes the final state such that the "what" is recoverable in instances where the "how" is unimportant, as in the set of tasks presented in this paper.
3) Letting the paper speak on this subject: "We considered this change of objective necessary because the GAIL-style reward would take arbitrarily low values for intermediate states visited by the agent, as the reward model will be confident as those are not goal states. The binary reward in AGILE carries a clear message to the policy that all non-goal states are equally undesirable." Firstly, all non-goal states are not equally undesirable in that some lead more easily to goal states though it is fair to argue this should be learned by the policy through expected reward. My primary gripe is the footnote following these sentences which says: "We tried values other than 0.5 for the binarization threshold, as well as not binarizing and using Dφ(c, st) directly as the reward. We got similar but slightly worse results." This seems to imply that this difference does not matter significantly, especially if different thresholds received significantly different hyperparameter tuning effort or were not conducted under multiple runs of random seeds.
A pessimistic summary would place AGILE to be a conditional GAIL with reduced ability to represent intermediate or trajectory based rewards and a possibly slightly helpful reward discretization scheme. Don't get me wrong, I think an conditional extension to GAIL is interesting and worth sharing with the community. However, this discussion comes very late and includes a design decisions (2/3) that I find poorly justified in text and completely unjustified experimentally.
I would like to hear from the authors if any of these criticisms are inaccurate. I would also welcome experiments evaluating the effect of these design decisions.
[B] In 3.2 its reported that each experiment was repeated five times however the presented results are not described as means and no variances are shown. I would like to see the results plots with shaded variances from at least 5 runs with differing random seeds.
[C] Unless I'm mistaken, the proposed architecture could also be trained with reward prediction. It would be interested in that case to see if improvement seen between A3C and A3C-AGILE extend to A3C-RP and A3C-RP-AGILE. As the authors note, the AGILE framework simply changes the source of the reward and is amicable to any RL approach. I would like to see this comparison.
[D] The reward generalization experiments seemed surprising to me. The policy was fine-tuned on the test environments but only improved from 52% to 69.3%. Trying to think about this more, I'm having trouble disentangling whether this implies poor generalization of the reward function or increased difficulty in policy learning. Could the authors provide the A3C and A3C-RP baselines for this experiment to help clarify?
[E] Just a Curiosity: What exactly is done in L2 weight clipping? (Training details in supplement)
[F] Just a Thought: In the reward-prediction (RP) setting, both the RP model and the policy share parameters. It would be possible with such an architecture to still apply the AGILE loss and I would be curious to see if this leads to interesting changes in performance. I understand that one of the advantages to learning a separate reward model is to generalize to new policies, but it is unclear if this approach would generalize less well (and finding it out would be cool!)
==========
Overview:
==========
I think extending generative adversarial imitation learning to a task-conditional setting a cool step made even more interesting in this work by having the task-specification be in compositional language. Further, the results and analysis are generally interesting though I do note some weaknesses above. Aside from some questions about the experiments, I'm mostly concerned about the positioning of the paper -- specifically with respect to prior work.  I'm looking forward to hearing from the authors and other reviewers.


LEARNING LOCALIZED GENERATIVE MODELS FOR 3D POINT CLOUDS VIA GRAPH CONVOLUTION
Point clouds are an important type of geometric data and have widespread use in computer graphics and vision. However, learning representations for point clouds is particularly challenging due to their nature as being an unordered collection of points irregularly distributed in 3D space. Graph convolution, a generalization of the convolution operation for data defined over graphs, has been recently shown to be very successful at extracting localized features from point clouds in supervised or semi-supervised tasks such as classification or segmentation. This paper studies the unsupervised problem of a generative model exploiting graph convolution. We focus on the generator of a GAN and define methods for graph convolution when the graph is not known in advance as it is the very output of the generator. The proposed architecture learns to generate localized features that approximate graph embeddings of the output geometry. We also study the problem of defining an upsampling layer in the graph-convolutional generator, such that it learns to exploit a self-similarity prior on the data distribution to sample more effectively.

The paper proposes a version of GANs specifically designed for generating point clouds. The core contribution of the work is the upsampling operation: in short, it takes as an input N points, and produces N more points (one per input) by applying a graph convolution-like operation.
Pros:
+ The problem of making scalable generative models for point clouds is clearly important, and using local operations in that context makes a lot of sense.
Cons:
- The paper is not particularly well-written, is often hard to follow, and contains a couple of confusing statements (see a non-exhaustive list of remarks below).
- The experimental evaluation seems insufficient: clearly it is possible to come up with more baselines. Even a comparison to other types of generative models would be useful (e.g. variants of VAEs, other types of GANs). There also alternative local graph-convolution-like operations (e.g. tangent convolutions) that are designed for point clouds. In addition, it is quite strange that results are reported not for all the classes in the dataset.
Various remarks:
p.1, “whereby it learns to exploit a self-similarity prior to sample the data distribution”: this is a confusing statement.
p.2, “(GANs) have been shown on images to provide better approximations of the data distribution than other generative models”: This statement is earthier too strong (all other models) or does not say much (some other models)
p.2, “However, this means that they are unable to learn localized features or exploit weight sharing.”: I see the point about no weight sharing in the generator, but feature learning
p.3, “the key difference with the work in this paper is that PointNet and PointNet++ are not
generative models, but are used in supervised problems such as classification or segmentation.”: Yet, the kind of operation that is used in the pointnet++ is quite similar to what you propose?
p.4: “because the high dimensionality of the feature vectors makes the gridding approach unfeasible.”: but you are actually dealing with the point clouds where each point is 3D?


CONTROLLING OVER-GENERALIZATION AND ITS EF- FECT ON ADVERSARIAL EXAMPLES GENERATION AND DETECTION
Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision. However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes. Further, they are also vulnerable to adversarial examples. We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set. We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization. As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples. To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an outdistribution set's fitness. We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection (dustbin) regions when generating actual adversaries.

The idea of having a separate class for out-distribution is a very interesting idea but unfortunately previously explored. In fact, in machine learning and NLP there is the OOV class which sometimes people in computer vision also use. Some of the claims in the paper can be further substantiated or explored. For example in abstract there is a simple claim that is presented too strong: We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. This claim is bigger than just CNNs and needs to be studied in a theoretical framework not an empirical one. Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax. In general it is very unlikely that you will be able to choose every variation of out-distribution cases. Much easier if you just try to solve the problem using a set of n Sigmoids (n total number of classes) and consider each output a probability distribution.
However, the studies in this paper are still valuable and I strongly recommend continuing on the same direction.


Many of our core assumptions about how neural networks operate remain empirically untested. One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks. For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers. Recently, however, interleaved pooling has largely been abandoned. This raises a number of questions: Are our intuitions about deformation stability right at all? Is it important? Is pooling necessary for deformation invariance? If not, how is deformation invariance achieved in its absence? In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers. (2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters. (3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification. (4) Pooling confers too much deformation stability for image classification at initialization, and during training, networks have to learn to counteract this inductive bias. Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.

It is often argued that one of the roles of pooling is to increase the stability of neural networks to
deformations. This paper presents empirical evidence to contest this assertion, or at least qualify it.
I appreciate empirical studies that question some of the widely accepted dogmas of deep learning.
From this point of view, the present paper is certainly interesting.
Unfortunately, the actual evidence presented is quite weak, and insufficient to draw far reaching
conclusions. An obvious objection is the authors only consider two datasets, and a very small number of
more or less standard pooling methodologies. The effect of pooling is evaluated in terms of cosine
similarlity, which is not necessarily a good proxy for the actual performance of a network.
A more serious issue is that they seem to very readily jump to unwarranted conclusions. For example,
the fact that stability to deformations (by which I necessarily mean the specific type of deformations
that they consider) tends to decrease in the middle layers of neural networks during training does not
mean that starting with a neural network with less stability would be better. Maybe some kind of
spontaneous coarse-to-fine optimization is going on in the network. Similarly, it is obvious that smoother
filters are going to lead to more stable representations. However, they might be less good at discriminative
tasks. Just because smoother filters are more stable does not automatically mean that they are more desirable.
Stability to deformations is an important but subtle topic in computer vision. For starters, it is difficult
to define what kind of deformations one wants to be insensitive to in the first place. A useful model would
likely incorporate some notion of deformations at multiple different length scales.
Just showing that one network is better than another wrt some arbitrarily defined simple class of deformations
with no reference to actual recognition performance, speed of training, or interpretation of the nature of
the deformations and the learned filters is not very convincing. I would particularly like to emphasize the
last point. I would really like to understand what pooling actually does, not just at the level of "if you
turn it off, then cosine similarity will decrease by this much or that much."